🐛 [cloud_functions] java.util.concurrent.ExecutionException: 1 out of 2 underlying tasks failed

[kinex]

I see several exceptions in Crashlytics (production app, release mode):

Non-fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: [firebase_functions/unknown] java.util.concurrent.ExecutionException: 1 out of 2 underlying tasks failed

#0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:607)
#1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:156)
<asynchronous suspension>
#2      MethodChannelHttpsCallable.call (package:cloud_functions_platform_interface/src/method_channel/method_channel_https_callable.dart:23)
<asynchronous suspension>
#3      HttpsCallable.call (package:cloud_functions/src/https_callable.dart:35)
<asynchronous suspension>

This looks same as https://github.com/FirebaseExtended/flutterfire/issues/6253. I don't know how to reproduce this. Crashing devices include Samsung, Huawei, Xiaomi, Motorola and Android versions 11, 10 and 9. I am not creating isolates in my own code.

Flutter 2.5.0 (stable) cloud_functions: ^3.0.2

[markusaksli-nc]

Hi @kinex Do you have a specific callable that does this? This also looks relevant https://github.com/invertase/react-native-firebase/issues/5399 Thank you

[kinex]

Hi @markusaksli-nc. It seems to happen at least with two different cloud functions I call (my other functions are called less frequently). So at least 95% of the same function calls work normally, but I still see tens of these exceptions daily. The issue you linked sounds different since I am not accessing the backend when the exceptions happen (for example, I am not updating functions). My functions have these runtime options: maxInstances 12, timeoutSeconds 30, memory 256MB.

[adamkoch]

I am seeing the same in production, I cannot reproduce locally.

Flutter 2.5.3 (stable) cloud_functions: ^3.0.4

Devices seeing issue: Galaxy S20 Ultra 5G (Android 11) Galaxy S20+ 5G (Android 11) Galaxy Note9 (Android 10)

Non-fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: refreshUser error: [firebase_functions/unknown] java.util.concurrent.ExecutionException: 1 out of 2 underlying tasks failed

#0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:607)
#1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:156)
<asynchronous suspension>
#2      MethodChannelHttpsCallable.call (package:cloud_functions_platform_interface/src/method_channel/method_channel_https_callable.dart:23)
<asynchronous suspension>
#3      HttpsCallable.call (package:cloud_functions/src/https_callable.dart:35)
<asynchronous suspension>
. Error thrown null.

[russellwheatley]

Hey @kinex, could you provide an MCVE for us to debug, please? This issue is rather stale and it appears to be an issue with either setup or android rather than a firebase_functions issue.

[kinex]

MCVE is probably too difficult to implement since I don't know why this happens. The issue still exists with Flutter 3 and with the current latest plugins and it is a very common crash in my app. Here is an updated crash log from the latest Flutter/plugins:

Non-fatal Exception: io.flutter.plugins.firebase.crashlytics.FlutterError: [firebase_functions/unknown] java.util.concurrent.ExecutionException: 1 out of 2 underlying tasks failed

#0      StandardMethodCodec.decodeEnvelope (package:flutter/src/services/message_codecs.dart:607)
#1      MethodChannel._invokeMethod (package:flutter/src/services/platform_channel.dart:167)
<asynchronous suspension>
#2      MethodChannelHttpsCallable.call (package:cloud_functions_platform_interface/src/method_channel/method_channel_https_callable.dart:23)
<asynchronous suspension>
#3      HttpsCallable.call (package:cloud_functions/src/https_callable.dart:49)
<asynchronous suspension>

I have been wondering could it be related that I am calling a cloud function from the onData handler of Firebase.firebaseAuth.userChanges().listen. Could there be an issue if a cloud function is called "too early" after the authentication. From my understanding this issue happens only occasionally, most of the time it works normally (with the same user and device).

[kinex]

Anyway, I believe you are right this is probably not a firebase_functions issue. It could rather be an issue with the Firebase functions Android SDK. Maybe something extraordinary happens in the backend (or in the app) and the Android SDK fails to handle it properly (to provide at least a clear error message). I have not seen any related issues in my iOS app.

I see from my backend logs that this same cloud function is tried to be called sometimes when not signed in (which should not happen). It could be related to this exception on the client app, but on the other hand I could not get this same exception if tried to call the cloud function without authentication on purpose. So difficult to say if those are related or not. Anyway, I am planning to add an extra authentication check just before calling a cloud function to prevent this error situation which can happen in some special cases. Interesting to see if this exception disappears after that.

[bazrafkan]

same issue Flutter (Channel stable, 3.3.0, on macOS 12.5.1 21G83 darwin-arm, locale en-US) cloud_functions: ^3.3.3

[kinex]

I just noticed that there is probably the same exception on iOS, the error message is just different (better):

Non-fatal Exception: FlutterError
[firebase_functions/17020] Network error (such as timeout, interrupted connection or unreachable host) has occurred.

Stack trace (excluding my own code):

Non-fatal Exception: FlutterError
0  ???                            0x0 StandardMethodCodec.decodeEnvelope + 653 (message_codecs.dart:653)
1  ???                            0x0 MethodChannel._invokeMethod + 296 (platform_channel.dart:296)
2  ???                            0x0 MethodChannelHttpsCallable.call + 23 (method_channel_https_callable.dart:23)
3  ???                            0x0 HttpsCallable.call + 49 (https_callable.dart:49)

I believe this is the "same" exception as on Android because the full stack traces (in my code) are equal and also the frequency of the exception is similar (top 2 exception both in Android and iOS).

Just interesting that this "timeout, interrupted connection or unreachable host" happens every day (actually many times a day).

Possible enhancement would be to improve the Android error message. It seems on Android the error message "1 out of 2 underlying tasks failed" actually means in this case something like "Network error (such as timeout, interrupted connection or unreachable host) has occurred.".

[arisAlexis]

I am getting the same error

java.util.concurrent.ExecutionException: 1 out of 2 underlying

It started to appear out of the blue. Using latest Flutter and Firebase Functions and the emulator.

Solution: Wiping the emulator of all data fixed the problem.

[MatteoAntolini]

This error is related to your API restriction configuration, not to your code. It only happens when the user is authenticated and the client tries to refresh the auth token after the expiration (3600 seconds). Allowing Identity Toolkit API and Token Service API solved the problem.

In my case, I forgot to allow Token Service API.

[kinex]

This error is related to your API restriction configuration, not to your code. It only happens when the user is authenticated and the client tries to refresh the auth token after the expiration (3600 seconds). Allowing Identity Toolkit API and Token Service API solved the problem.

In my case, I forgot to allow Token Service API.

No, I have both of those APIs already enabled and still have the issue.

I have also implemented a simple retrying logic (retry max 4 times, delays between retries 5,10, 20 and 30 seconds). I see from the statistics that sometimes retrying helps, but very often it still fails after tried total of 5 times.

Flutter 3.7.5 (stable) cloud_functions: ^4.0.11

[MatteoAntolini]

Does your debug key has the same restrictions as your production key?

[kinex]

Basicly the same, but I wouldn't keep that very interesting as the number of users is naturally on a completely different level in production where I see the error.

These are my API restrictions:

Cloud Firestore API
Cloud Functions API
Cloud Storage for Firebase API
Firebase App Check API
Firebase Dynamic Links API
Firebase Installations API
Firebase Remote Config API
Identity Toolkit API
Mobile Crash Reporting API
Token Service API

In addition I have set application restriction (Android / iOS). I am using now also AppCheck, but the problem occurred already before applying it.

[MatteoAntolini]

You definitely should remove Cloud Functions API, you don't need it to call your endpoints. It's used for creating, updating, and removing functions (reference)

Looking at my configuration I also have Firebase Management API enable. You can try to add it...

[kinex]

You are right. Cloud Functions API looks definitely unnecessary, I will remove it. Based on the Firebase Management API description, it looks like an even bigger security risk than the Cloud Functions API. My app has been live several years without that API, so I don't believe it is needed and difficult to believe that API would be related to this issue either.

[MatteoAntolini]

Well yes, I was wrong... You don't need Firebase Management API for your flutter app. My app is still not in production and on the dashboard there is no usage of that API You can try to check if in your debug dashboard there are some APIs that are not present in your production dashboard. Or maybe you got that message for a quota exceeded. Also double-check your sha1 and sha256 and make sure are the same in Google Play Console. Check also if the apiKey in your google-services.json or firebase configuration is the correct one Let me know if you find out which ones to use, I'm going to publish my app soon and I don't want to make API changes in production

[jpv123]

Any solutions yet? I'm having this issue on 1/10 users in my app

[ccctask]

This error is related to your API restriction configuration, not to your code. It only happens when the user is authenticated and the client tries to refresh the auth token after the expiration (3600 seconds). Allowing Identity Toolkit API and Token Service API solved the problem.

In my case, I forgot to allow Token Service API.

I am getting the same error

java.util.concurrent.ExecutionException: 1 out of 2 underlying

It started to appear out of the blue. Using latest Flutter and Firebase Functions and the emulator.

Solution: Wiping the emulator of all data fixed the problem.

it's work for me

[Lyokone]

Since we cannot reproduce and working solutions have been provided, I'll close this. Feel free to open a new issue if this is still happening.

[kinex]

Since we cannot reproduce and working solutions have been provided, I'll close this. Feel free to open a new issue if this is still happening.

I have to correct that working solutions have NOT been provided. I wouldn't call "retrying" as a solution.

This is still top 1 issue in my app as reported in Crashlytics. Anyway, I have not received any user feedback which I could associate with this issue, so the retry logic in my app seems to work. That's why I haven't really spent more time on investigating this further. But the exception still happens, not any real solutions have been provided.