[bug]: `flutterfire configure` throws if too many projects

[guidezpl]

Is there an existing issue for this?

• [X] I have searched the existing issues.

CLI Version

0.2.0

Firebase Tools version

11.23.1

Flutter Doctor Output

N/A

Description

flutter configure throws if the list of projects is long enough

flutterfire configure
⠏ Fetching available Firebase projects...                                                                                         
Unhandled exception:
FormatException: Unterminated string (at line 1970, character 13)
        "hos
            ^

#0      _ChunkedJsonParser.fail (dart:convert-patch/convert_patch.dart:1383:5)
#1      _ChunkedJsonParser.close (dart:convert-patch/convert_patch.dart:494:9)
#2      _parseJson (dart:convert-patch/convert_patch.dart:36:10)
#3      JsonDecoder.convert (dart:convert/json.dart:610:36)
#4      runFirebaseCommand (package:flutterfire_cli/src/firebase.dart:95:25)
<asynchronous suspension>
#5      getProjects (package:flutterfire_cli/src/firebase.dart:114:20)
<asynchronous suspension>
#6      ConfigCommand._selectFirebaseProject (package:flutterfire_cli/src/commands/config.dart:300:24)
<asynchronous suspension>
#7      ConfigCommand.run (package:flutterfire_cli/src/commands/config.dart:390:37)
<asynchronous suspension>
#8      CommandRunner.runCommand (package:args/command_runner.dart:212:13)
<asynchronous suspension>
#9      main (file:///Users/plg/.pub-cache/hosted/pub.dev/flutterfire_cli-0.2.7/bin/flutterfire.dart:57:5)
<asynchronous suspension>

Steps to reproduce

1. Run flutterfire configure with a list of projects from firebase projects:list --json that is ~2000 lines or more

Expected behavior

No crashing, better error

Screenshots

No response

Additional context and comments

No response

[ssdev98]

Try this solution https://github.com/invertase/flutterfire_cli/issues/27#issuecomment-1443371577

[guidezpl]

I tried that, however, in this case, it seems that there is a limit on the JSON string that can be obtained from the firebase command.

[FDuhen]

Same here. Filtering on the project ID has no effect either

[HE-LU]

I have exactly the same issue. This is quite a blocker. This needs to be fixed!

[HE-LU]

@russellwheatley @Salakar This happens from both my Linux machine and macOS machine. So it is platform-independent. I cannot urge, how critical this bug is. It completely disables our company to use it on any of our projects. The whole flutterfire tool is completely useless and no part of it can be used because of this bug.

EDIT: The issue will be probably here in reading the stdout: https://github.com/invertase/flutterfire_cli/blob/main/packages/flutterfire_cli/lib/src/firebase.dart#L88

  final process = await Process.run(
    'firebase',
    execArgs,
    workingDirectory: workingDirectoryPath,
    runInShell: true,
  );

  final jsonString = process.stdout.toString();

[russellwheatley]

hmmmm, I've just created a JSON file with the output received from firebase projects:list --json. I essentially created a JSON file with 3600+ lines of dummy projects.

Sample with 3600+ lines of dummy projects

{
  "status": "success",
  "result": [
     {
      "projectId": "wwwwwwwwwwwwwwwwwwwwwwwwwwww",
      "projectNumber": "34905345345",
      "displayName": "wwwwwwwwwwwwwwwwwwwwwwwwwwww",
      "name": "projects/wwwwwwwwwwwwwwwwwwwwwwwwwwww",
      "resources": {
        "hostingSite": "wwwwwwwwwwwwwwwwwwwwwwwwwwww"
      },
      "state": "ACTIVE",
      "etag": "wwwwwwwwwwwwwwwwwwwwwwwwwwww"
    }
]
}

I made a "called.dart" file like so:

import 'dart:io';

void main(List<String> args) async {
  final String fileContent = await File('./large.json').readAsString();

  stdout.write(fileContent);
}

Which I call from this "caller.dart" script which mimics the logic used when calling firebase projects:list --json.

import 'dart:convert';
import 'dart:io';

void main(List<String> args) async {
  final result = await Process.run(
    'dart', 
    ['called.dart'],
    runInShell: true, 
  );

  final output = result.stdout.toString();

  final commandResult = Map<String, dynamic>.from(
    const JsonDecoder().convert(output) as Map,
  );

  print(commandResult);
}

It works fine, I cannot recreate this issue. Does anyone have a repro I could use?

[russellwheatley]

Is someone who has this problem able to test this PR for me? https://github.com/invertase/flutterfire_cli/pull/203

You will need to:

1. clone the repo
2. checkout the the PR branch - fix-firebase-command.
3. install melos - dart pub global activate melos
4. run melos bootstrap in project repo - melos bs
5. Use local version of FlutterFire CLI. Run this in the root of the project repo - dart pub global activate --source="path" . --executable="flutterfire"
6. Now run flutterfire configure in a Flutter app and see if this error still occurs.

Would be a great help. Thanks!

[guidezpl]

I tried it but I'm still running into the issue

FirebaseCommandException: An error occured on the Firebase CLI when attempting to run a command.
COMMAND: firebase projects:list --json 
ERROR: Failed to parse JSON response from Firebase CLI. JSON response: {
...
    },
    {

 Error response: FormatException: Unexpected end of input (at line 1979, character 6)

     ^

[russellwheatley]

Thanks for checking @guidezpl!

when you run this command, what is happening?

firebase projects:list --json

presumably you get a JSON response with a result array that has a list of all your projects?

Also - what machine are you running this on?

[guidezpl]

Right, with a list of all projects I have access to (in a large organization)

{
  "status": "success",
  "result": [
  // huge array
  ]
}

This on a mac. Is there a "piping" limit? Because the actual line count is closer to 3000

firebase projects:list --json | wc -l
✔ Preparing the list of your Firebase projects
    1978

i.e. this copies 1978 lines to the clipboard

firebase projects:list --json | pcopy

[russellwheatley]

Excellent work, @guidezpl, I've just created 2 different executable scripts from the command line and placed in my /usr/local/bin directory to mimic command line behaviour:

one in nodejs:

#!/usr/bin/env node

const generateJSON = () => {
  const dataList = [];

  for (let i = 0; i < 400; i++) {
    dataList.push({
      projectId: `flutterfire-e2e-tests-${i}`,
      projectNumber: '866672724757',
      displayName: `flutterfire-e2e-tests-${i}`,
      name: `projects/flutterfire-e2e-tests-${i}`,
      resources: {
        hostingSite: `flutterfire-e2e-tests-${i}`,
        storageBucket: `flutterfire-e2e-tests-${i}.appspot.com`,
        locationId: 'europe-west2',
      },
      state: 'ACTIVE',
      etag: '1_e518b189-c486-4e4e-a83f-2f12207343c4',
    });
  }

  const output = {
    status: 'success',
    result: dataList,
  };

  console.log(JSON.stringify(output, null, 2));
};

generateJSON();

and another in Dart:

#!/usr/bin/env dart

import 'dart:convert';
import 'dart:io';

void main() {
  final List<Map<String, dynamic>> dataList = [];

  for (int i = 0; i < 400; i++) {
    dataList.add({
      'projectId': 'flutterfire-e2e-tests-$i',
      'projectNumber': '866672724757',
      'displayName': 'flutterfire-e2e-tests-$i',
      'name': 'projects/flutterfire-e2e-tests-$i',
      'resources': {
        'hostingSite': 'flutterfire-e2e-tests-$i',
        'storageBucket': 'flutterfire-e2e-tests-$i.appspot.com',
        'locationId': 'europe-west2',
      },
      'state': 'ACTIVE',
      'etag': '1_e518b189-c486-4e4e-a83f-2f12207343c4',
    });
  }

  final Map<String, dynamic> output = {
    'status': 'success',
    'result': dataList,
  };

  stdout.write(jsonEncode(output));
}

They both output JSON responses with over 5000 lines. I used another dart script to execute them both and received the valid JSON response from both. It seems to me to be a problem with the Firebase CLI not responding with the full, valid JSON response.

This correlates with your output as well. If you're expecting a JSON response from firebase projects:list --json with over 3000 lines and getting 1978, that indicates incorrect output.

Could you do one last thing, @guidezpl. Could you copy/paste the response from that command and put into a JSON validator like: https://jsonformatter.org/ and confirm the output is corrupted? If it is, this is an issue on the Firebase CLI and I will raise it there. Thanks 👍

[guidezpl]

The output from firebase projects:list --json | pbcopy is definitely corrupted, but the output from firebase projects:list --json is not.

I'm not familiar with bash but could firebase be chunking its output stream? That would explain it appearing correct in the console, while failing when piping to other commands and processes.

[russellwheatley]

I think it must be something to do with the terminal or OS perhaps. I've created 250 Firebase projects 😅 to try to reproduce. It outputs a JSON response over 3000 lines but it worked fine in my Iterm & VS Code terminal 🤔.

FYI; flutter doctor command:

[✓] Flutter (Channel stable, 3.13.1, on macOS 13.4 22F66 darwin-arm64 (Rosetta), locale en-GB)
[✓] Android toolchain - develop for Android devices (Android SDK version 33.0.1)
[✓] Xcode - develop for iOS and macOS (Xcode 14.3.1)
[✓] Chrome - develop for the web
[✓] Android Studio (version 2022.3)
[✓] IntelliJ IDEA Community Edition (version 2020.1.1)
[✓] VS Code (version 1.81.1)
[✓] Connected device (2 available)
[✓] Network resources

[HE-LU]

I can confirm that the output of firebase projects:list --json is not corrupted. And I can also confirm that using: firebase projects:list --json | xclip -selection clipboard gives me the incomplete and corrupted output. (I guess Linux is using xclip instead of pbcopy?). The output of xclip is just slightly longer than the fail point of flutterfire configure command.

[russellwheatley]

I think it is the Firebase CLI response in certain environments that is causing the bug. I found this issue on the firebase-tools repo: https://github.com/firebase/firebase-tools/issues/3286

and a potential fix here that actually refers to this bug on the FlutterFire CLI:https://github.com/firebase/firebase-tools/pull/5310

I've elevated this issue/PR with the Firebase team so hopefully we can get this resolved upstream. Thank you for your patience!

[russellwheatley]

Hey folks, the Firebase CLI has just released a version which ought to fix this issue: https://github.com/firebase/firebase-tools/releases/tag/v12.5.4

Could you please run npm i -g firebase-tools to install the latest Firebase CLI, and check if this has fixed this bug on the FlutterFire CLI? (i.e. running flutterfire configure w/ arguments). Thanks.

[guidezpl]

I think that did the trick! With firebase 12.5.4:

[russellwheatley]

Thank you for confirming, @guidezpl! I'll close this issue out.