(firebase_vertexai: ^0.2.2) FormatException: Unhandled CitationMetadata format

[niluved]

Description of the bug:

I'm using firebase_vertexai: ^0.2.2 to generate text from text and a single image. As long as i pass Gemini a "jpg" image everything is fine, but when i try with a "png" or even "jpeg" (which should be same as jpg if i'm not misunderstood) i get the exception: "FormatException: Unhandled CitationMetadata format"

Actual vs expected behavior:

You can find below the piece of code i wrote for this purpose (basically copied from the official documentation):

const String _modelName = 'gemini-1.5-flash';
final _vertexAI = FirebaseVertexAI.instanceFor(location: 'europe-west8');

static Future<String?> callVertexAI({required XFile image}) async {
    final model = _vertexAI.generativeModel(model: _modelName);
    const String prompt = "What's in the picture?";
    final Uint8List imageBytes = await File(image.path).readAsBytes();
    // utilizzo il package mime per identificare il mimetype
    final lookup = lookupMimeType(image.path);
    debugPrint('mimetype individuato: $lookup');
    final content = [
      Content.multi([
        TextPart(prompt),
        DataPart(lookup ?? 'image/jpg', imageBytes),
      ])
    ];
    try {
      final response = await model.generateContent(content);
      return response.text;
    } catch (e) {
      debugPrint(e.toString());
      return null;
    }
  }

The issue persists even if i hardcode the correct mimetype in DataPart constructor.

Any other information you'd like to share?

No response

[CurtlyCritchlow]

I experience the same bug using firebase_vertexai ^0.2.2 when using both 'gemini-1.5-pro' and 'gemini-1.5-flash' models.

Sometimes the model returns the answer but most times I get FormatException: Unhandled CitationMetadata format

final model = FirebaseVertexAI.instance.generativeModel(
      model: 'gemini-1.5-flash',
      systemInstruction: Content.system(
        kChatContext,
      ),
      generationConfig: GenerationConfig(
        maxOutputTokens: 1024,
        temperature: 1,
        topP: 0.95,
      ),
      safetySettings: [
        SafetySetting(HarmCategory.dangerousContent, HarmBlockThreshold.none),
        SafetySetting(HarmCategory.harassment, HarmBlockThreshold.none),
        SafetySetting(HarmCategory.hateSpeech, HarmBlockThreshold.none),
        SafetySetting(HarmCategory.sexuallyExplicit, HarmBlockThreshold.none),
      ],
    );
  final chat = model.startChat();
  var response = await chat.sendMessage(
    Content.text('What is osmosis'),
  );

[garysm]

I think it's related to the differences in the API mentioned in #83. I'm running into the same issue, but only on specific multimodal prompts.

[garysm]

Changing to 'citations' instead of 'citationSources' works. You can make the change in lib/src/api.dart to work with Vertex as a workaround.

CitationMetadata _parseCitationMetadata(Object? jsonObject) {
  return switch (jsonObject) {
    {'citations': final List<Object?> citationSources} =>
      CitationMetadata(citationSources.map(_parseCitationSource).toList()),
    _ => throw FormatException('Unhandled CitationMetadata format', jsonObject),
  };
}

[niluved]

also i get this same error when i upload some pdf or mp3 files. Usually the shortest the file , the most likely it works. The longer the file is, the more likely it returns this exception.

[Claudiu-Dinea]

Same problem here..

[natebosch]

Looks like the vertex SDK can no longer rely on the developer SDK for parsing this JSON content. I think the stack trace for the exception points to this SDK, but the fix likely needs to be published in the vertex SDK.

[natebosch]

A fix for this has been published. If your project is impacted you can dart pub upgrade or flutter pub upgrade to pick up the fix.

[SebastianKlaiber]

@natebosch is the fix published to firebase_vertexai package? I still get the error on version firebase_vertexai: ^0.2.2+3

[devoncarew]

It should be fixed by running dart pub upgrade; you'll want to verify that you've transitively gotten version 0.4.4 of package:google_generative_ai.

@cynthiajoan - we should probably publish a new version of firebase_vertexai that revs it min. dep on google_generative_ai to 0.4.4; it'll make it explicit that firebase_vertexai requires the latest bug fix release of google_generative_ai.

[natebosch]

we should probably publish a new version of firebase_vertexai that revs it min. dep on google_generative_ai to 0.4.4; it'll make it explicit that firebase_vertexai requires the latest bug fix release of google_generative_ai.

I had decided against republishing firebase_vertexai at the time because it wouldn't directly impact users. I overlooked the utility of a constraint we can advertise to folks without having to get into the weeds of checking what version of transitive deps you have.

We might want to look for tools we can ship in the SDK to make it easier to understand what is happening with transitive packages.