search

davidmigloz / langchain_dart

Build LLM-powered Dart/Flutter applications.
https://langchaindart.dev
MIT License
416 stars 74 forks source link

Not able to going further of 'thread.message.completed' event. #426

Closed saulram closed 5 months ago

saulram commented 5 months ago

System Info

Flutter (openai_dart) After migrating to V3.2 (Also tested in 3.1) I'm not able to process the thread.message.completed event, so when trying to display the final text result of the query, got stuck into an error.

I'm attaching the code I'm running on this and also the error / stacktrace.

Code


import 'package:aspiral_flutter_client/features/app/ui/app_viewmodel.dart';
import 'package:logger/logger.dart';
import 'package:openai_dart/openai_dart.dart' hide Image;

class AlertDetailViewModel extends NavigationModel {
  late OpenAIClient client;
  String threadId = "";

  // Text to be displayed in the chat
  String _text = '';

  String get text => _text;

  set text(String value) {
    _text = value;
    notifyListeners();
  }

  final Logger log = Logger();

  final String apiKey;
  final String assistantId;

  AlertDetailViewModel({
    required this.apiKey,
    required this.assistantId,
  }) {
    log.i("Alert Detail ViewModel Created");
    client = OpenAIClient(
      apiKey: apiKey,
    );
  }

  Future<void> init({
    required String alertId,
    required String brand,
    required String lang,
  }) async {
    final String errorMessage =
        'Language: $lang. Can you help me understand what the error code $alertId means on a $brand device?';

    try {
      final response = await client.createThread(
        request: CreateThreadRequest(messages: [
          CreateMessageRequest(
            role: MessageRole.user,
            content: CreateMessageRequestContent.text(errorMessage),
          ),
        ]),
      );

      log.i("Thread created with id: ${response.id}");
      threadId = response.id;

      fetchData();
    } catch (e, s) {
      log.e("Failed to create thread: $e\n$s");
    }

    notifyListeners();
  }

  void handleThreadMessageDelta(AssistantStreamEvent response) {
    MessageDeltaObject? message = response.data as MessageDeltaObject?;
    message?.delta.content?.first.when(
      text: (int index, String type, MessageDeltaContentText? textContent) {
        if (textContent?.value != null) {
          text += textContent?.value ?? '';
          notifyListeners();
        } else {
          log.e("Text content was null");
        }
      }, imageFile: (int index, String type, MessageContentImageFile? imageFile) {
        log.i("Image file found");
    },
    );
  }

  Future<void> fetchData() async {
    if (threadId.isEmpty) {
      log.e("Thread ID is empty");
      busy = false;
      notifyListeners();
      return;
    }
    try {
      final stream = client.createThreadRunStream(
        threadId: threadId,
        request: CreateRunRequest(
          assistantId: assistantId,
          model: const CreateRunRequestModel.enumeration(RunModels.gpt4o),
        ),
      );

      await for (var event in stream) {
        if (event.event == EventType.threadMessageCompleted) {
          MessageObject message = event.data as MessageObject;
          text = message.content[0].text ?? '';
          busy = false;
          notifyListeners();
          break;
        }

        log.i("event: ${event.event}");
        switch (event.event) {
          case EventType.threadMessageDelta:
            handleThreadMessageDelta(event);
            break;
          case EventType.threadMessageCompleted:
            log.i("Completed event data: ${event.data}");
            MessageObject message = event.data as MessageObject;
            text = message.content[0].text ?? '';
            notifyListeners();
            break;
          default:
            busy = text.isEmpty;
            break;
        }
      }
    } catch (e, s) {
      log.e("Error fetching data: $e\n$s");
      busy = false;
      notifyListeners();
    }
  }
}

Every thread message delta is processed in the right way until we reach the message.completed event which breaks the package. this is the error:

┌───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────
│ #0   packages/aspiral_flutter_client/features/alert_detail/viewmodels/alert_detail_screen_viewmodel.dart 130:7  fetchData
├┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄┄
│ ⛔ Error fetching data: TypeError: null: type 'Null' is not a subtype of type 'String'
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/errors.dart 297:3                         throw_
│ ⛔ dart-sdk/lib/_internal/js_shared/lib/rti.dart 1385:3                                                _failedAsCheck
│ ⛔ dart-sdk/lib/_internal/js_shared/lib/rti.dart 1363:3                                                _generalAsCheckImplementation
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 3909:32                                     _$36$36MessageContentTextAnnotationsFileCitationImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 40484:7                               fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 40372:53                              _$36MessageContentTextAnnotationsFileCitationFromJson
│ ⛔ packages/openai_dart/src/generated/schema/message_content_text_annotations_file_citation.dart 29:7  fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 5239:67                                     _$36$36MessageContentTextAnnotationsFileCitationObjectImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 55020:7                               fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 54754:62                              _$36MessageContentTextAnnotationsFromJson
│ ⛔ packages/openai_dart/src/generated/schema/message_content_text_annotations.dart 65:7                fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 3883:45                                     <fn>
│ ⛔ dart-sdk/lib/internal/iterable.dart 425:31                                                          elementAt
│ ⛔ dart-sdk/lib/internal/iterable.dart 354:26                                                          moveNext
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 1140:20                   next
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/patch/core_patch.dart 555:14                                  of
│ ⛔ dart-sdk/lib/internal/iterable.dart 224:7                                                           toList
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 3883:55                                     _$36$36MessageContentTextImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 40288:7                               fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 40176:30                              _$36MessageContentTextFromJson
│ ⛔ packages/openai_dart/src/generated/schema/message_content_text.dart 28:7                            fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 5166:32                                     _$36$36MessageContentTextObjectImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 53998:7                               fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 53401:39                              _$36MessageContentFromJson
│ ⛔ packages/openai_dart/src/generated/schema/message_content.dart 57:7                                 fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 3568:38                                     <fn>
│ ⛔ dart-sdk/lib/internal/iterable.dart 425:31                                                          elementAt
│ ⛔ dart-sdk/lib/internal/iterable.dart 354:26                                                          moveNext
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/private/ddc_runtime/operations.dart 1140:20                   next
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/patch/core_patch.dart 555:14                                  of
│ ⛔ dart-sdk/lib/internal/iterable.dart 224:7                                                           toList
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 3568:48                                     _$36$36MessageObjectImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 37281:7                               fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 36947:25                              _$36MessageObjectFromJson
│ ⛔ packages/openai_dart/src/generated/schema/message_object.dart 67:7                                  fromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.g.dart 5835:27                                     _$36$36MessageStreamEventImplFromJson
│ ⛔ packages/openai_dart/src/generated/schema/schema.freezed.dart 62774:7                               fromJson
│ ⛔ packages/openai_dart/src/client.dart 252:37                                                         <fn>
│ ⛔ dart-sdk/lib/async/stream_pipe.dart 213:31                                                          [_handleData]
│ ⛔ dart-sdk/lib/async/stream_pipe.dart 153:5                                                           [_handleData]
│ ⛔ dart-sdk/lib/async/zone.dart 1594:9                                                                 runUnaryGuarded
│ ⛔ dart-sdk/lib/async/stream_impl.dart 339:5                                                           [_sendData]
│ ⛔ dart-sdk/lib/async/stream_impl.dart 515:13                                                          perform
│ ⛔ dart-sdk/lib/async/stream_impl.dart 620:10                                                          handleNext
│ ⛔ dart-sdk/lib/async/stream_impl.dart 591:7                                                           callback
│ ⛔ dart-sdk/lib/async/schedule_microtask.dart 40:11                                                    _microtaskLoop
│ ⛔ dart-sdk/lib/async/schedule_microtask.dart 49:5                                                     _startMicrotaskLoop
│ ⛔ dart-sdk/lib/_internal/js_dev_runtime/patch/async_patch.dart 181:7                                  <fn>
│ ⛔ 
└───────────────────────────────────────────────────────────────────────────────────────────────────────────────────────

Also, debugged a little bit and I guess that the issue is in the MessageObject class, I found some null values in the response that seems to being not handled in the right way. I'm attaching also the response of the unprocesed events that I could catch from the browser network tab.

event: thread.message.completed
data: {"id":"msg_Fn6zTwr3kziqsxPDsk3fU9LR","object":"thread.message","created_at":1715820384,"assistant_id":"asst_K4iArkRmMciSi5PHdr2ycYID","thread_id":"thread_3N31Erp0eCVhAhKh5BdMk1aQ","run_id":"run_icH57L9iJ15QX68dFK5wMnRO","status":"completed","incomplete_details":null,"incomplete_at":null,"completed_at":1715820391,"role":"assistant","content":[{"type":"text","text":{"value":"### Código de Error 2 en un Dispositivo SUNGROW\n\n#### Número del Error\n002\n\n#### Descripción del Error\nSobretensión de red\n\n#### Causas Potenciales\nLa tensión de la red supera el valor de protección establecido.\n\n#### Efectos en el Sistema\nEl inversor se desconecta de la red para protegerse y proteger la instalación eléctrica hasta que la tensión vuelva a los niveles seguros.\n\n#### Recomendaciones\n1. En general, el inversor se volverá a conectar a la red después de que la tensión vuelva a niveles normales.\n2. Si el fallo ocurre repetidamente:\n   - Mida el voltaje real de la red y comuníquese con su proveedor de energía eléctrica si este supera el valor establecido.\n   - Revise los parámetros de protección configurados en la aplicación o en la pantalla LCD del dispositivo.\n   - Verifique si el área de la sección transversal del cable cumple con los requisitos.\n   - Si el fallo no es causado por ninguna de las situaciones anteriores, comuníquese con el soporte técnico de SUNGROW.\n\n#### Origen de la Información\n[Knowledgebase]\n\nPuedes encontrar más detalles en el documento que has proporcionado 【6:0†source】.","annotations":[{"type":"file_citation","text":"【6:0†source】","start_index":1109,"end_index":1121,"file_citation":{"file_id":"file-TrXXpBh28xHodvArV1z8Zp4U"}}]}}],"attachments":[],"metadata":{}}

event: thread.run.step.completed
data: {"id":"step_QbMHrEYAAjH3I3wq0us6abqw","object":"thread.run.step","created_at":1715820384,"run_id":"run_icH57L9iJ15QX68dFK5wMnRO","assistant_id":"asst_K4iArkRmMciSi5PHdr2ycYID","thread_id":"thread_3N31Erp0eCVhAhKh5BdMk1aQ","type":"message_creation","status":"completed","cancelled_at":null,"completed_at":1715820392,"expires_at":1715820981,"failed_at":null,"last_error":null,"step_details":{"type":"message_creation","message_creation":{"message_id":"msg_Fn6zTwr3kziqsxPDsk3fU9LR"}},"usage":{"prompt_tokens":14915,"completion_tokens":253,"total_tokens":15168}}

event: thread.run.completed
data: {"id":"run_icH57L9iJ15QX68dFK5wMnRO","object":"thread.run","created_at":1715820381,"assistant_id":"asst_K4iArkRmMciSi5PHdr2ycYID","thread_id":"thread_3N31Erp0eCVhAhKh5BdMk1aQ","status":"completed","started_at":1715820382,"expires_at":null,"cancelled_at":null,"failed_at":null,"completed_at":1715820392,"required_action":null,"last_error":null,"model":"gpt-4o","instructions":"Instrucciones para el Asistente de Diagnóstico de Errores Eléctricos:\nRecepción del Error:\nSiempre recibirás un prompt que incluirá el número del error. Este número es clave para iniciar el proceso de diagnóstico.\nBúsqueda en la Base de Datos:\nTu primera acción será buscar el número de error en nuestra base de datos interna (knowledgebase). Esta base contiene información detallada sobre una amplia variedad de errores relacionados con sistemas y equipos eléctricos industriales.\nProcedimiento si no se Encuentra en la Base de Datos:\nSi el error no está registrado en nuestra base de datos, procede a buscar información relevante en internet automáticamente. Asegúrate de consultar fuentes confiables y actualizadas.\nFormato de Respuesta:\nAl proporcionar la información encontrada, debes formatearla claramente en Markdown. Indica las siguientes secciones en cada respuesta:\nNúmero del Error: Indica el código del error consultado.\nDescripción del Error: Detalla qué representa este error dentro del sistema.\nCausas Potenciales: Enumera las posibles causas que pueden generar este error.\nEfectos en el Sistema: Explica cómo este error podría afectar el funcionamiento del sistema.\nRecomendaciones: Ofrece pasos concretos o medidas preventivas para abordar el error, basándote en las mejores prácticas del sector.\nOrigen de la Información:\nEs esencial que especifiques si la información proviene de la base de datos interna o de fuentes externas en internet. Usa las etiquetas [Knowledgebase] o [Internet] al inicio de tu respuesta para indicar el origen.\nMensaje de No Encontrado:\nSi no se encuentra información relevante, se devolverá un mensaje formateado en Markdown indicando la ausencia de datos sobre el error consultado.\nIdioma:\nLas respuestas deberán ser proporcionadas en español, a menos que se solicite explícitamente en inglés.\nFinalización:\nConfirma que la respuesta es completa y verifica que todos los datos proporcionados son precisos y están actualizados antes de enviar la respuesta al usuario.","tools":[{"type":"code_interpreter"},{"type":"file_search"}],"tool_resources":{"code_interpreter":{"file_ids":[]}},"metadata":{},"temperature":1.0,"top_p":1.0,"max_completion_tokens":null,"max_prompt_tokens":null,"truncation_strategy":{"type":"auto","last_messages":null},"incomplete_details":null,"usage":{"prompt_tokens":16085,"completion_tokens":271,"total_tokens":16356},"response_format":"auto","tool_choice":"auto"}

Related Components

Reproduction

Run the code sample, make sure to pass the assistantId and attach data into the assistant for file search.

Expected behavior

when reached message completed event, should fail

davidmigloz commented 5 months ago

@saulram thanks for reporting it!

The quote property from the MessageContentTextAnnotationsFileCitation class was supposed to be required as per the OpenAI API spec. But it seems the server doesn't always send it.

I've made it nullable in the client (https://github.com/davidmigloz/langchain_dart/pull/428) and reported it to OpenAI (https://github.com/openai/openai-openapi/issues/263).

Let me know if you face any other issues.

davidmigloz commented 5 months ago

Released in openai_dart: 0.3.2+1

saulram commented 5 months ago

Thanks for the hard work! @davidmigloz I'll check and report if found something else.