search

datastax / pulsar-transformations

Apache License 2.0
10 stars 8 forks source link

[compute-ai-embeddings] The embeddings field must be a GenericArray in AVRO #81

Closed eolivelli closed 1 year ago

eolivelli commented 1 year ago

Summary:

eolivelli commented 1 year ago

This is my error:

2023-06-23T09:41:54,686+0200 [public/default/embeddings-0] WARN org.apache.pulsar.functions.instance.JavaInstanceRunnable - Encountered exception when processing message PulsarRecord(topicName=Optional[persistent://public/default/products], partition=0, message=Optional[org.apache.pulsar.client.impl.MessageImpl@6d2c511b], schema=org.apache.pulsar.client.impl.schema.generic.GenericAvroSchema@504c5562, failFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$529/0x0000000800801ae0@7dcd51bd, ackFunction=org.apache.pulsar.functions.source.PulsarSource$$Lambda$528/0x00000008008018b8@6f739f05) java.lang.ClassCastException: class java.util.ArrayList cannot be cast to class [Ljava.lang.Object; (java.util.ArrayList and [Ljava.lang.Object; are in module java.base of loader ‘bootstrap’) at com.datastax.oss.pulsar.functions.transforms.util.JsonConverter.toJson(JsonConverter.java:91) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.util.JsonConverter.toJson(JsonConverter.java:49) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.TransformContext.toJsonSerializable(TransformContext.java:236) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.TransformContext.toJsonRecord(TransformContext.java:217) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.ChatCompletionsStep.process(ChatCompletionsStep.java:59) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:302) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:292) ~[?:?] at com.datastax.oss.pulsar.functions.transforms.TransformFunction.process(TransformFunction.java:138) ~[?:?] at org.apache.pulsar.functions.instance.JavaInstance.handleMessage(JavaInstance.java:94) ~[pulsar-functions-instance.jar:2.10.4.4-SNAPSHOT] at org.apache.pulsar.functions.instance.JavaInstanceRunnable.run(JavaInstanceRunnable.java:330) ~[pulsar-functions-instance.jar:2.10.4.4-SNAPSHOT] at java.lang.Thread.run(Thread.java:833) ~[?:?]

cbornet commented 1 year ago

This JsonConverter is a copy of the one we use in various places (ES Sink, Kinesis Sink, ...) So we should see if the fix applies there too.

eolivelli commented 1 year ago

@cbornet maybe in this case there is something weird performed by the query string and at the point in the code we should see a GenericArray and not a List. Let me dig into the problem