search

pathwaycom / pathway

Python ETL framework for stream processing, real-time analytics, LLM pipelines, and RAG.
https://pathway.com
Other
4.34k stars 139 forks source link

Allow sending columns with raw bytes in the python connector #39

Closed lewymati closed 2 months ago

lewymati commented 7 months ago

Is your feature request related to a problem? Please describe. Cannot stream a table with multiple columns via a Python connector, when one of them is of type bytes. For example:

message = {"col1": "foo", "col2": some_bytes}
self.next_json(message)

doesn't work as bytes are not json serializable. self.next_bytes is not useful as well as it expects bytes encoded json string.

Some workaround is to use base64 to serialize/deserialize, but it's not ideal.

KamilPiechowiak commented 2 months ago

Passing column with type bytes is supported since version 0.11.2. You can use next method of the pw.io.python.ConnectorSubject for that:

self.next(col1="foo", col2=some_bytes)

Passing bytes with next_json won't work as it expects the content to be json-serializable.