Implement data streaming to data entries

[jlorieau]

A data streaming library would allow more efficient processing of large data files, and it would allow the stream of live datasets.

Many possible libraries exist with their own advantages and disadvantages:

• Pathway.

  • Simple to use and has many connectors (database, file, Kafka).
  • Rust engine for running code
  • Seems designed to only work with one pipeline per application. (run pw.run)

• Joblib

  • Simplified pipelining for python
  • Simplified forking
  • Doesn't include connectors for data. Largely focused on efficient memoization (checkpoints).

• Smartopen

  • An open alternative with built-in smart file streaming.
  • Doesn't support concurrency.
  • Includes backends like files, s3, http
  • Transparent compression and decompression.

• Apache beam

  • Powerful stream processor
  • Python bindings are confusing and a bit hard to use for new developers (example)

[jlorieau]

Pyspark Datasources (files)

• SQL
• Parquet
• JSON
• CSV
• Binary files

Data streams