Consider streaming

[dglazkov]

If we do, let's use jsonl.

But do we need it?

[PaulKinlan]

I think it has a number of benefits (trying to document here)

One thing that I would love to enable is more flexible tooling:

• It's easier to pipe out from the CLI and have it consumed line-by-line in another tool. e.g, we could
  • polymath ingest rss ...blah... > rss-library.json << we can remove file saving logic from CLI
  • polymath ingest rss ...blah... | pinecone-upload << another tool might be able to enable incremental upload
  • polymath export --server dalmaer.com | polymath ingest stdin

Yes, we could just pipe the full json file around, but if we want things like incremental uploads then we have to do it on an entry by entry basis anyway.

With a JSONL structure, we can:

• more easily support incremental operations on the stream without having a SAX-like JSON parser.
• if the user wants to store data in files, quickly process the list without using all the memory required to load the file
• concat all of our local file libraries into one and all the relevant data (model, embedding etc) is on one line.
  • This would enable us to potentially have different models used for embedding in the same file