search

polygon-io / issues

Quickly track and report problems with polygon.io
29 stars 0 forks source link

Apache arrow format for the endpoints #257

Open alippai opened 11 months ago

alippai commented 11 months ago

It should be safer, faster and easier to process the data. Streaming frequent Apache arrow batches is a good replacement for json and csv, it can be used as-is in most client languages (also it's a backend for pandas without any conversion or parsing). Depending on your storage format, there is a chance it can be up to 10x faster on the server side as well (no serialization needed, you just put it on the wire)

alippai commented 6 months ago

I see some arrow related forks by polygon. Can you share any info on the progress? Do you see it promising?