nomic-ai / deepscatter

Zoomable, animated scatterplots in the browser that scales over a billion points
Other
1.03k stars 61 forks source link

Switch file format from '.feather' to '.arrow' #53

Open bmschmidt opened 2 years ago

bmschmidt commented 2 years ago

I've used '.feather' as the file format here, but over the last couple years the community has standardized around '.arrow' instead, to the point where the mailing list is now discussing deprecating the 'feather' methods entirely and reminding me that ".arrow is the official registered extension."

This will only get worse as time goes on, so will be changed in the next version bump. This is going to cause some breakage and will probably need an ability to specify, in plots, use of the '.feather' extension.

cceyda commented 1 month ago

I see there is an arrow_table: option in the prefs but how can I format the arrow table so that it is tiled(batched) like with quadfeather? If I pass an arrow table naively webgl buffers error out after 2**16 rows. Edit: aha I guess you just add batch_id column according to example here: https://github.com/nomic-ai/deepscatter/blob/901a038dc731995c15f21c4645ab9556f82517e0/integers.html#L79