Open BlazZupan opened 6 years ago
If we will migrate to .pickle.xz, I suggest we create a new "repository on serverfiles" and migrate to that. Otherwise, Datasets will start crashing on old(er) versions of the software.
We're creating another file format to support? why not standardize on tab and loom?
tab, mtx, loom, pickle...
@anupparikh, it's not a new format, it is just the way of storing tab (csv) files. A quick trick to substantially reduce the size of our demo datasets and speed-up the loading.
So this is just for demo datasets, customers won't be creating and using pickled data.
On Mon, Jun 4, 2018 at 12:34 PM Blaž notifications@github.com wrote:
@anupparikh https://github.com/anupparikh, it's not a new format, it is just the way of storing tab (csv) files. A quick trick to substantially reduce the size of our demo datasets and speed-up the loading.
— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/biolab/orange3-single-cell/issues/208#issuecomment-394472074, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGz9Aez2korZkB61Yu38RsQFxytRdQ0ks5t5Yu2gaJpZM4UYKH6 .
This issue is related to PR https://github.com/biolab/orange3/pull/3047, which enables saving and loading of compressed pickle files. Once this is merged into Orange and released, I propose to:
This should substantially reduce the transfer and loading time of data sets. For instance, the largest data set currently included (bone marrow with AML) has 64MB, while its pickled xz variant has on 2.4MB.
This update will create an issue with backward compatibility, which will be broken.