remove pytorch dependency for parsing .pt

flatironinstitute / Cryo-EM-Heterogeneity-Challenge-1

The Inaugural Flatiron Institute Cryo-EM Heterogeneity Community Challenge

https://osf.io/8h6fz/

MIT License

5 stars 0 forks source link

remove pytorch dependency for parsing .pt #79

Open geoffwoollard opened 1 month ago

geoffwoollard commented 1 month ago

External programs that need access to the volumes in .pt (for external map to map distances), would need to have pytorch as a dependency.

There are a few options

[ ] leave as is, and require pytorch to be installed to parse these files
[x] replace with .npz format, or .pkl
[ ] see if .pt can be read it without pytorch (e.g. read in as pickle or some other seralized reader). This may require saving the .pt file in a special way (serlize it with some encoding?)

geoffwoollard commented 1 month ago

@DSilva27 what do you think?

geoffwoollard commented 1 month ago

@DavidHerreros to look into writing a dataloader (without pytorch as dependency) for these, which will help external map to map distances.

It can be installed as a external and related repo (without pytorch as a dependency)

geoffwoollard commented 1 month ago

Note that the ground truth maps are a flat file of the array. They are in OSF as a .pt and .npy format

In the map to map config file, you need to use the .pt format. But this could be extended.

data/dataset_2_ground_truth/maps_gt_flat.pt

DSilva27 commented 1 month ago

If you really think this is necessary, you could simply do np.save with allow_pickle=True. We are saving dictionaries, so this should work. The only issue is that when you load you have to do np.load with allow_pickle=True, and then do a np.ndarray.tolist() transform.