search

fractal-analytics-platform / fractal-server

Fractal backend
https://fractal-analytics-platform.github.io/fractal-server/
BSD 3-Clause "New" or "Revised" License
11 stars 3 forks source link

Failure in zipping large folder (9k images, 900M) #1955

Open tcompa opened 1 week ago

tcompa commented 1 week ago

Ref #1954

We observed a failure upon zipping a 900 MB with this kind of structure

138M    0_convert_cellvoyager_multiplexing_to_ome_zarr
183M    1_calculate_registration__image_based_
8.1M    2_find_registration_consensus
124M    3_apply_registration_to_image
7.2M    4_convert_channel_to_label
7.1M    5_filter_label_by_size
7.4M    6_segment_secondary_objects
7.2M    7_filter_label_by_size
7.2M    8_clip_label_image
147M    9_measure_features
124M    11_measure_features
7.2M    12_label_assignment_by_overlap
7.2M    13_label_assignment_by_overlap
8.3M    14_aggregate_feature_tables
147M    10_measure_features
8.3M    15_aggregate_feature_tables
8.3M    16_aggregate_feature_tables

which corresponds to a dataset with 9282 images.

We should reproduce this locally, to see whether it has to do with a specific filesystem - and possibly move to wrapping some unix commands rather than using python built-in zip library.

tcompa commented 1 week ago

(another question is why these folders are so large, likely due to the large number of images and then files)