search

jamesdolezal / slideflow

Deep learning library for digital pathology, with both Tensorflow and PyTorch support.
https://slideflow.dev
GNU General Public License v3.0
230 stars 38 forks source link

Unable to use sf.DatasetFeatures with zip files #380

Open maxxfarma opened 1 month ago

maxxfarma commented 1 month ago

Description

sf.DatasetFeatures is not working for a zip file even though I export packages before running sf.DatasetFeatures. %sh export SF_BACKEND=torch export SF_SLIDE_BACKEND=cucim export SF_ALLOW_ZIP=0

OSError: [Errno 95] Operation not supported

This model is developed and licensed by Owkin, Inc. The license for use is provided in the LICENSE file in the same directory as this source file (slideflow/model/extractors/histossl/LICENSE), and is also available at https://github.com/owkin/HistoSSLscaling. By using this feature extractor, you agree to the terms of the license.

100%|██████████| 1.51G/1.51G [01:47<00:00, 14.1MB/s]

Exception ignored in: <function ZipFile.del at 0x7fe8e9f70790>

Traceback (most recent call last): File "/usr/lib/python3.10/zipfile.py", line 1821, in del self.close() File "/usr/lib/python3.10/zipfile.py", line 1843, in close self._fpclose(fp) File "/usr/lib/python3.10/zipfile.py", line 1943, in _fpclose fp.close() OSError: [Errno 95] Operation not supported WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be WARNING:Comm:No such comm: 68ffee887ecd48ec936a9c270212d5be

Also tried with export SF_ALLOW_ZIP = 1 and got this error:

<function ZipFile.del at 0x7f176ee2e290> Traceback (most recent call last): File "/usr/lib/python3.10/zipfile.py", line 1821, in del self.close() File "/usr/lib/python3.10/zipfile.py", line 1843, in close self._fpclose(fp) File "/usr/lib/python3.10/zipfile.py", line 1943, in _fpclose fp.close() OSError: [Errno 95] Operation not supported WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 Exception ignored in: Exception ignored in: WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 <function ZipFile.del at 0x7f176ee2e290> WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 <function ZipFile.del at 0x7f176ee2e290>WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 Traceback (most recent call last): WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 File "/usr/lib/python3.10/zipfile.py", line 1821, in del Traceback (most recent call last): WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 File "/usr/lib/python3.10/zipfile.py", line 1821, in del WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 self.close()self.close()WARNING:Comm:No such comm: 3827dbdfa1134711ae1408ad6d42c629 File "/usr/lib/python3.10/zipfile.py", line 1843, in close

OSError: [Errno 95] Operation not supported

Expected behavior

This method should extract features for a deep learning algorithm and has worked on zip files in the past.

Environment:

On databricks

Additional context

Using zip files on this method in the past has worked with earlier versions of slideflow.

Meijian commented 3 weeks ago

@jamesdolezal Hi James, we are having a similar zip file issue in 2.3.0 to what we had previously with older versions of Slideflow when using it in Data Bricks. You helped address the issue by setting the environmental variable "export SF_ALLOW_ZIP=1". But now this command seems not working anymore. Could you please help address this? Thanks!