Downloader: Not a directory: 'data/.DS_Store'

[vadimnazarov]

Hi everyone, thank you for the fantastic tool! This is a significant step forward after more traditional ways of downloading the data directly or submitting prediction files. I'm currently in the Broad challenge(s) and encountered an error with the data loading on Mac, python 3.12.6, crunch-cli 5.5.0.

The error is relatively straightforward:

data/DC1.zarr.zip: <blah blah>
data/DC1.zarr.zip: <blah blah>
Traceback (most recent call last):
  File "/Users/vdn/Projects/broad/.venv/bin/crunch", line 8, in <module>
    sys.exit(cli())
             ^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/click/core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/click/core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/crunch/cli.py", line 211, in setup
    command.download(force=True)
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/crunch/command/download.py", line 81, in download
    file_paths = downloader.save_all(
                 ^^^^^^^^^^^^^^^^^^^^
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/crunch/downloader.py", line 156, in save_all
    save_one(data_file, force, print, progress_bar)
  File "/Users/vdn/Projects/broad/.venv/lib/python3.12/site-packages/crunch/downloader.py", line 139, in save_one
    shutil.rmtree(destination_path)
  File "/usr/local/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/shutil.py", line 759, in rmtree
    _rmtree_safe_fd(stack, onexc)
  File "/usr/local/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd
    onexc(func, path, err)
  File "/usr/local/Cellar/python@3.12/3.12.6/Frameworks/Python.framework/Versions/3.12/lib/python3.12/shutil.py", line 686, in _rmtree_safe_fd
    with os.scandir(topfd) as scandir_it:
         ^^^^^^^^^^^^^^^^^
NotADirectoryError: [Errno 20] Not a directory: 'data/.DS_Store'

Are there quick fixes for that? It seems like a trivial thing to do.

Edit:

Forgot to add that the command is the following one, ran directly from the notebook cell:

!crunch setup --notebook --size default broad-1 voyage --token <token>

[Caceresenzo]

Thanks for the report, I think I know how to fix it. In the meantime, please remove the .DS_Store from the data/ directory yourself.

MacOS creates them automatically and it is hard to handle them properly.

[vadimnazarov]

Thank you for the prompt response!

[Caceresenzo]

I released a fix, please upgrade the cli:

pip install --upgrade crunch-cli