search

OGGM / oggm

Open Global Glacier Model
http://oggm.org
BSD 3-Clause "New" or "Revised" License
218 stars 105 forks source link

NotADirectoryError in get_test_dir on MacOS #1294

Open oberrauch opened 3 years ago

oberrauch commented 3 years ago

Deleting all directories inside the OGGM test directory (since #1064) can result in a error when trying to delete the DS_Store file on MacOS. Following the traceback:

Traceback (most recent call last):
  File "/Users/oberrauch/miniconda3/envs/oggm_env/lib/python3.6/site-packages/IPython/core/interactiveshell.py", line 3325, in run_code
    exec(code_obj, self.user_global_ns, self.user_ns)
  File "<ipython-input-4-528df78320dc>", line 1, in <module>
    get_test_dir()
  File "/Users/oberrauch/oggm-fork/oggm/tests/funcs.py", line 324, in get_test_dir
    shutil.rmtree(os.path.join(cfg.PATHS['test_dir'], d))
  File "/Users/oberrauch/miniconda3/envs/oggm_env/lib/python3.6/shutil.py", line 494, in rmtree
    return _rmtree_unsafe(path, onerror)
  File "/Users/oberrauch/miniconda3/envs/oggm_env/lib/python3.6/shutil.py", line 376, in _rmtree_unsafe
    onerror(os.listdir, path, sys.exc_info())
  File "/Users/oberrauch/miniconda3/envs/oggm_env/lib/python3.6/shutil.py", line 374, in _rmtree_unsafe
    names = os.listdir(path)
NotADirectoryError: [Errno 20] Not a directory: '/Users/oberrauch/OGGM/tests/.DS_Store'

Quick and dirty solution would be to set ignore_errors=True in shutil.rmtree. The more elaborate solution probably to supply a handler via onerror, since this could also happen for other files (even though I'm not sure how the should end up in the tests directory in the first place).

fmaussion commented 3 years ago

Thanks! That's weird. Does it happen often? I any case, any PR is welcome, no need to be too fancy here.

JordiBolibar commented 2 years ago

Following our discussion in Slack, here's a similar issue which might be related.

I’m getting an error when re-running workflow.init_glacier_directories() on a Linux SSH server. I haven’t encountered this behaviour in local, but when I run gdirs = workflow.init_glacier_directories(rgi_ids, from_prepro_level=3, prepro_border=10) for the second time I get:

ERROR: LoadError: PyError ($(Expr(:escape, :(ccall(#= /nethome/bolib001/.julia/packages/PyCall/BD546/src/pyfncall.jl:43 =# @pysym(:PyObject_Call), PyPtr, (PyPtr, PyPtr, PyPtr), o, pyargsptr, kw))))) <class 'OSError'>
OSError(39, 'Directory not empty')
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 529, in init_glacier_directories
    gdirs = execute_entity_task(gdir_from_prepro, entities,
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 191, in execute_entity_task
    out = [pc(gdir) for gdir in gdirs]
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 191, in <listcomp>
    out = [pc(gdir) for gdir in gdirs]
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 108, in __call__
    res = self._call_internal(func, arg, kwargs)
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 102, in _call_internal
    return call_func(gdir, **kwargs)
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/workflow.py", line 251, in gdir_from_prepro
    return oggm.GlacierDirectory(entity, from_tar=from_tar)
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/site-packages/oggm/utils/_workflow.py", line 2126, in __init__
    shutil.rmtree(_dir)
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/shutil.py", line 719, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/nethome/bolib001/.conda/envs/oggm_env/lib/python3.8/shutil.py", line 717, in rmtree
    os.rmdir(path)

If I close my session and open a new one I no longer get this error. So basically I can still run OGGM, but it strongly hampers development and debugging.

fmaussion commented 2 years ago

OK so looking into it these are two separate problems.

@JordiBolibar we have discussed your problem offline

@oberrauch the DS_Store problem really baffles me, but it seems to be a recurring issue with these files. I'll try to have a look when time permits, but in the meantime just delete this folder manually

fmaussion commented 2 years ago

I can find a lot of stackoverflow issues related to the infamous DS_Store files, but never in relation to rmtree... I'm quite confused...

oberrauch commented 2 years ago

Deleting it by hand works fine for me for now. As far as I understood it the problem is that .DS_Store is a file and hence os.listdir('.DS_Store') throws this error. Maybe using os.path.isdir() would be the cleanest option like

# If new ident, remove all other dirs so spare space
for d in os.listdir(cfg.PATHS['test_dir']):
    if d and d != s and os.path.isdir(d):
        shutil.rmtree(os.path.join(cfg.PATHS['test_dir'], d))
fmaussion commented 2 years ago

but this is code in python itself (rmtree), not something we can control

fmaussion commented 2 years ago

What I find fascinating is that it should be a bug report for Python itself I would say

fmaussion commented 2 years ago

Can you check if this still happens in more recent python versions? (not urgent)