search

harmslab / topiary

Python framework for doing ancestral sequence reconstruction
MIT License
31 stars 7 forks source link

OSError: [Errno 39] Directory not empty #27

Closed jkaczmarski closed 1 year ago

jkaczmarski commented 1 year ago

Not sure if this is a general issue or maybe something to do with my own setup/environment or dataset. But was getting the following error: 

  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper
    value = func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 305, in find_best_model
    out_list = threads.thread_manager(kwargs_list,
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/threads.py", line 120, in thread_manager
    results.append(fcn(**kwargs))
                   ^^^^^^^^^^^^^
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 135, in _model_thread_function
    shutil.rmtree(tmp_dir)
  File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 737, in rmtree
    onerror(os.rmdir, path, sys.exc_info())
  File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 735, in rmtree
    os.rmdir(path, dir_fd=dir_fd)
OSError: [Errno 39] Directory not empty: 'tmp_CWFOUXAPzJ'

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper
    value = func(*args, **kwargs)
            ^^^^^^^^^^^^^^^^^^^^^
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/pipeline/alignment_to_ancestors.py", line 322, in alignment_to_ancestors
    topiary.find_best_model(df,
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper
    raise WrappedFunctionException(err) from e
topiary._private.interface.WrappedFunctionException: 

Caught exception in function 'find_best_model'. Returning to starting
directory and cleaning up. Check error stack for cause of
this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 185, in wrap_function
    ret = fcn(**fcn_args.__dict__)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper
    raise WrappedFunctionException(err) from e
topiary._private.interface.WrappedFunctionException: 

Caught exception in function 'alignment_to_ancestors'. Returning to starting
directory and cleaning up. Check error stack for cause of
this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 26, in <module>
    main()
  File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 21, in main
    wrap_function(alignment_to_ancestors,
  File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 189, in wrap_function
    raise RuntimeError(err) from e
RuntimeError: 

Function alignment_to_ancestors raised an error.

To see command line help, run topiary-alignment-to-ancestors --help

As a temporary fix, I added an ignore_errors=True flag to line 135 of raxml/model.py and it now runs fine, although I am not sure if this will have any downstream consequences... (?):
shutil.rmtree(tmp_dir, ignore_errors=True)

Python version 3.11.0

harmsm commented 1 year ago

Hi Joe,

Thanks so much for the bug report.

I think adding “ignore_errors=True” should work fine. (You will likely just have a bunch of folders, one for each model that topiary tries).

I just tried to reproduce in python 3.11 on my Mac and did not get the error. I didn’t think shutil.rmtree ever cared about whether a directory is empty or not.

Did this happen once, or more than once? I wonder if it is a name collision, where two models on two threads are using the same temporary directory by happenstance? As one is being deleted, another writes to it, so rmtree fails? Seems farfetched. I’ll see what I can dig up.

Best,

Mike

On Jan 11, 2023, at 4:12 PM, Joe Kaczmarski @.***> wrote:

Not sure if this is a general issue or maybe something to do with my own setup/environment or dataset. But was getting the following error:

File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 305, in find_best_model out_list = threads.thread_manager(kwargs_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/threads.py", line 120, in thread_manager results.append(fcn(kwargs)) ^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 135, in _model_thread_function shutil.rmtree(tmp_dir) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 737, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 735, in rmtree os.rmdir(path, dir_fd=dir_fd) OSError: [Errno 39] Directory not empty: 'tmp_CWFOUXAPzJ'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/pipeline/alignment_to_ancestors.py", line 322, in alignment_to_ancestors topiary.find_best_model(df, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'find_best_model'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 185, in wrap_function ret = fcn(**fcn_args.dict) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'alignment_to_ancestors'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 26, in main() File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 21, in main wrap_function(alignment_to_ancestors, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 189, in wrap_function raise RuntimeError(err) from e RuntimeError:

Function alignment_to_ancestors raised an error.

To see command line help, run topiary-alignment-to-ancestors --help As a temporary fix, I added an ignore_errors=True flag to line 135 of raxml/model.py and it now runs fine, although I am not sure if this will have any downstream consequences... (?): shutil.rmtree(tmp_dir, ignore_errors=True)

Python version 3.11.0

— Reply to this email directly, view it on GitHub https://github.com/harmslab/topiary/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFZA6TLFKOAIEHDJNO4IS3WR5D5XANCNFSM6AAAAAATYUKRS4. You are receiving this because you are subscribed to this thread.

harmsm commented 1 year ago

I just thought about this some more. Another possibility—maybe more likely?—is that RAxML is still writing out its log file when shutil.rmtree gets called. Python deletes the directory contents, then goes to delete the directory with os.rmdir, but in the meantime RAxML has dumped more file content so the directory is no longer empty. os.rmdir then throws an error.

I noticed this odd behavior in a different context, where it seems a subprocess.run call returns as complete even though files are still being updated. I haven’t been able to find documentation for this—can a process return complete while requested file operations are still occurring?—but empirically it seems to happen.

If this is happening, I would predict the tmp_XXXXXXXX folder that topiary chokes on has nothing in it but a mangled log file. Since I haven’t been able to reproduce here, would you mind checking for that?

Thanks!

Mike

On Jan 11, 2023, at 4:40 PM, Mike Harms @.***> wrote:

Hi Joe,

Thanks so much for the bug report.

I think adding “ignore_errors=True” should work fine. (You will likely just have a bunch of folders, one for each model that topiary tries).

I just tried to reproduce in python 3.11 on my Mac and did not get the error. I didn’t think shutil.rmtree ever cared about whether a directory is empty or not.

Did this happen once, or more than once? I wonder if it is a name collision, where two models on two threads are using the same temporary directory by happenstance? As one is being deleted, another writes to it, so rmtree fails? Seems farfetched. I’ll see what I can dig up.

Best,

Mike

On Jan 11, 2023, at 4:12 PM, Joe Kaczmarski @.***> wrote:

Not sure if this is a general issue or maybe something to do with my own setup/environment or dataset. But was getting the following error:

File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 305, in find_best_model out_list = threads.thread_manager(kwargs_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/threads.py", line 120, in thread_manager results.append(fcn(kwargs)) ^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 135, in _model_thread_function shutil.rmtree(tmp_dir) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 737, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 735, in rmtree os.rmdir(path, dir_fd=dir_fd) OSError: [Errno 39] Directory not empty: 'tmp_CWFOUXAPzJ'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/pipeline/alignment_to_ancestors.py", line 322, in alignment_to_ancestors topiary.find_best_model(df, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'find_best_model'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 185, in wrap_function ret = fcn(**fcn_args.dict) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'alignment_to_ancestors'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 26, in main() File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 21, in main wrap_function(alignment_to_ancestors, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 189, in wrap_function raise RuntimeError(err) from e RuntimeError:

Function alignment_to_ancestors raised an error.

To see command line help, run topiary-alignment-to-ancestors --help As a temporary fix, I added an ignore_errors=True flag to line 135 of raxml/model.py and it now runs fine, although I am not sure if this will have any downstream consequences... (?): shutil.rmtree(tmp_dir, ignore_errors=True)

Python version 3.11.0

— Reply to this email directly, view it on GitHub https://github.com/harmslab/topiary/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFZA6TLFKOAIEHDJNO4IS3WR5D5XANCNFSM6AAAAAATYUKRS4. You are receiving this because you are subscribed to this thread.

jkaczmarski commented 1 year ago

Hi Mike,

Thanks for helping out with this (and for putting together topiary – it looks amazing and I can’t wait to test it out on a few upcoming projects!).

I think I have worked out the issue… Seems it is only a problem when I was trying to run from within a folder on a mounted drive (fuseblk), but runs fine without choking if run in home drive (ext4). I had come across a few others discussing similar issues who found it was due to issues with nfs file systems keeping tmp files in directories and shutil.rmtree failing, which made me think to check if same issue here and try on home drive (which I should have probably been doing anyway..). So all running fine for me now on home drive. I think it is possible to add some extra loops to check for nfs tmp files (e.g. https://github.com/easybuilders/easybuild-framework/pull/353/files/5474ed6be0606119128f7155aab02366bdd1878a#diff-0ca2799267ffd6a801a47e695fbe66c047a44775b616b26148100ec11e6ce5e5) in case users try and run on nfs drive, but maybe this is unnecessary.

See also:

Ps. Before I worked this out I did check tmp_XX directories that were choking, but they were all empty.

Cheers,

Joe

From: Mike Harms @.> Date: Thursday, 12 January 2023 at 12:51 pm To: harmslab/topiary @.> Cc: Joe Kaczmarski @.>, Author @.> Subject: Re: [harmslab/topiary] OSError: [Errno 39] Directory not empty (Issue #27) I just thought about this some more. Another possibility—maybe more likely?—is that RAxML is still writing out its log file when shutil.rmtree gets called. Python deletes the directory contents, then goes to delete the directory with os.rmdir, but in the meantime RAxML has dumped more file content so the directory is no longer empty. os.rmdir then throws an error.

I noticed this odd behavior in a different context, where it seems a subprocess.run call returns as complete even though files are still being updated. I haven’t been able to find documentation for this—can a process return complete while requested file operations are still occurring?—but empirically it seems to happen.

If this is happening, I would predict the tmp_XXXXXXXX folder that topiary chokes on has nothing in it but a mangled log file. Since I haven’t been able to reproduce here, would you mind checking for that?

Thanks!

Mike

On Jan 11, 2023, at 4:40 PM, Mike Harms @.***> wrote:

Hi Joe,

Thanks so much for the bug report.

I think adding “ignore_errors=True” should work fine. (You will likely just have a bunch of folders, one for each model that topiary tries).

I just tried to reproduce in python 3.11 on my Mac and did not get the error. I didn’t think shutil.rmtree ever cared about whether a directory is empty or not.

Did this happen once, or more than once? I wonder if it is a name collision, where two models on two threads are using the same temporary directory by happenstance? As one is being deleted, another writes to it, so rmtree fails? Seems farfetched. I’ll see what I can dig up.

Best,

Mike

On Jan 11, 2023, at 4:12 PM, Joe Kaczmarski @.***> wrote:

Not sure if this is a general issue or maybe something to do with my own setup/environment or dataset. But was getting the following error:

File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 305, in find_best_model out_list = threads.thread_manager(kwargs_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/threads.py", line 120, in thread_manager results.append(fcn(kwargs)) ^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 135, in _model_thread_function shutil.rmtree(tmp_dir) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 737, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 735, in rmtree os.rmdir(path, dir_fd=dir_fd) OSError: [Errno 39] Directory not empty: 'tmp_CWFOUXAPzJ'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/pipeline/alignment_to_ancestors.py", line 322, in alignment_to_ancestors topiary.find_best_model(df, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'find_best_model'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 185, in wrap_function ret = fcn(**fcn_args.dict) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'alignment_to_ancestors'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 26, in main() File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 21, in main wrap_function(alignment_to_ancestors, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 189, in wrap_function raise RuntimeError(err) from e RuntimeError:

Function alignment_to_ancestors raised an error.

To see command line help, run topiary-alignment-to-ancestors --help As a temporary fix, I added an ignore_errors=True flag to line 135 of raxml/model.py and it now runs fine, although I am not sure if this will have any downstream consequences... (?): shutil.rmtree(tmp_dir, ignore_errors=True)

Python version 3.11.0

— Reply to this email directly, view it on GitHub https://github.com/harmslab/topiary/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFZA6TLFKOAIEHDJNO4IS3WR5D5XANCNFSM6AAAAAATYUKRS4. You are receiving this because you are subscribed to this thread.

— Reply to this email directly, view it on GitHubhttps://github.com/harmslab/topiary/issues/27#issuecomment-1379709099, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF3LEG3S3JIHJAA2BOSV3D3WR5PQHANCNFSM6AAAAAATYUKRS4. You are receiving this because you authored the thread.Message ID: @.***>

harmsm commented 1 year ago

Hi Joe,

That makes sense; thanks for chasing it down! I think adding an NFS check is a good idea. If I’ve learned anything about writing software for high performance computing environments, it’s that you want to assume as little as possible about things like file systems!

I’ve got a few tweaks to push; I’ll add this to the stack. Thanks for the input. I’ll keep this open until I push the fix.

Best,

Mike

On Jan 11, 2023, at 6:51 PM, Joe Kaczmarski @.***> wrote:

Hi Mike,

Thanks for helping out with this (and for putting together topiary – it looks amazing and I can’t wait to test it out on a few upcoming projects!).

I think I have worked out the issue… Seems it is only a problem when I was trying to run from within a folder on a mounted drive (fuseblk), but runs fine without choking if run in home drive (ext4). I had come across a few others discussing similar issues who found it was due to issues with nfs file systems keeping tmp files in directories and shutil.rmtree failing, which made me think to check if same issue here and try on home drive (which I should have probably been doing anyway..). So all running fine for me now on home drive. I think it is possible to add some extra loops to check for nfs tmp files (e.g. https://github.com/easybuilders/easybuild-framework/pull/353/files/5474ed6be0606119128f7155aab02366bdd1878a#diff-0ca2799267ffd6a801a47e695fbe66c047a44775b616b26148100ec11e6ce5e5) in case users try and run on nfs drive, but maybe this is unnecessary.

See also:

Ps. Before I worked this out I did check tmp_XX directories that were choking, but they were all empty.

Cheers,

Joe

From: Mike Harms @.> Date: Thursday, 12 January 2023 at 12:51 pm To: harmslab/topiary @.> Cc: Joe Kaczmarski @.>, Author @.> Subject: Re: [harmslab/topiary] OSError: [Errno 39] Directory not empty (Issue #27) I just thought about this some more. Another possibility—maybe more likely?—is that RAxML is still writing out its log file when shutil.rmtree gets called. Python deletes the directory contents, then goes to delete the directory with os.rmdir, but in the meantime RAxML has dumped more file content so the directory is no longer empty. os.rmdir then throws an error.

I noticed this odd behavior in a different context, where it seems a subprocess.run call returns as complete even though files are still being updated. I haven’t been able to find documentation for this—can a process return complete while requested file operations are still occurring?—but empirically it seems to happen.

If this is happening, I would predict the tmp_XXXXXXXX folder that topiary chokes on has nothing in it but a mangled log file. Since I haven’t been able to reproduce here, would you mind checking for that?

Thanks!

Mike

On Jan 11, 2023, at 4:40 PM, Mike Harms @.***> wrote:

Hi Joe,

Thanks so much for the bug report.

I think adding “ignore_errors=True” should work fine. (You will likely just have a bunch of folders, one for each model that topiary tries).

I just tried to reproduce in python 3.11 on my Mac and did not get the error. I didn’t think shutil.rmtree ever cared about whether a directory is empty or not.

Did this happen once, or more than once? I wonder if it is a name collision, where two models on two threads are using the same temporary directory by happenstance? As one is being deleted, another writes to it, so rmtree fails? Seems farfetched. I’ll see what I can dig up.

Best,

Mike

On Jan 11, 2023, at 4:12 PM, Joe Kaczmarski @.***> wrote:

Not sure if this is a general issue or maybe something to do with my own setup/environment or dataset. But was getting the following error:

File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 305, in find_best_model out_list = threads.thread_manager(kwargs_list, ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/threads.py", line 120, in thread_manager results.append(fcn(kwargs)) ^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/raxml/model.py", line 135, in _model_thread_function shutil.rmtree(tmp_dir) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 737, in rmtree onerror(os.rmdir, path, sys.exc_info()) File "/home/joek/.conda/envs/topiary/lib/python3.11/shutil.py", line 735, in rmtree os.rmdir(path, dir_fd=dir_fd) OSError: [Errno 39] Directory not empty: 'tmp_CWFOUXAPzJ'

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 32, in wrapper value = func(*args, **kwargs) ^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/pipeline/alignment_to_ancestors.py", line 322, in alignment_to_ancestors topiary.find_best_model(df, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'find_best_model'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 185, in wrap_function ret = fcn(**fcn_args.dict) ^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/interface.py", line 38, in wrapper raise WrappedFunctionException(err) from e topiary._private.interface.WrappedFunctionException:

Caught exception in function 'alignment_to_ancestors'. Returning to starting directory and cleaning up. Check error stack for cause of this error.

The above exception was the direct cause of the following exception:

Traceback (most recent call last): File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 26, in main() File "/home/joek/.conda/envs/topiary/bin/topiary-alignment-to-ancestors", line 21, in main wrap_function(alignment_to_ancestors, File "/home/joek/.conda/envs/topiary/lib/python3.11/site-packages/topiary/_private/wrap.py", line 189, in wrap_function raise RuntimeError(err) from e RuntimeError:

Function alignment_to_ancestors raised an error.

To see command line help, run topiary-alignment-to-ancestors --help As a temporary fix, I added an ignore_errors=True flag to line 135 of raxml/model.py and it now runs fine, although I am not sure if this will have any downstream consequences... (?): shutil.rmtree(tmp_dir, ignore_errors=True)

Python version 3.11.0

— Reply to this email directly, view it on GitHub https://github.com/harmslab/topiary/issues/27, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFZA6TLFKOAIEHDJNO4IS3WR5D5XANCNFSM6AAAAAATYUKRS4. You are receiving this because you are subscribed to this thread.

— Reply to this email directly, view it on GitHubhttps://github.com/harmslab/topiary/issues/27#issuecomment-1379709099, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AF3LEG3S3JIHJAA2BOSV3D3WR5PQHANCNFSM6AAAAAATYUKRS4. You are receiving this because you authored the thread.Message ID: @.***> — Reply to this email directly, view it on GitHub https://github.com/harmslab/topiary/issues/27#issuecomment-1379745124, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABFZA6VFBKDPE2TPA3D6R4DWR5WUZANCNFSM6AAAAAATYUKRS4. You are receiving this because you commented.

harmsm commented 1 year ago

I've written a wrapper for shutil.rmtree that should work for an NFS file system. I don't have a convenient NFS system to test this on. @jkaczmarski: Any chance you could try it out and see if it resolves your issue? I have an open PR you could clone and try (https://github.com/harmslab/topiary/pull/29). If you don't have the bandwidth, no worries.

jkaczmarski commented 1 year ago

hi @harmsm. Tested this open PR on the setup that was originally causing issues (i.e. external drive) and it ran without any issues, so seems like it is all good now - thanks!

harmsm commented 1 year ago

Fantastic! Thanks so much for checking it. I'm merging PR.