Permission denied errors in `self.clean_tmp()`

jo-mueller commented 2 months ago

Describe the bug A collaborator of mine is receiving a permission denied error simply when opening the Annotator plugin. She is working on a workstation where she is having limited permissions (i.e., no sudo access).

This is the full traceback of the error message:

```python Traceback (most recent call last): File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/menus/plugins_menu.py", line 105, in _add_toggle_widget self._win.add_plugin_dock_widget(*key) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 897, in add_plugin_dock_widget wdg = _instantiate_dock_widget( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 1551, in _instantiate_dock_widget return wdg_cls(**kwargs) ^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/annotator_module.py", line 70, in __init__ self.minimal_contour_widget = MinimalContourWidget(viewer, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/minimal_contour_widget.py", line 102, in __init__ self.feature_manager = FeatureManager(viewer) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 24, in __init__ self.clean_tmp() File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 136, in clean_tmp shutil.rmtree(fold) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 759, in rmtree _rmtree_safe_fd(stack, onexc) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd onexc(func, path, err) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 674, in _rmtree_safe_fd topfd = os.open(name, os.O_RDONLY | os.O_NONBLOCK, dir_fd=dirfd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/tmp/tmpl6dib9_u_nd_annotator' Traceback (most recent call last): File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/menus/plugins_menu.py", line 105, in _add_toggle_widget self._win.add_plugin_dock_widget(*key) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 897, in add_plugin_dock_widget wdg = _instantiate_dock_widget( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 1551, in _instantiate_dock_widget return wdg_cls(**kwargs) ^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/annotator_module.py", line 70, in __init__ self.minimal_contour_widget = MinimalContourWidget(viewer, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/minimal_contour_widget.py", line 102, in __init__ self.feature_manager = FeatureManager(viewer) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 24, in __init__ self.clean_tmp() File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 136, in clean_tmp shutil.rmtree(fold) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 759, in rmtree _rmtree_safe_fd(stack, onexc) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd onexc(func, path, err) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 674, in _rmtree_safe_fd topfd = os.open(name, os.O_RDONLY | os.O_NONBLOCK, dir_fd=dirfd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/tmp/tmpl6dib9_u_nd_annotator' Traceback (most recent call last): File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/menus/plugins_menu.py", line 105, in _add_toggle_widget self._win.add_plugin_dock_widget(*key) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 897, in add_plugin_dock_widget wdg = _instantiate_dock_widget( ^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari/_qt/qt_main_window.py", line 1551, in _instantiate_dock_widget return wdg_cls(**kwargs) ^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/annotator_module.py", line 70, in __init__ self.minimal_contour_widget = MinimalContourWidget(viewer, self) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/_widgets/minimal_contour_widget.py", line 102, in __init__ self.feature_manager = FeatureManager(viewer) ^^^^^^^^^^^^^^^^^^^^^^ File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 24, in __init__ self.clean_tmp() File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/site-packages/napari_nd_annotator/minimal_contour/feature_manager.py", line 136, in clean_tmp shutil.rmtree(fold) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 759, in rmtree _rmtree_safe_fd(stack, onexc) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 703, in _rmtree_safe_fd onexc(func, path, err) File "/home/pol_dye/saja957d/miniconda3/envs/gt_env/lib/python3.12/shutil.py", line 674, in _rmtree_safe_fd topfd = os.open(name, os.O_RDONLY | os.O_NONBLOCK, dir_fd=dirfd) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ PermissionError: [Errno 13] Permission denied: '/tmp/tmpl6dib9_u_nd_annotator' ```

It seems like the tempfile package is trying to write something into the root tmpfile rather than into the users workspace, which would be a better go-to.

To Reproduce Steps to reproduce the behavior:

Open plugin on Linux system as non-sudo user

Expected behavior Plugin widget should open without error

napari info Copy information from Help -> napari Info.

Suggested fix

Something to set the /tmp file to the user's homedir should probably do the trick, e.g.:

os.environ['TMPDIR'] = os.path.expanduser('~/tmp')

bauerdavid commented 1 month ago

Hey @jo-mueller, thanks for the report! I will check the problem asap. But this bug also shows, that the plugin needs better exception handling, as problems with persistence shouldn't break the whole plugin... By the way, recently I realized that the magicgui package can store widget values on disk, so I'll check whether it would be a feasible solution. Thanks again!

jo-mueller commented 1 month ago

Hi @bauerdavid ,

thanks for the superfast reply :) Maybe another way to avoid this would be to write the plugin as a Qt widget from the get-go, rather than going the magicgui route? It's probably something for the next major version, but it would solve these persistence error problems and would maybe also make some of the "overhead layers" in the viewer unnecessary? (I take it that these are used to pass data between instances of the annotator plugin?)

Things like the Qt designer would certainly make at least the design of the UI relatively easy on that part.

Edit: Besides the design considerations, simply commenting out the self.clean_tmp() made the plugin functional again. But I don't know the implications for ither functionality...

bauerdavid commented 1 month ago

Okay, I was completely wrong :D This is not about widget persistence, but the temporarily stored image features used by minimal contour. My guess is that the temp files are not closed properly, so they cannot be removed.

bauerdavid commented 1 month ago

The temp files are supposed to be removed by clean() when Python exits (achieved by atexit). The self.clean_tmp function runs on startup to remove any leftover temp files in case of something going wrong with the cleanup. This was done to make sure that the disk will never be flooded with leftover, as on Windows I've seen some cases where napari crashed and these files weren't removed.

I've looked into the code of the clean() method, and I see the two empty except blocks with no Exception type provided, which can be very dangerous.

So my theory is:

when clean() runs, there's an exception, but is not visible due to the empty except block
whatever prevents clean() from removing the temp files, also prevents clean_tmp() from doing so

bauerdavid commented 1 month ago

I created a new branch and changed the handling of exceptions (from completely ignoring to printing the stack trace). Could you try it on the problematic machine? What I hope would happen is 1) the plugin won't break on exception and 2) We'll see what the problem is. Unfortunately there's some C/C++ and Cython code in the repository, if you have issues with installing, please let me know.

jo-mueller commented 1 month ago

Hi @bauerdavid ,

sorry for not replying in such a long time. I just tried from your branch and I still receive an error but the traceback does seem to be a bit more verbose :)

Full traceback:

```python --------------------------------------------------------------------------- PermissionError Traceback (most recent call last) File ~/miniforge3/envs/some_env/lib/python3.9/site-packages/napari/_qt/menus/plugins_menu.py:105, in PluginsMenu._add_plugin_actions.._add_toggle_widget(key=('napari-nD-annotator', 'Annotation Toolbox'), hook_type='dock') 102 return 104 if hook_type == 'dock': --> 105 self._win.add_plugin_dock_widget(*key) key = ('napari-nD-annotator', 'Annotation Toolbox') self._win = self = 106 else: 107 self._win._add_plugin_function_widget(*key) File ~/miniforge3/envs/some_env/lib/python3.9/site-packages/napari/_qt/qt_main_window.py:897, in Window.add_plugin_dock_widget(self=, plugin_name='napari-nD-annotator', widget_name='Annotation Toolbox', tabify=False) 894 wdg = wdg._magic_widget 895 return dock_widget, wdg --> 897 wdg = _instantiate_dock_widget( Widget = self = 898 Widget, cast('Viewer', self._qt_viewer.viewer) 899 ) 901 # Add dock widget 902 dock_kwargs.pop('name', None) File ~/miniforge3/envs/some_env/lib/python3.9/site-packages/napari/_qt/qt_main_window.py:1551, in _instantiate_dock_widget(wdg_cls=, viewer=Viewer(camera=Camera(center=(0.0, 127.5, 127.5),...ouse_drag_gen={}, _mouse_wheel_gen={}, keymap={})) 1546 break 1547 # cannot look for param.kind == param.VAR_KEYWORD because 1548 # QWidget allows **kwargs but errs on unknown keyword arguments 1549 1550 # instantiate the widget -> 1551 return wdg_cls(**kwargs) kwargs = {'viewer': Viewer(camera=Camera(center=(0.0, 127.5, 127.5), zoom=4.787109375, angles=(0.0, 0.0, 90.0), perspective=0.0, mouse_pan=True, mouse_zoom=True), cursor=Cursor(position=(29.0, 204.63067139229273, -244.5655622934073), scaled=True, size=10, style=), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 60.0, 1.0), (0.0, 256.0, 1.0), (0.0, 256.0, 1.0)), current_step=(29, 127, 127), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[, , ], help='use <1> for activate the label eraser, use <2> for activate the paint brush, use <3> for activate the fill bucket, use <4> for pick mode', status='', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=False, mouse_move_callbacks=[], mouse_drag_callbacks=[], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={})} wdg_cls = File ~/Github/napari-nD-annotator/src/napari_nd_annotator/_widgets/annotator_module.py:70, in AnnotatorWidget.__init__(self=, viewer=Viewer(camera=Camera(center=(0.0, 127.5, 127.5),...ouse_drag_gen={}, _mouse_wheel_gen={}, keymap={})) 67 self.interpolation_widget = InterpolationWidget(viewer, self) 68 tabs_widget.addTab(self.interpolation_widget, "Interpolation") ---> 70 self.minimal_contour_widget = MinimalContourWidget(viewer, self) viewer = Viewer(camera=Camera(center=(0.0, 127.5, 127.5), zoom=4.787109375, angles=(0.0, 0.0, 90.0), perspective=0.0, mouse_pan=True, mouse_zoom=True), cursor=Cursor(position=(29.0, 204.63067139229273, -244.5655622934073), scaled=True, size=10, style=), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 60.0, 1.0), (0.0, 256.0, 1.0), (0.0, 256.0, 1.0)), current_step=(29, 127, 127), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[, , ], help='use <1> for activate the label eraser, use <2> for activate the paint brush, use <3> for activate the fill bucket, use <4> for pick mode', status='', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=False, mouse_move_callbacks=[], mouse_drag_callbacks=[], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={}) self = MinimalContourWidget = 71 tabs_widget.addTab(self.minimal_contour_widget, "Minimal Contour") 73 if MinimalSurfaceWidget is not None: File ~/Github/napari-nD-annotator/src/napari_nd_annotator/_widgets/minimal_contour_widget.py:102, in MinimalContourWidget.__init__(self=, viewer=Viewer(camera=Camera(center=(0.0, 127.5, 127.5),...ouse_drag_gen={}, _mouse_wheel_gen={}, keymap={}), parent=) 100 self.feature_inverted = False 101 self.prev_timer_id = None --> 102 self.feature_manager = FeatureManager(viewer) viewer = Viewer(camera=Camera(center=(0.0, 127.5, 127.5), zoom=4.787109375, angles=(0.0, 0.0, 90.0), perspective=0.0, mouse_pan=True, mouse_zoom=True), cursor=Cursor(position=(29.0, 204.63067139229273, -244.5655622934073), scaled=True, size=10, style=), dims=Dims(ndim=3, ndisplay=2, last_used=0, range=((0.0, 60.0, 1.0), (0.0, 256.0, 1.0), (0.0, 256.0, 1.0)), current_step=(29, 127, 127), order=(0, 1, 2), axis_labels=('0', '1', '2')), grid=GridCanvas(stride=1, shape=(-1, -1), enabled=False), layers=[, , ], help='use <1> for activate the label eraser, use <2> for activate the paint brush, use <3> for activate the fill bucket, use <4> for pick mode', status='', tooltip=Tooltip(visible=False, text=''), theme='dark', title='napari', mouse_over_canvas=False, mouse_move_callbacks=[], mouse_drag_callbacks=[], mouse_double_click_callbacks=[], mouse_wheel_callbacks=[], _persisted_mouse_event={}, _mouse_drag_gen={}, _mouse_wheel_gen={}, keymap={}) self = 104 # Ctrl+Z handling 105 self.change_idx = dict() File ~/Github/napari-nD-annotator/src/napari_nd_annotator/minimal_contour/feature_manager.py:26, in FeatureManager.__init__(self=, viewer=Viewer(camera=Camera(center=(0.0, 127.5, 127.5),...ouse_drag_gen={}, _mouse_wheel_gen={}, keymap={})) 24 self.memmaps: list[Optional[Union[np.ndarray, np.memmap]]] = [None, None] 25 self.slices_calculated = dict() ---> 26 self.clean_tmp() self = 27 self.temp_folder = tempfile.mkdtemp(suffix=TEMP_SUFFIX) 28 # map layers to file prefix File ~/Github/napari-nD-annotator/src/napari_nd_annotator/minimal_contour/feature_manager.py:133, in FeatureManager.clean_tmp(self=) 131 temp_folders = glob.glob(os.path.join(temp_dir, "%s*%s" % (tempfile.gettempprefix(), TEMP_SUFFIX))) 132 for fold in temp_folders: --> 133 shutil.rmtree(fold) fold = '/tmp/tmpolmrems8_nd_annotator' File ~/miniforge3/envs/some_env/lib/python3.9/shutil.py:730, in rmtree(path='/tmp/tmpolmrems8_nd_annotator', ignore_errors=False, onerror=.onerror>) 728 fd_closed = False 729 except Exception: --> 730 onerror(os.open, path, sys.exc_info()) path = '/tmp/tmpolmrems8_nd_annotator' 731 return 732 try: File ~/miniforge3/envs/some_env/lib/python3.9/shutil.py:727, in rmtree(path='/tmp/tmpolmrems8_nd_annotator', ignore_errors=False, onerror=.onerror>) 725 return 726 try: --> 727 fd = os.open(path, os.O_RDONLY) path = '/tmp/tmpolmrems8_nd_annotator' os.O_RDONLY = 0 728 fd_closed = False 729 except Exception: PermissionError: [Errno 13] Permission denied: '/tmp/tmpolmrems8_nd_annotator' ```

bauerdavid commented 1 month ago

What puzzles me is that the script had permission to create the folder, but afterwards it doesn't have permission to delete it. Could you check if the user has write access, and also whether you can delete the folder/its contents manually? I started to experiment a bit, and what I found is that I'm actually able to remove the files the memmaps are using manually even when the memmaps are still in use. This means that removing currently accessed files is not the problem. I'm a bit stuck on what is causing this, I cannot reproduce it on my machine...

jo-mueller commented 1 month ago

I just tried to forcefully remove it from the command line and got this:

(pytorch) johamuel@biapol-ws1:/tmp$ rm -rf tmp9zlwxwjq_nd_annotator/
rm: cannot remove 'tmp9zlwxwjq_nd_annotator/': Operation not permitted

I had a look at the permissions of these temporary files and this is more enlightening, I think:

drwx------  2 saja957d pol_dye      4096 Jul  8 16:06 tmp9zlwxwjq_nd_annotator
drwx------  2 saja957d pol_dye      4096 Jul  8 13:56 tmpolmrems8_nd_annotator

There are somehow two temporary folders from the nd annotator that were created by another user. Is it possible that the nd-annotator tries to reuse these and then fails in removing them?

bauerdavid commented 1 month ago

yup, this should be it. Added code for ignoring PermissionErrors, check if it works.

bauerdavid commented 2 weeks ago

Checked it, the change does fix the problem.

bauerdavid / napari-nD-annotator

Permission denied errors in `self.clean_tmp()` #49