search

datalad / datalad-fuse

DataLad extension to provide FUSE file system access
Other
1 stars 4 forks source link

"interesting" PytestWarning warnings start to appearing #60

Closed yarikoptic closed 2 years ago

yarikoptic commented 2 years ago

Troubleshooting #56 in a loop. 1st run failed (#59), 2-3rd had "[WARNING] Destroying fsspecs and collection of 30 fhs " warning, and on 4th run I started to receive "interesting" warnings 

$> for s in {1..100}; do echo $s; date; python -m pytest -vv -k 'test_fuse' --libfuse datalad_fuse; done
...
4
Tue Feb  8 12:25:43 PM EST 2022
================================================================================================ test session starts ================================================================================================
platform linux -- Python 3.9.10, pytest-6.2.5, py-1.10.0, pluggy-1.0.0 -- /home/yoh/proj/datalad/datalad-fuse/venvs/dev3/bin/python
cachedir: .pytest_cache
rootdir: /home/yoh/proj/datalad/datalad-fuse, configfile: tox.ini
plugins: cov-2.12.1
collected 83 items / 61 deselected / 22 selected                                                                                                                                                                    

datalad_fuse/tests/test_fuse.py::test_fuse[remote-False] PASSED                                                                                                                                               [  4%]
datalad_fuse/tests/test_fuse.py::test_fuse[remote-True] PASSED                                                                                                                                                [  9%]
datalad_fuse/tests/test_fuse.py::test_fuse[local-False] PASSED                                                                                                                                                [ 13%]
datalad_fuse/tests/test_fuse.py::test_fuse[local-True] PASSED                                                                                                                                                 [ 18%]
datalad_fuse/tests/test_fuse.py::test_fuse[cloned-False] PASSED                                                                                                                                               [ 22%]
datalad_fuse/tests/test_fuse.py::test_fuse[cloned-True] PASSED                                                                                                                                                [ 27%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-False-None] PASSED                                                                                                                               [ 31%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-False-recursive] PASSED                                                                                                                          [ 36%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-False-visited] PASSED                                                                                                                            [ 40%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-True-None] PASSED                                                                                                                                [ 45%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-True-recursive] PASSED                                                                                                                           [ 50%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[remote-True-visited] PASSED                                                                                                                             [ 54%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-False-None] PASSED                                                                                                                                [ 59%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-False-recursive] PASSED                                                                                                                           [ 63%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-False-visited] PASSED                                                                                                                             [ 68%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-True-None] PASSED                                                                                                                                 [ 72%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-True-recursive] PASSED                                                                                                                            [ 77%]
datalad_fuse/tests/test_fuse.py::test_fuse_subdataset[local-True-visited] PASSED                                                                                                                              [ 81%]
datalad_fuse/tests/test_fuse.py::test_fuse_transparent_hash_object PASSED                                                                                                                                     [ 86%]
datalad_fuse/tests/test_fuse.py::test_fuse_transparent_hash_object_subdataset PASSED                                                                                                                          [ 90%]
datalad_fuse/tests/test_fuse.py::test_fuse_lock PASSED                                                                                                                                                        [ 95%]
datalad_fuse/tests/test_fuse.py::test_fuse_git_status PASSED                                                                                                                                                  [100%]

---------- coverage: platform linux, python 3.9.10-final-0 -----------
Name                                     Stmts   Miss  Cover   Missing
----------------------------------------------------------------------
datalad_fuse/__init__.py                    27      2    93%   103-109
datalad_fuse/consts.py                       1      0   100%
datalad_fuse/fsspec.py                     161    111    31%   34, 50-51, 55-85, 90-140, 149-186, 199, 202-204, 210-219, 222-230, 239-243, 246-248, 251-252, 256, 262
datalad_fuse/fsspec_cache_clear.py          22      0   100%
datalad_fuse/fsspec_head.py                 36     14    61%   72-85, 89
datalad_fuse/fuse_.py                      250    155    38%   42-45, 72-73, 82-83, 96, 113-173, 176-207, 210-222, 225-233, 236-245, 248-262, 265-266, 280-283, 286-292, 309, 313-316, 320-323, 335, 339-340, 344-345, 348-351, 355-357, 361, 364, 370-389
datalad_fuse/tests/__init__.py               0      0   100%
datalad_fuse/tests/conftest.py             102     11    89%   38-41, 64-67, 87-88, 92
datalad_fuse/tests/test_fsspec_head.py      48     28    42%   14, 18-19, 30-31, 41-42, 52-53, 63-64, 74-75, 86-87, 97-98, 108-109, 119-120, 130-136
datalad_fuse/tests/test_fuse.py            112      0   100%
datalad_fuse/tests/test_util.py             12      2    83%   56, 71
datalad_fuse/utils.py                       24     17    29%   12-37
----------------------------------------------------------------------
TOTAL                                      795    340    57%

=================================================================================== 22 passed, 61 deselected in 189.93s (0:03:09) ===================================================================================
fusefs(ok): /home/yoh/.tmp/pytest-of-yoh/pytest-103/test_fuse_git_status0/mount
[WARNING] Destroying fsspecs and collection of 30 fhs 
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-e7d547d7-ad4d-4383-9537-344800edeb27/test_fuse_remote_False_0
<class 'OSError'>: [Errno 22] Invalid argument: 'test_fuse_remote_False_0'
  warnings.warn(
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-e7d547d7-ad4d-4383-9537-344800edeb27
<class 'OSError'>: [Errno 39] Directory not empty: '/home/yoh/.tmp/pytest-of-yoh/garbage-e7d547d7-ad4d-4383-9537-344800edeb27'
  warnings.warn(
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-15af05d6-4bd3-4ac9-9d36-12c8cb25dbc7/test_fuse_remote_False_0
<class 'OSError'>: [Errno 22] Invalid argument: 'test_fuse_remote_False_0'
  warnings.warn(
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-15af05d6-4bd3-4ac9-9d36-12c8cb25dbc7
<class 'OSError'>: [Errno 39] Directory not empty: '/home/yoh/.tmp/pytest-of-yoh/garbage-15af05d6-4bd3-4ac9-9d36-12c8cb25dbc7'
  warnings.warn(
python -m pytest -vv -k 'test_fuse' --libfuse datalad_fuse  104.08s user 49.98s system 80% cpu 3:11.81 total

and then all subsequent runs do not have "[WARNING] Destroying fsspecs and collection of 30 fhs " and have fewer warnings:

=================================================================================== 22 passed, 61 deselected in 192.24s (0:03:12) ===================================================================================
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-5b2f6173-7c27-41ca-b356-f922d94a0a03/test_fuse_remote_False_0
<class 'OSError'>: [Errno 22] Invalid argument: 'test_fuse_remote_False_0'
  warnings.warn(
/home/yoh/proj/datalad/datalad-fuse/venvs/dev3/lib/python3.9/site-packages/_pytest/pathlib.py:80: PytestWarning: (rm_rf) error removing /home/yoh/.tmp/pytest-of-yoh/garbage-5b2f6173-7c27-41ca-b356-f922d94a0a03
<class 'OSError'>: [Errno 39] Directory not empty: '/home/yoh/.tmp/pytest-of-yoh/garbage-5b2f6173-7c27-41ca-b356-f922d94a0a03'
  warnings.warn(

any idea what those are about @jwodder ?

jwodder commented 2 years ago

@yarikoptic I believe those warnings happened because some datalad fusefs processes were still running, leaving their FUSE mounts mounted, which prevented pytest from cleaning up old temporary directories.

yarikoptic commented 2 years ago

may be... although I thought that there was only one process running (so shouldn't be multiple of those entries) but may be they did die off eventually so at the end we had only one left. ok, let's let it RiP for now.