HTTPFileSystem tests occasionally stalling

[amotl]

Hi Martin,

while working on #560, I observed occasional hiccups on my workstation where the test suite would stall completely at some point when invoking pytest -vvvv -k test_http. Specifically, it freezes at test_isdir or test_policy_arg. However, every second or third attempt, it it will work though.

FWIW, I am using Homebrew's Python 3.9.4 on macOS 10.15.7.

With kind regards, Andreas.

_Originally posted by @amotl in https://github.com/intake/filesystem_spec/issues/560#issuecomment-826101128_

[amotl]

This is the output I am receiving when interrupting the test suite on one of the two spots outlined above.

STDOUT

$ pytest -vvvv -k test_http
===================================================================== test session starts ======================================================================
platform darwin -- Python 3.9.4, pytest-6.2.2, py-1.10.0, pluggy-0.13.1 -- /Users/amo/dev/earthobservations/filesystem_spec/.venv/bin/python
cachedir: .pytest_cache
rootdir: /Users/amo/dev/earthobservations/filesystem_spec
collected 467 items / 436 deselected / 8 skipped / 23 selected

fsspec/implementations/tests/test_http.py::test_list PASSED                                                                                              [  3%]
fsspec/implementations/tests/test_http.py::test_list_invalid_args PASSED                                                                                 [  6%]
fsspec/implementations/tests/test_http.py::test_list_cache PASSED                                                                                        [  9%]
fsspec/implementations/tests/test_http.py::test_list_cache_with_expiry_time_cached PASSED                                                                [ 12%]
fsspec/implementations/tests/test_http.py::test_list_cache_with_expiry_time_purged PASSED                                                                [ 16%]
fsspec/implementations/tests/test_http.py::test_list_cache_reuse PASSED                                                                                  [ 19%]
fsspec/implementations/tests/test_http.py::test_list_cache_with_max_paths PASSED                                                                         [ 22%]
fsspec/implementations/tests/test_http.py::test_list_cache_with_skip_instance_cache PASSED                                                               [ 25%]
fsspec/implementations/tests/test_http.py::test_isdir ^C

Traceback after CTRL+C

Error in atexit._run_exitfuncs:
Traceback (most recent call last):
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 656, in _exitfunc
    f()
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/weakref.py", line 580, in __call__
    return info.func(*info.args, **(info.kwargs or {}))
  File "/Users/amo/dev/earthobservations/filesystem_spec/fsspec/implementations/http.py", line 104, in close_session
    sync(loop, session.close)
  File "/Users/amo/dev/earthobservations/filesystem_spec/fsspec/asyn.py", line 51, in sync
    event.wait(timeout)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 574, in wait
    signaled = self._cond.wait(timeout)
  File "/usr/local/Cellar/python@3.9/3.9.4/Frameworks/Python.framework/Versions/3.9/lib/python3.9/threading.py", line 312, in wait
    waiter.acquire()
KeyboardInterrupt