Credential process fails on Windows

[riklopfer]

We have an aws credential_process which fsspec attempts to call via subprocess. https://github.com/fsspec/filesystem_spec/pull/658 changed it so that the event loop always has the same policy WindowsSelectorEventLoopPolicy which leads to the same NotImplementedError being thrown as reported in https://github.com/fsspec/sshfs/issues/29

@krwoodley fyi

Traceback (most recent call last):
  File "C:\GitHub\test_project\do_stuff.py", line 32, in <module>
    main()
  File "C:\GitHub\test_project\do_stuff.py", line 20, in main
    that()
  File "C:\GitHub\test_project\do_stuff.py", line 16, in that
    with fs.open(DOC_PATH) as f:
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\spec.py", line 1303, in open
    f = self._open(
  File "C:\GitHub\test_project\venv\lib\site-packages\s3fs\core.py", line 688, in _open
    return S3File(
  File "C:\GitHub\test_project\venv\lib\site-packages\s3fs\core.py", line 2182, in __init__
    super().__init__(
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\spec.py", line 1742, in __init__
    self.size = self.details["size"]
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\spec.py", line 1755, in details
    self._details = self.fs.info(self.path)
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\asyn.py", line 118, in wrapper
    return sync(self.loop, func, *args, **kwargs)
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\asyn.py", line 103, in sync
    raise return_result
  File "C:\GitHub\test_project\venv\lib\site-packages\fsspec\asyn.py", line 56, in _runner
    result[0] = await coro
  File "C:\GitHub\test_project\venv\lib\site-packages\s3fs\core.py", line 1374, in _info
    out = await self._call_s3(
  File "C:\GitHub\test_project\venv\lib\site-packages\s3fs\core.py", line 358, in _call_s3
    await self.set_session()
  File "C:\GitHub\test_project\venv\lib\site-packages\s3fs\core.py", line 544, in set_session
    self._s3 = await s3creator.__aenter__()
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\session.py", line 25, in __aenter__
    self._client = await self._coro
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\session.py", line 177, in _create_client
    credentials = await self.get_credentials()
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\session.py", line 90, in get_credentials
    self._credentials = await self._components.get_component(
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\credentials.py", line 961, in load_credentials
    creds = await provider.load()
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\credentials.py", line 493, in load
    creds_dict = await self._retrieve_credentials_using(credential_process)
  File "C:\GitHub\test_project\venv\lib\site-packages\aiobotocore\credentials.py", line 512, in _retrieve_credentials_using
    p = await self._popen(
  File "C:\Users\A6NRPZZ\AppData\Local\Programs\Python\Python310\lib\asyncio\subprocess.py", line 218, in create_subprocess_exec
    transport, protocol = await loop.subprocess_exec(
  File "C:\Users\A6NRPZZ\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 1681, in subprocess_exec
    transport = await self._make_subprocess_transport(
  File "C:\Users\A6NRPZZ\AppData\Local\Programs\Python\Python310\lib\asyncio\base_events.py", line 498, in _make_subprocess_transport
    raise NotImplementedError

Changing _selector_policy to yield makes things work.

@contextmanager
 def _selector_policy():
     yield
     # original_policy = asyncio.get_event_loop_policy()
     # try:
     #     if (
     #         sys.version_info >= (3, 8)
     #         and os.name == "nt"
     #         and hasattr(asyncio, "WindowsSelectorEventLoopPolicy")
     #     ):
     #         asyncio.set_event_loop_policy(asyncio.WindowsSelectorEventLoopPolicy())

     #     yield
     # finally:
     #     asyncio.set_event_loop_policy(original_policy)

[martindurant]

What is it actually that's not implemented? Some event loops can create subprocesses but others cannot?

[riklopfer]

Right. It appears that WindowsSelectorEventLoopPolicy does not allow subprocesses.

[martindurant]

OK, thanks for letting me know. We can think about offering a public way to configure this for the future.

[riklopfer]

Sounds good. Feel free to close this issue. This work-around is working just fine in the meantime.

@contextmanager
def _noop(*args, **kwargs):
    yield

fsspec.asyn._selector_policy = _noop