fsspec / filesystem_spec

A specification that python filesystems should adhere to.
BSD 3-Clause "New" or "Revised" License
949 stars 343 forks source link

[Bug]Reading process hangs when in debug mode #1614

Open forestbat opened 1 month ago

forestbat commented 1 month ago

I want to read data from my minio storage in Intellij IDEA like this:

import xarray as xr
import fsspec
# hdscc.MINIO_PARAM = {'endpoint_url': 'xxx', 'key': 'yyy', 'secret': 'zzz'}
test_path = 'simplecache::s3://era5-origin/era5/grib/single_levels_tp/2016/02/02/2016020200.grib'
time_ds = xr.open_dataset(fsspec.open_local(time_path, s3=hdscc.MINIO_PARAM, filecache=
                {'cache_storage': '/tmp/files'}), engine='cfgrib')

when in run mode, this code is normal, but when in debug mode, this code stucks in here:

# fsspec.asyn.sync#L89
while True:
        # this loops allows thread to get interrupted
        if event.wait(1):
            break
        if timeout is not None:
            timeout -= 1
            if timeout < 0:
                raise FSTimeoutError

So what happened in fsspec? How to solve it? Hope your reply.

martindurant commented 1 month ago

Sorry, I have no idea how IDEA might be calling the code; however, fsspec uses an event loop running in a thread other than the main one, so maybe you need to enable some setting to allow for that.