search

HDFGroup / hsds

Cloud-native, service based access to HDF data
https://www.hdfgroup.org/solutions/hdf-kita/
Apache License 2.0
125 stars 52 forks source link

Listkeys #333

Closed jreadey closed 3 months ago

jreadey commented 3 months ago

Break up list_keys in fileClient with some asyncio.sleep()'s so that other tasks have a chance to run (c.f. issue #332). Also, reduce log verbosity.

assaron commented 3 months ago

@jreadey hope I'm not invading here, but did you mean the opposite condition here? https://github.com/HDFGroup/hsds/blob/c87fa68392644f5530761ceab1a855782e87b3f0/hsds/util/fileClient.py#L356 As I understand, currently it sleeps after each file, except for 1000th, but I guess the idea was to sleep only each 1000th file.

mattjala commented 3 months ago

I second @assaron's observation. Other than that, LGTM.

jreadey commented 3 months ago

Yes, silly mistake!

assaron commented 3 months ago

Sorry, I didn't specify, but there is second place with that problem as well: https://github.com/HDFGroup/hsds/blob/398df2752ed8db26bc99af827347d54f397474dd/hsds/util/fileClient.py#L370

jreadey commented 3 months ago

I'm consistent at least...