Open oruebel opened 2 months ago
In this case, does the s3_url point to an hdf5 file in the bucket? If so, that's not going to work... you need to point to the corresponding prepared json file. If it's a public dandi file, I can provide a formula for determining the proper url.
The parameters are:
''' lindi_remote_rfs_parameter_cases = dict(
BaseExample=dict(
s3_url="https://kerchunk.neurosift.org/dandi/dandisets/000939/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/zarr.json",
object_name="accelerometer",
slice_range=(slice(0, 30_000), slice(0, 3)),
),
) '''
I.e., for now I am just using one of the examples from the README to check that the benchmarks are working. We are generating LINDI files for the actual test datasets we are planning to use. I used the version of lindi from PiPy. Let me know in case I should use a different file or version of lindi.
It looks like the read itself is working, ie, we are getting back an array with the expected shape from the slicing. It looks like the error is occurring in the teardown afterwards when the subprocess is being cleaned up.
I am not able to reproduce. See below. I am on Ubuntu, and I installed nwb_benchmarks in a conda environment as in the docs. What OS are you using?
$ nwb_benchmarks run --bench time_remote_slicing.LindiFileReadRemoteReferenceFileSystemContinuousSliceBenchmark --debug
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python -c import sys; print(str(sys.version_info[0]) + "." + str(sys.version_info[1]))'
OUTPUT -------->
3.11
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git config init.defaultBranch'
OUTPUT -------->
main
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse 64cc427^{commit}'
OUTPUT -------->
64cc427d04423c5ce7ec93ca535d58e59cbd45f8
· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse main^{commit}'
OUTPUT -------->
64cc427d04423c5ce7ec93ca535d58e59cbd45f8
· Discovering benchmarks
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-parse main^{commit}'
OUTPUT -------->
64cc427d04423c5ce7ec93ca535d58e59cbd45f8
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py discover /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/tmpe540k9g8/result.json' in existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python /home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py discover /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/tmpe540k9g8/result.json'
· Running 1 total benchmarks (1 commits * 1 environments * 1 benchmarks)
[ 0.00%] · Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git name-rev --name-only --exclude=remotes/* --no-undefined 64cc427d04423c5ce7ec93ca535d58e59cbd45f8'
OUTPUT -------->
main
[ 0.00%] · For nwb_benchmarks commit 64cc427d <main>:
[ 0.00%] ·· Building for existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ·· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/git rev-list -n 1 --format=%at 64cc427d04423c5ce7ec93ca535d58e59cbd45f8'
OUTPUT -------->
commit 64cc427d04423c5ce7ec93ca535d58e59cbd45f8
1714497072
[ 0.00%] ·· Benchmarking existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ··· Running '/home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py run_server /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/asv-forkserver-9fj1bwct/socket' in existing-py_home_magland_miniconda3_envs_nwb_benchmarks_bin_python
[ 0.00%] ··· Running '/home/magland/miniconda3/envs/nwb_benchmarks/bin/python /home/magland/miniconda3/envs/nwb_benchmarks/lib/python3.11/site-packages/asv/benchmark.py run_server /home/magland/src/nwb_benchmarks/src/nwb_benchmarks/benchmarks /tmp/asv-forkserver-9fj1bwct/socket'
[100.00%] ··· ...ileSystemContinuousSliceBenchmark.time_slice ok
[100.00%] ··· ============================================================================================================= =============== ============================================ ============
s3_url object_name slice_range
------------------------------------------------------------------------------------------------------------- --------------- -------------------------------------------- ------------
https://kerchunk.neurosift.org/dandi/dandisets/000939/assets/11f512ba-5bcf-4230-a8cb-dc8d36db38cb/zarr.json accelerometer (slice(0, 30000, None), slice(0, 3, None)) 1.97±0.2ms
============================================================================================================= =============== ============================================ ============
tag @CodyCBakerPhD
EDIT: I also tried this on github codespaces and did not see any error
What OS are you using?
I am running on MacOS 12.5.1 (monterey) with a M1 Max chip
Also replicated on Windows 11, but I'll see if I can't make a minimal example and get it in a CI
Unfortunately I don't have access to mac or windows right now...
@oruebel I pushed a new version of lindi to pypi. While I don't understand what could be causing the problem, there is always a chance that something may have been accidentally fixed. :) So it's worth a try to run your tests again with version 0.3.2.
Thanks, I'll give it a try
This appears to no longer be an issue
@magland @rly we are working on the nwb-benchmarks and am encountering a strange issue https://github.com/NeurodataWithoutBorders/nwb_benchmarks/issues/57 Do you have an idea? It looks like this might be related to running LINDI in a subprocess. The particular benchmark loads data from a remote file by slicing into a LINDI dataset. Simply opening a file with LINDI (incl. read with PyNWB) seems to work. The relevant code in the benchmark suite is: https://github.com/NeurodataWithoutBorders/nwb_benchmarks/blob/main/src/nwb_benchmarks/benchmarks/time_remote_slicing.py#L139-L156 which also uses these helper functions to open the file https://github.com/NeurodataWithoutBorders/nwb_benchmarks/blob/fb464f1a7ab6a2c4b4ce91b798ac68b088cf514d/src/nwb_benchmarks/core/_streaming.py#L205-L226