search

SlideRuleEarth / h5coro

The HDF5 Cloud Optimized Read Only Python Package
BSD 3-Clause "New" or "Revised" License
22 stars 4 forks source link

Chunk decode error when running against local file #7

Open jpswinski opened 10 months ago

jpswinski commented 10 months ago

While running one of the benchmarks, there was an internal exception generated for decoding the chunk. This is intermittent.

jswinski@eragon [h5coro] (concurrent_list_group):~/meta/h5coro
$ python benchmarks/subset_atl06.py
Profiling s3fs... [1140]: 4.95 4.67
Profiling sliderule-h5p... [1140]: 0.98 1.02
Profiling h5coro... [1140]: 0.66 0.46
Profiling h5py-local... [1140]: 0.02 0.06
Profiling h5coro-local... Traceback (most recent call last):
  File "/home/jswinski/meta/h5coro/benchmarks/subset_atl06.py", line 121, in <module>
    result = subsetted_read(profiler, region, variable=args.variable06)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/meta/h5coro/benchmarks/subset_atl06.py", line 63, in subsetted_read
    geocoords = profiler.read(geodatasets)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/meta/h5coro/benchmarks/profiler.py", line 189, in read
    values = self.reader.read(datasets)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/meta/h5coro/benchmarks/profiler.py", line 177, in read
    return self.h5obj.readDatasets(datasets=datasets, block=True, enableAttributes=args.enableAttributes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5coro.py", line 127, in readDatasets
    return H5Promise(self, datasetTable, block, earlyExit=earlyExit, metaOnly=metaOnly, enableAttributes=enableAttributes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5promise.py", line 83, in __init__
    resultThread(self, futures)
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5promise.py", line 54, in resultThread
    h5dataset = future.result()
                ^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5promise.py", line 47, in datasetThread
    return H5Dataset(resourceObject, dataset, startRow, numRows, earlyExit=earlyExit, metaOnly=metaOnly, enableAttributes=enableAttributes)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5dataset.py", line 220, in __init__
    self.readBTreeV1(buffer, buffer_offset, dataset_level)
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5dataset.py", line 1857, in readBTreeV1
    buffer[buffer_index:buffer_index+chunk_bytes] = self.inflateChunk(self.dataChunkFilterBuffer)
                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/miniconda3/envs/h5coro/lib/python3.11/site-packages/h5coro/h5dataset.py", line 1918, in inflateChunk
    return zlib.decompress(input)
           ^^^^^^^^^^^^^^^^^^^^^^
zlib.error: Error -3 while decompressing data: incorrect header check
jpswinski commented 10 months ago

Additional error running in local mode:

Profiling h5coro-local... 1692204958.704175 WARNING [h5promise.py:   49] H5Coro encountered error reading /gt3l/geolocation/reference_photon_lat: invalid b-tree signature: 0x8179bd3a
1692204958.704323 WARNING [h5promise.py:   49] H5Coro encountered error reading /gt3l/geolocation/reference_photon_lon: key did not include a trailing zero: 10557365890972436331
Traceback (most recent call last):
  File "/home/jswinski/meta/h5coro/benchmarks/subset_atl03.py", line 160, in <module>
    result = subsetted_read(profiler, region, variable=args.variable03)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/jswinski/meta/h5coro/benchmarks/subset_atl03.py", line 76, in subsetted_read
    while index < len(lat_dataset):
                  ^^^^^^^^^^^^^^^^
TypeError: object of type 'NoneType' has no len()