search

BirdVox / birdvoxdetect

A pre-trained deep learning system for detecting bird flight calls in continuous recordings
MIT License
78 stars 15 forks source link

keras.models.load_model Heisenbug #35

Closed lostanlen closed 4 years ago

lostanlen commented 4 years ago

race condition in: https://github.com/BirdVox/birdvoxdetect/blob/master/birdvoxdetect/core.py#L106

2020-01-26 16:20:39.092832: E tensorflow/stream_executor/cuda/cuda_driver.cc:318] failed call to cuInit: CUDA_ERROR_NO_DEVICE: no CUDA-capable device is detected
2020-01-26 16:20:37.134087 Start.
Running BirdVoxDetect on BirdVox-300h.
WAV name: 2015-11-26_09-37-00_unit06.wav.

birdvoxdetect version: 0.2.0b1
h5py version: 2.10.0
librosa version: 0.7.0
numpy version: 1.16.4
pandas version: 0.25.1
resampy version: 0.2.2
scipy version: 1.4.1
soundfile version: 0.10.3
tensorflow version: 1.15.0

Traceback (most recent call last):
  File "/home/vl1019/birdvoxdetect/birdvoxdetect/core.py", line 107, in process_file
    detector_model_path, custom_objects=custom_objects)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 143, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 188, in load_model_from_hdf5
    optimizer_weight_values = load_optimizer_weights_from_hdf5_group(f)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 601, in load_optimizer_weights_from_hdf5_group
    return [weights_group[weight_name] for weight_name in optimizer_weight_names]
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 601, in <listcomp>
    return [weights_group[weight_name] for weight_name in optimizer_weight_names]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/h5py/_hl/group.py", line 264, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (file read failed: time = Sun Jan 26 16:20:39 2020\n, filename = '/home/vl1019/birdvoxdetect/birdvoxdetect/models/birdvoxdetect-v03_trial-12_network_epoch-068.h5', file descriptor = 5, errno = 5, error message = 'Input/output error', buf = 0x563cf06ffd90, total read size = 704, bytes this sub-read = 704, bytes actually read = 18446744073709551615, offset = 5515120)"

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "../../../src/011_birdvoxdetect-300h.py", line 61, in <module>
    export_confidence=True)
  File "/home/vl1019/birdvoxdetect/birdvoxdetect/core.py", line 113, in process_file
    raise BirdVoxDetectError(exc_formatted_str)
birdvoxdetect.birdvoxdetect_exceptions.BirdVoxDetectError: Could not open model "/home/vl1019/birdvoxdetect/birdvoxdetect/models/birdvoxdetect-v03_trial-12_network_epoch-068.h5":
Traceback (most recent call last):
  File "/home/vl1019/birdvoxdetect/birdvoxdetect/core.py", line 107, in process_file
    detector_model_path, custom_objects=custom_objects)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/save.py", line 143, in load_model
    return hdf5_format.load_model_from_hdf5(filepath, custom_objects, compile)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 188, in load_model_from_hdf5
    optimizer_weight_values = load_optimizer_weights_from_hdf5_group(f)
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 601, in load_optimizer_weights_from_hdf5_group
    return [weights_group[weight_name] for weight_name in optimizer_weight_names]
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/tensorflow_core/python/keras/saving/hdf5_format.py", line 601, in <listcomp>
    return [weights_group[weight_name] for weight_name in optimizer_weight_names]
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "/home/vl1019/miniconda3/envs/bvdev/lib/python3.6/site-packages/h5py/_hl/group.py", line 264, in __getitem__
    oid = h5o.open(self.id, self._e(name), lapl=self._lapl)
  File "h5py/_objects.pyx", line 54, in h5py._objects.with_phil.wrapper
  File "h5py/_objects.pyx", line 55, in h5py._objects.with_phil.wrapper
  File "h5py/h5o.pyx", line 190, in h5py.h5o.open
KeyError: "Unable to open object (file read failed: time = Sun Jan 26 16:20:39 2020\n, filename = '/home/vl1019/birdvoxdetect/birdvoxdetect/models/birdvoxdetect-v03_trial-12_network_epoch-068.h5', file descriptor = 5, errno = 5, error message = 'Input/output error', buf = 0x563cf06ffd90, total read size = 704, bytes this sub-read = 704, bytes actually read = 18446744073709551615, offset = 5515120)"