WeidiXie / VGG-Speaker-Recognition

Utterance-level Aggregation For Speaker Recognition In The Wild
363 stars 97 forks source link

error in data generation MP 2 #29

Closed gogyzzz closed 5 years ago

gogyzzz commented 5 years ago

I tried to train with voxceleb2

5272/7492 [====================>.........] - ETA: 49:13 - loss: 7.6936 - acc: 0.0133

and got message while training

multiprocessing.pool.RemoteTraceback: 
"""
Traceback (most recent call last):
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/data/ghostvlad-speaker-original/src/utils.py", line 28, in load_data
    linear_spect = lin_spectogram_from_wav(wav, hop_length, win_length, n_fft)
  File "/data/ghostvlad-speaker-original/src/utils.py", line 22, in lin_spectogram_from_wav
    linear = librosa.stft(wav, n_fft=n_fft, win_length=win_length, hop_length=hop_length) # linear spectrogram
  File "/usr/local/lib/python3.5/dist-packages/librosa/core/spectrum.py", line 165, in stft
    y = np.pad(y, int(n_fft // 2), mode=pad_mode)
  File "/usr/local/lib/python3.5/dist-packages/numpy/lib/arraypad.py", line 1290, in pad
    " in axis {} of `array`".format(axis))
ValueError: There aren't any elements to reflect in axis 0 of `array`
"""
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
  File "main.py", line 196, in <module>
    main()
  File "main.py", line 136, in main
    verbose=1)
  File "/usr/local/lib/python3.5/dist-packages/keras/legacy/interfaces.py", line 91, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training.py", line 1418, in fit_generator
    initial_epoch=initial_epoch)
  File "/usr/local/lib/python3.5/dist-packages/keras/engine/training_generator.py", line 181, in fit_generator
    generator_output = next(output_generator)
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py", line 601, in get
    six.reraise(*sys.exc_info())
  File "/usr/local/lib/python3.5/dist-packages/six.py", line 693, in reraise
    raise value
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py", line 595, in get
    inputs = self.queue.get(block=True).get()
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 119, in worker
    result = (True, func(*args, **kwds))
  File "/usr/local/lib/python3.5/dist-packages/keras/utils/data_utils.py", line 401, in get_index
    return _SHARED_SEQUENCES[uid][i]
  File "/data/ghostvlad-speaker-original/src/generator.py", line 42, in __getitem__
    X, y = self.__data_generation_mp(list_IDs_temp, indexes)
  File "/data/ghostvlad-speaker-original/src/generator.py", line 58, in __data_generation_mp
    X = np.expand_dims(np.array([p.get() for p in X]), -1)
  File "/data/ghostvlad-speaker-original/src/generator.py", line 58, in <listcomp>
    X = np.expand_dims(np.array([p.get() for p in X]), -1)
  File "/usr/lib/python3.5/multiprocessing/pool.py", line 608, in get
    raise self._value
ValueError: There aren't any elements to reflect in axis 0 of `array`
I0528 18:57:06.038911 22079 executor.cpp:675] Container exited with status 1
W0528 18:57:06.038911 22072 logging.cpp:93] RAW: Received signal SIGTERM from process 16635 of user 0; exiting

And I think the accuracy is too low. How about in your case?

WeidiXie commented 5 years ago

I think this is something wrong with python multiprocess.