Closed yuanshenli closed 5 years ago
Hi @yuanshenli , This is because of an mir_eval
issue where it fails when there is no matching which was only recently fixed. This no-matching situation inevitably happens in early iterations. There is a recent patch merged to mir_eval
to fix this, but this hasn't made it through a new release at the moment.
So for now, I'd recommend installing the latest commit of mir_eval by:
pip uninstall mir_eval
pip install git+https://github.com/craffel/mir_eval.git
in order to install the most recent version of the mir_eval
package out of the master
branch.
After doing this, the training should run just fine; let me know if you encounter any issues.
Hi @yuanshenli , This is because of an
mir_eval
issue where it fails when there is no matching which was only recently fixed. This no-matching situation inevitably happens in early iterations. There is a recent path merged tomir_eval
to fix this, but this hasn't made it through a new release at the moment.So for now, I'd recommend installing the latest commit of mir_eval by:
pip uninstall mir_eval pip install git+https://github.com/craffel/mir_eval.git
in order to install the most recent version of the
mir_eval
package out of themaster
branch.After doing this, the training should run just fine; let me know if you encounter any issues.
Thank you so much for the timely reply! so far it made through the first 2000 iterations, it did get the warning warnings.warn("Estimated notes are empty.")
at first 500 iterations like before, but as you suggested it just happed in early iterations.
May I ask what values you used for batch_size
and model_complexity
? I kept getting cuda out_of_memory error. below is my gpu configuration:
Mon Mar 4 18:32:31 2019
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 390.77 Driver Version: 390.77 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 GeForce GTX TIT... Off | 00000000:02:00.0 Off | N/A |
| 57% 83C P2 89W / 250W | 525MiB / 12212MiB | 4% Default |
+-------------------------------+----------------------+----------------------+
| 1 GeForce GTX 108... Off | 00000000:03:00.0 Off | N/A |
| 29% 60C P2 133W / 250W | 2951MiB / 11178MiB | 84% Default |
+-------------------------------+----------------------+----------------------+
| 2 GeForce GTX 108... Off | 00000000:04:00.0 On | N/A |
| 0% 48C P8 14W / 280W | 657MiB / 11175MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
Hi @yuanshenli , This is because of an
mir_eval
issue where it fails when there is no matching which was only recently fixed. This no-matching situation inevitably happens in early iterations. There is a recent path merged tomir_eval
to fix this, but this hasn't made it through a new release at the moment.So for now, I'd recommend installing the latest commit of mir_eval by:
pip uninstall mir_eval pip install git+https://github.com/craffel/mir_eval.git
in order to install the most recent version of the
mir_eval
package out of themaster
branch.After doing this, the training should run just fine; let me know if you encounter any issues.
Just as I thought it was working, same error occurred:
0%| | 499/500000 [01:36<26:19:05, 5.27it/s]/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription.py:167: UserWarning: Estimated notes are empty.
warnings.warn("Estimated notes are empty.")
/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/multipitch.py:275: UserWarning: Estimate frequencies are all empty.
warnings.warn("Estimate frequencies are all empty.")
0%| | 2499/500000 [09:13<26:18:47, 5.25it/s]/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py:185: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions.
To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`.
ref_matched_velocities)[0]
ERROR - train_transcriber - Failed after 0:09:22!
Traceback (most recent calls WITHOUT Sacred internals):
File "train.py", line 117, in train
for key, value in evaluate(validation_dataset, model).items():
File "/home/lab/Documents/CS224NProject/Sample/onsets-and-frames/evaluate.py", line 70, in evaluate
p, r, f, o = evaluate_notes_with_velocity(i_ref, p_ref, v_ref, i_est, p_est, v_est, velocity_tolerance=0.1)
File "/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py", line 291, in precision_recall_f1_overlap
offset_min_tolerance, strict, velocity_tolerance)
File "/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py", line 178, in match_notes
ref_matched_velocities = ref_velocities[matching[:, 0]]
IndexError: too many indices for array
Hi @yuanshenli , This is because of an
mir_eval
issue where it fails when there is no matching which was only recently fixed. This no-matching situation inevitably happens in early iterations. There is a recent path merged tomir_eval
to fix this, but this hasn't made it through a new release at the moment. So for now, I'd recommend installing the latest commit of mir_eval by:pip uninstall mir_eval pip install git+https://github.com/craffel/mir_eval.git
in order to install the most recent version of the
mir_eval
package out of themaster
branch. After doing this, the training should run just fine; let me know if you encounter any issues.Just as I thought it was working, same error occurred:
0%| | 499/500000 [01:36<26:19:05, 5.27it/s]/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription.py:167: UserWarning: Estimated notes are empty. warnings.warn("Estimated notes are empty.") /home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/multipitch.py:275: UserWarning: Estimate frequencies are all empty. warnings.warn("Estimate frequencies are all empty.") 0%| | 2499/500000 [09:13<26:18:47, 5.25it/s]/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py:185: FutureWarning: `rcond` parameter will change to the default of machine precision times ``max(M, N)`` where M and N are the input matrix dimensions. To use the future default and silence this warning we advise to pass `rcond=None`, to keep using the old, explicitly pass `rcond=-1`. ref_matched_velocities)[0] ERROR - train_transcriber - Failed after 0:09:22! Traceback (most recent calls WITHOUT Sacred internals): File "train.py", line 117, in train for key, value in evaluate(validation_dataset, model).items(): File "/home/lab/Documents/CS224NProject/Sample/onsets-and-frames/evaluate.py", line 70, in evaluate p, r, f, o = evaluate_notes_with_velocity(i_ref, p_ref, v_ref, i_est, p_est, v_est, velocity_tolerance=0.1) File "/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py", line 291, in precision_recall_f1_overlap offset_min_tolerance, strict, velocity_tolerance) File "/home/lab/.linuxbrew/Cellar/python/3.7.2_2/lib/python3.7/site-packages/mir_eval/transcription_velocity.py", line 178, in match_notes ref_matched_velocities = ref_velocities[matching[:, 0]] IndexError: too many indices for array
Hmm just realized that my mir_eval wasn't updated properly, let me try again.
I'm running with the following parameters in train.py
And I got the following error:
BTW, another issue, I first tried all the default parameters, and I kept getting GPU out of memory message. My system should meet the minimum system requirements.
Does anyone have insights into these two issues?