search

jongwook / onsets-and-frames

A Pytorch implementation of Onsets and Frames (Hawthorne 2018)
MIT License
209 stars 66 forks source link

Many warnings during training #10

Closed KinWaiCheuk closed 4 years ago

KinWaiCheuk commented 5 years ago

During training, I get tons of warning for Sequential, ConvStack, LSTM and etc. The warning message is like this

"type " + obj.name + ". It won't be checked " /opt/conda/envs/onset_model/lib/python3.6/site-packages/torch/serialization.py:292: UserWarning: Couldn't retrieve source code for container of type LSTM. It won't be checked for correctness upon loading.

Is it normal?

I created a new conda environment to run this code and installed all the dependencies via requirements.txt.

jongwook commented 4 years ago

I haven't seen those errors, but only similar ones when I load a model from pickled checkpoints. I suspect for some reasons Python is not loading the source code of pytorch (Sequential and LSTM) and onsets-and-frames (ConvStack) from your file system.

It should be harmless though. Source code (accessed via inspect.getsourcefile) is only used for safety check.

KinWaiCheuk commented 4 years ago

As for the training, we train on 500,000 iterations, and each iteration contains 8 spectrograms. Am I right?

On my system, it shows that 70 hours are required to finish 500,000 iterations. Is it a reasonable speed? Or is it too slow?

I am asking the warnings because I am not sure if the training time is affected by it or not.

jongwook commented 4 years ago

The speed looks fine. It took about a week for me to train for 1M iterations on a single 1080 Ti.