kracwarlock
commented
8 years ago Hi @ae86208 the PD time (which is the data loading time) is quite large (~640s it should generally be < 5s) in your experiments. While testing this is going to take a huge amount of time since it will go over all batches. You will need to investigate why this is happening.
Maybe the hard drive you are using for the dataset has a lot of latency? In any case you can either
- rewrite the data handler code for loading batches
- modify it to load all the batches at once provided you have enough memory
After update 100, the model begin to predict, while as shown above, the prediction has been running for over 24 hours without progress to new model update, is that normal? If not, what maybe the main cause of this problem? I'm using your code of issue #6 to do the data preprocessing, with train/validate/test 6:2:2, thanks!