AMairesse
commented
7 years ago Hi, I'm using a GTX 960 and getting to a good result can take as long as a month (24/24). Obviously not practical so I usually end up renting a p2.xlarge on AWS. The last pre-trained network took 9 days to train on that hardware. I'm will probably buy a GTX 1080 TI in the coming weeks, which should give better result than the p2.xlarge of Amazon :+1: The number of steps isn't a good way to count because you can change the number file in a batch or the mini-batch size. One thing making the p2.xlarge more efficient is the ability to use a bigger batch_size, which depend a lot of the video memory available. In the dev branch I'm working on a new way to make the learning more efficient (I hope) :
- the dataset is ordered by size
- the model is re-created before running a batch but the unrolling is limited to the size of the larger file in this portion of the dataset. The bigger the unroll, the longer the processing, so this is really time effective. But for now it seems that the quality of the learning process is affected and I can't obtain a good result.
@AMairesse , can you give some details about training. How much time it takes in training for some number of steps (about hardware).