MicrobeLab
commented
4 years ago Hi, the model checkpoint files record the current step in their names. Take a look at the files saved in the model_dir. For example, model.ckpt-10000.data-00000-of-00001 means that the model was saved at the 10000 steps of training. The log for training/prediction will show the steps of training or the name of model checkpoints. Make sure that the model checkpoint from the exact training step is loaded without random initialization.
I don't think there were somethong wrong with your data (given that the training process was normal during the first round of training), as long as you've noticed that the structures of tfrecord for training and prediction are different.
Hi, I'm trying to run DeepMicrobes on my own data but I'm running into some trouble. When I train seq2species through DeepMicrobes, everything seems to be working fine and training accuracy rises/loss drops, however when I reload the model and run it again on the training data, be it for new rounds of training or eval/prediction, the model performs no better than random. The output of the prediction file shows that it just predicts the same class continuously at P=1.0. No errors are thrown and the structure of the tfrecord file seems fine. I also manually checked the weights and they are loaded correctly. This even occurs in a 4-class problem of clearly divergent species.
As a sanity check I've ran with very simple sequences, just 4 species with only homopolymers (so that's poly-A, poly-T, poly-C and poly-G), and then the accuracy is retained after a model reload.
I'm guessing there's still something wrong with the data but I can't imagine what anymore...I've attached the sequences in fasta, tfrecord and json (i.e. converted from tfrecord), and a yml of the conda env I'm running in. Could you have a look and see if I've done anything wrong? simple_tst_refseq_100nt.zip