search

Tessellate-Imaging / monk_v1

Monk is a low code Deep Learning tool and a unified wrapper for Computer Vision.
Apache License 2.0
582 stars 215 forks source link

results have deteriorated since this warning #103

Closed swankyshahir closed 4 years ago

swankyshahir commented 4 years ago

same dataset used to give upto 94% validationn accuracy but since 15 days , i am getting the following error with

gtf.Train(); Training Start WARNING:tensorflow:period argument is deprecated. Please use save_freq to specify the frequency in number of batches seen.

please enlighten

abhi-kumar commented 4 years ago

Could you please elaborate on the issue by providing the following information a) OS and System details b) Previous and current TF and Keras versions c) Has the val accuracy decreased post retraining or with a previously trained model

swankyshahir commented 4 years ago

using GPU in google colab. i was always getting in the range of 93-94% val accuracy whenever i was running it... my bad i didnt save a model.... now since around 2-3 weeks, same data, same code but val accuracy has fallen to 15-20% and a new warning as mentioned above is showing up... :(

swankyshahir commented 4 years ago

problem only with keras.... working fine in pytorch

abhi-kumar commented 4 years ago

We analysed the issue. Colab has upgraded tf 2.2 to tf 2.3. Hence the changes in results. Lower the learning rate a bit and try when using keras on colab.

swankyshahir commented 4 years ago

ok. i will try lowering the learning rate. Current learning rate was 0.01 . Now i have run it in pytorch and downloaded the best model. can you please tell me the code to load the model back for inference as i want to use only that model everytime and dont want to train again to infer from the workspace. also workspace inference is using the final model and not the best model, any ideas ?

abhi-kumar commented 4 years ago

Best model is copied to final model post training.

To run using the same model, remove all intermediate models from workspace directory, and download that directory. Everytime you want to use it, upload your directory to pwd (present working directory) and see study roadmap examples for inference code - https://github.com/Tessellate-Imaging/monk_v1/tree/master/study_roadmaps/1_getting_started_roadmap/1_getting_started_with_monk

swankyshahir commented 4 years ago

Thanks a lot for the prompt help. I have finally trained it in pytorch , got the original results back and saved the work space directory... However, I did like to point out if it helps that keras isn't working even with lower learning rate... Also when save_intermediate_models is set to False, even the pytorch performance was dropping, bug ?

swankyshahir commented 4 years ago

Also I did like to thank you for making this fantabulous product that has enabled non-programmers like me , dabble in deep learning.

abhi-kumar commented 4 years ago

Am glad you liked the library. save_intermediate_models when set as False simply doesn't save intermediate models. It has no effect on training. And no two runs even with similar parameters and settings will generate similar accuracies.

swankyshahir commented 4 years ago

well i understood it that way but somehow the accuracies were way off the mark but rectified itself on setting it back to true... dont know if it was pure chance or otherwise... still much gratitude for helping