continious learning notebook

[johnsmithm]

• predict unlabeled data
• sort the predicted labels based on confidence (to be done) #17
• correct some images
• trained the model #33
• evaluate the model (to be done) #7
• repeat until we have a good model

[johnsmithm]

for each iteration, create a folder in data/iteration_n,

• save the prediction for unlabeled data in data/iteration_n/prediction.csv.
• save the corrected in data/iteration_n/corected.csv
• save the training model in data/iteration_n/models
• save the evaluation result in data/iteration_n/eval.csv use dvc to store data, and reproduse

[johnsmithm]

use notebooks/0.1-im-DeepForest_Training.ipynb

[johnsmithm]

split into 2 tasks

46

33

[johnsmithm]

data https://drive.google.com/file/d/1WRusYYA2BiOIiefW1GZ5rA0pRag-nYQ3/view?usp=sharing

[johnsmithm]

fara dvc, folosind drive

1. creem un folder /data/images_dataset cu imagini 400x400 fara etichete, zicem 100 de imaginii
2. folosim modelul initial pentru a face predictii pe cele 100 de imagini, salvam prezicerile ca xml in aceasi mapa + csv cu scoruri pt sortare
3. sortam prezicerile dupa increderea modelului, alegem k=10 cele mai nesigure imagini si le mutam intr-o mapa data/training/iteratia_1 -> imagini si xlm preziceri, mutam nu copiem
4. corectam cele k=10 imagini din data/training/iteratia_1 cu labelimg, iarasi salvam corectarile ca xml
5. adaugam cateva m=2 exemple din data/training/iteratia_1 in data/test/iteratia_1
6. incarcam folderul data in drive
7. rulam in colab antrenarea modelului, salvam modelul in data/models/iteratia1/model.pl si raportul evaluarii in /data/models/iteratia1/raport.json
8. prezicem cele 90 de imagini ramase in /data/images_dataset si salvam ca xml
9. sortam prezicerile dupa increderea modelului, alegem k=10 cele mai neincrezute imagini si le mutam intr-o mapa data/training/iteratia_2 -> imagini si xlm preziceri, mutam nu copiem
10. descarcam data/training/iteratia_2 pe calculator
11. corectam cele k=10 imagini din data/training/iteratia_2 cu labelimg, iarasi salvam corectarile ca xml
12. adaugam cateva m=2 exemple din data/training/iteratia_1 in data/test/iteratia_2
13. incarcam xml-urile din data/training/iteratia_2 in drive
14. repetam de la pasul 7 pana avem un model bun sau terminam imaginile din /data/images_dataset