improving run time of generator

[Markomrnkvc]

we need a ,ethod on how to improve the run time

especially regarding the pickle file--> how do we avoid opening and closing the pickle file for every images that gets loaded into the file but instead implement some sort of checkpoints

• for example: loading 50 images, then closing the pickle file (and perhaps the csv file) to save the progress, and then opening it again for the next 50 files
• this needs to be done for both csv and pickle in the same way to avoid having dissimal progress (both filees need to have the same progress, so that all images are getting loaded with the rigth IDs into both files)
• csv more imoprtant because generator is based on the csv (ID count, checking which image has already been loaded)
• improving the way we check which images have already been loaded into csv (as of now we generate a list whit all the paths in the csv and then, for every image we want to load, we check if the path is in said list)

[Markomrnkvc]

finished at least party (first bullet point)

[Markomrnkvc]

could update way of checking which file has already been loaded

• try using df and taking biggest image_id, the starting the generator at said id