search

neptune-ai / open-solution-mapping-challenge

Open solution to the Mapping Challenge :earth_americas:
https://www.crowdai.org/challenges/mapping-challenge
MIT License
378 stars 96 forks source link

Missing 4,549 items in masks_overlayed_eroded_0_dilated_2/val/distances #187

Closed carbonox-infernox closed 5 years ago

carbonox-infernox commented 5 years ago

For some reason I am missing 4,549 masks in the masks_overlayed_eroded_0_dilated_2/val/distances directory, which I only found out after trying to train the first epoch and having it fail because it couldn't find one.

What do I do? Prepare masks again? How could doing it again fix the problem when it failed to create 4,549 masks the first time? Is there a way to make training just skip missing val masks? I still have the vast majority of them.

kamil-kaczmarek commented 5 years ago

@jakubczakon do you have any idea what are possible reasons?

jakubczakon commented 5 years ago

Hi, @carbonox-infernox .

metadata.csv is generated based on what the target folders contain. So you could re-run prepare_metadata and it should create it based on what is there. It is quick so I would suggest you do that and check if the size of the new metadata.csv is different than the old metadata.csv .

carbonox-infernox commented 5 years ago

I ended up preparing masks again, and before I started training I made sure that the number of files in each sub-folder (distances, masks, sizes) matched the number of images in the data set for train and val. Then I prepared metadata again and started training. I set it to go for just one epoch for now, and It'll be done in about 2 hours, so I'll let you guys know how that turned out.