search

neptune-ai / open-solution-mapping-challenge

Open solution to the Mapping Challenge :earth_americas:
https://www.crowdai.org/challenges/mapping-challenge
MIT License
377 stars 96 forks source link

when training finishes it hits the memory error #210

Open jakubczakon opened 4 years ago

jakubczakon commented 4 years ago

Running train crashes when the pipelines are collating results from running transform on the entire dataset.

animeshsahu80 commented 4 years ago

did you find any solution

jakubczakon commented 4 years ago

Well, the training is working and the checkpoints are saved after every epoch (if improved) so the trained model is available.

All that needs to be done is:

cp experiment/checkpoints/unet/best.torch experiment/transformers/unet

and then run the evaluation/prediction.

So it is more of an inconvenience than a big problem.