search

cc-ai / kdb

AI can help Visualizing the Impacts of Climate Change. This is an open forum to share our work
https://mila.quebec/en/ai-society/visualizing-climate-change/
Apache License 2.0
19 stars 3 forks source link

Simulated data in flood generator doubles training time #114

Open 51N84D opened 4 years ago

51N84D commented 4 years ago

The addition of simulated data (with domain adaptation) to the flood generator makes training significantly slower

vict0rsch commented 4 years ago

Any investigation result to understand why this is hapenning?

51N84D commented 4 years ago

I'm still investigating, but I believe it's an issue on the dataloading side. Interestingly, I noticed this in my latest MaskGenerator experiments.

Would it make sense that resizing/transforming a very large image (like the simulated data) takes a long time?

sashavor commented 4 years ago

We can figure this out easily by resizing all of the images up front :)

On Tue., Apr. 7, 2020, 5:00 p.m. Sunand Raghupathi, < notifications@github.com> wrote:

I'm still investigating, but I believe it's an issue on the dataloading side. Interestingly, I noticed this in my latest MaskGenerator experiments.

Would it make sense that resizing/transforming a very large image (like the simulated data) takes a long time?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/cc-ai/kdb/issues/114#issuecomment-610617712, or unsubscribe https://github.com/notifications/unsubscribe-auth/ADMMIIVIZC2QPNZIFC3K7S3RLOH6PANCNFSM4L2QONRQ .

vict0rsch commented 4 years ago

Also @51N84D if you write a test case for that you should be able to figure it out: data loading should not take more than a forward pass I think