search

CAIIVS / chuchichaestli

Where you find all the state-of-the-art cooking utensils (salt, pepper, gradient descent... the usual).
GNU General Public License v3.0
3 stars 0 forks source link

New old implementation of UNet #15

Closed bil-y closed 2 months ago

bil-y commented 2 months ago

I reworked the UNet implementation in response to the memory issues we've come across. In other news, this PR also contains an experimental memory debugging tool (subject to improvement and extension)

phdenzel commented 2 months ago

Just a thought I had while reviewing... our ResNet blocks use GroupNorm which performs better for small batch sizes, but... uses considerably more memory and is computationally more expensive than BatchNorm. Perhaps this is the discrepancy between the monai implementation and ours. Also, the default of 32 might be too high?

https://python-code.dev/articles/335206134

bil-y commented 2 months ago

Just a thought I had while reviewing... our ResNet blocks use GroupNorm which performs better for small batch sizes, but... uses considerably more memory and is computationally more expensive than BatchNorm. Perhaps this is the discrepancy between the monai implementation and ours. Also, the default of 32 might be too high?

https://python-code.dev/articles/335206134

GroupNorm was one of the first things I changed (I tried both Batch- and LayerNorm in its place). I don't think it matters too much, but I haven't benchmarked it for this implementation. We could also just make it configurable.