Abysmal memory allocation during the backward pass

SciML / OperatorLearning.jl

No need to train, he's a smooth operator

https://operatorlearning.sciml.ai/dev

MIT License

43 stars 8 forks source link

Abysmal memory allocation during the backward pass #12

Closed pzimbrod closed 2 years ago

pzimbrod commented 3 years ago

When running burgers.jl with commit c62d8deb2efb72b11fb9b903720f6615e4ea93b6, Julia allocates massive amounts of memory to a point that Julia gets killed on a 128GB RAM machine.

This behavior is abnormal and likely somewhere an abundant amount of copies takes place. This is reproducible when running both on CPU and GPU.

pzimbrod commented 2 years ago

Maybe related to a reported problem that Zygote has with differentiating FFTs.

Possible fix could be to work with @view and do in-place transforms as suggested in the linked Discourse thread.

pzimbrod commented 2 years ago

Found a workaround for now. Closing.