search

microsoft / nnfusion

A flexible and efficient deep neural network (DNN) compiler that generates high-performance executable from a DNN model description.
MIT License
967 stars 164 forks source link

Can nnfusion train VHR images whoes feature map in a CNN model exceeds the gpu memory? #530

Open AugustineYang opened 11 months ago

AugustineYang commented 11 months ago

Hi. I'm trying to do some segmentation tasks on FCN32s on some 25000*25000 RGB RS images. In your paper of "Welder: Scheduling Deep Learning Memory Access via Tile-graph", you mentioned that your work supports handling DNN models with large input (e.g. high-resolution images). But I still get "CUDA out of memory" ERROR following the instruction How to use NNFusion Python interface for inference/training here. I guess it comes from "PTTrainer(model, loss_func, "cuda:0")". How should I set the args of the trainer to achieve the goal? Thanks for your help.

AugustineYang commented 11 months ago

It seems that "Welder" is implemented in a seperated branch other than "main" branch. What should I do to make Welder engage in the compilation stage?

LeiWang1999 commented 11 months ago

checkout this branch https://github.com/microsoft/nnfusion/tree/osdi2023welder or https://github.com/nox-410/Welder

AugustineYang commented 11 months ago

checkout this branch https://github.com/microsoft/nnfusion/tree/osdi2023welder or https://github.com/nox-410/Welder

Thanks for your reply. Does Welder provide python interface for training or inference, just like this link How to use NNFusion Python interface for inference/training?