lamikr / rocm_sdk_builder

Other
113 stars 8 forks source link

clean mechanism for python project build files under it's src directory #57

Closed lamikr closed 3 weeks ago

lamikr commented 3 weeks ago

Python projects pytorch, pytorch_audio, pytorch_vision, onnxruntime, DeepSpeed, torch_migraphx and bitsandbytes does not build cleanly under builddir but instead under src-directory.

When doing a clean build, files under "src_projects/pytorch/build" for example does not get cleaned up and can prevent project reconfiguration if patches or other dependencies have changed.

Need to have a way to clean them properly and I was first thinking that there may be need for new hook-call for python.

Ended up easier solution where each of these projects should have preconfig_rocm.sh script that can be called on project preconfig time. This way it's only called once and for each clean build. And if it is wanted to call again, user should remove it's build dir. (for example builddir/039_01_pytorch)

daniandtheweb commented 3 weeks ago

I still haven't tried it but doesn't git clean -fdx work?

lamikr commented 3 weeks ago

I did't know about the existence of "git -fdx", its very nice. For now I called "python setup.py clean" which seems to be semi-often supported. I checked that 2 or 3 of python projects we have seemed to have somekind of support for it. For rest I checked the existence of build dir or py/build and delete it if it exist when preconfig has not been done yet.

But "git -fdx" could be even better if it works. In pytorch I have seen situation when creating some patch to .cu file, that the hip file generated from it did not get updated just by cleaning the build directory. (On beginning of pytorch build there are always executed hipify commands that does that)