implement new CI and fixed the one related to the compilation issue.

oliviermattelaer commented 1 month ago

Hi,

This PR is basically the same as #865 but not targetting master but testsuite_only. (and with a lot of cherry-pick/squashing from the original branch), so #865 is just the sum of this PR and the one related to testsuite_only.

Note that this PR need a change to the upstream madgraph (which should be urgently approved) and the associated PR is https://github.com/mg5amcnlo/mg5amcnlo/pull/132

valassi commented 1 month ago

Hi @oliviermattelaer thanks so far looks good.

One 'minor' point, I would probably prefer (need to cross check) to split the clean logic in Source from

clean:
    $(RM) *.o $(LIBRARIES) $(BINARIES)
    cd PDF; make clean; cd ..
    cd PDF/gammaUPC; make clean; cd ../../
    cd DHELAS; make clean; cd ..
    cd CERNLIB; make clean; cd ..
    cd MODEL; make clean; cd ..
    if [ -d RUNNING ]; then cd RUNNING; make clean; cd ..; fi
    cd BIAS/dummy; make clean; cd ../..
    cd BIAS/ptj_bias; make clean; cd ../..
    if [ -d $(CUTTOOLSDIR) ]; then cd $(CUTTOOLSDIR); make clean; cd ..; fi
    if [ -d $(IREGIDIR) ]; then cd $(IREGIDIR); make clean; cd ..; fi
    for i in `ls -d ../SubProcesses/P*`; do cd $$i; make clean; cd -; done;

to the following (what I have now in master through patches)

cleanSource:
    $(RM) *.o $(LIBRARIES) $(BINARIES)
    cd PDF; make clean; cd ..
    cd PDF/gammaUPC; make clean; cd ../../
    cd DHELAS; make clean; cd ..
    cd CERNLIB; make clean; cd ..
    cd MODEL; make clean; cd ..
    if [ -d RUNNING ]; then cd RUNNING; make clean; cd ..; fi
    cd BIAS/dummy; make clean; cd ../..
    cd BIAS/ptj_bias; make clean; cd ../..
    if [ -d $(CUTTOOLSDIR) ]; then cd $(CUTTOOLSDIR); make clean; cd ..; fi
    if [ -d $(IREGIDIR) ]; then cd $(IREGIDIR); make clean; cd ..; fi

clean: cleanSource
    for i in `ls -d ../SubProcesses/P*`; do cd $$i; make clean; cd -; done;

The motivation is that this allows faster make cleanall of cudacpp: by calling cleanSource I ONLY clean Source, then the more complex cleaning looping through P I do elsewhere (I need to delete the build. directories).

I can do that no problem, I propose

I would change the default Source makefile making the split above (might look useless, but it is harmless, make clean does the same)
I would add some comments to explain what the different clean commands do
I would then fix the rest using your nice mechanism
By the way, I would then prefer to actually add the clean at the end rather than in the middle, so I can add one dictionary entry 'append_to_makefile' or something instead of using `additional_dependencies'

I will let you know when done

valassi commented 1 month ago

The motivation is that this allows faster make cleanall of cudacpp: by calling cleanSource I ONLY clean Source, then the more complex cleaning looping through P I do elsewhere (I need to delete the build. directories).

I did a simple test to confirm this, on gq_ttq which is the simplest process with more than one P* in the repo.

Using upstream/master (i.e. what I would lik to go back to)

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> make cleanall | \grep Entering 
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/DHELAS'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/CERNLIB'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/MODEL'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/dummy'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/ptj_bias'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'

Using instead what is currently in this PR

[avalassi@itscrd90 gcc11/usr] /data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu> make cleanall | \grep Entering 
make[1]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/PDF/gammaUPC'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/DHELAS'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/CERNLIB'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/MODEL'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/dummy'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/Source/BIAS/ptj_bias'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[3]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[4]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'
make[5]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/src'

The difference is

11,12d10
< make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gux_ttxux'
< make[2]: Entering directory '/data/avalassi/GPU2023/madgraph4gpuX/epochX/cudacpp/gq_ttq.mad/SubProcesses/P1_gu_ttxu'

In a project with many P, where looping through P is slow, this is an issue. I will go back to the upstream/master logic (with the new codegen however)

valassi commented 1 month ago

Hi @oliviermattelaer some of your confusion in https://github.com/madgraph5/madgraph4gpu/pull/984#issuecomment-2323026877 probably came from the fact that this testsuite_only_fixed did not include the latest upstream/master.

I have now merged master into testsuite_only, and I have also merged testsuite_only into this testsuite_only_fixed.

valassi commented 1 month ago

I have now merged #984 into this PR as discussed with Olivier.

After the CI runs I will merge 979 and this 980 into master.

valassi commented 1 month ago

Ok the CI is as expected, only the three pptt012j tests fail.

I will merge #979 and then this 980.

valassi commented 1 month ago

I have merged #979 into master.

Now merging into master this #980 which also includes #984.

madgraph5 / madgraph4gpu

implement new CI and fixed the one related to the compilation issue. #980