valassi
commented
2 years ago Hi Olivier, just to mention that this is related to a few things I have been working on these days https://github.com/madgraph5/madgraph4gpu/issues/455 https://github.com/madgraph5/madgraph4gpu/issues/458 and even https://github.com/oliviermattelaer/mg5amc_test/issues/15
Just to summarise some possible options, and the directions I have taken (that we can rediscuss! also because all this needs integration upstream):
- I completely agree with you (I guess this is the main point of this #9) that one should be able to vary the vector size not only at code generation time but also later on (runcard i.e. anyway build time, or even runtime)
- I went for being able to modify it at runtime, for maximum flexibility (presently through the input file "madevent < inputfile", but eventually this may be an env variable, or a cad file read at runtime): this is similar to what we do in cudacpp check.exe/gcheck.exe through command line arguments
- The big difference with cudacpp check.exe/gcheck.exe is that in CUDA/C++ we use malloc dynamically, while for the moment in Fortran we use arrays that have dimensions fixed at build time. We could maybe change that also in Fortran, but it risks being very complex. The strategy I used (https://github.com/madgraph5/madgraph4gpu/issues/455) is to split your NB_PAGE into two variables, NB_PAGE_MAX for the array dimensions at build time, and NB_PAGE_LOOP (<= NB_PAGE_MAX) for the number of elements which are actually used. It is NB_PAGE_LOOP that I am taking from the inputfile at runtime. The NB_PAGE_MAX is fixed in the code (but this we could even take from a run card as you suggest?)
- The main motivation for all of this was to increase the cuda grid sizes. To have efficient GPU processing, we need 16k-500k events per grid. I am now using 16k for the array dimension NB_PAGE_MAX: if I go above, the program segfaults, I guess it is out of memory. We need to look into this further. The point of having also NB_PAGE_LOOP at runtime is, for instance, that I may want to process only fewer events, without recompiling: in that case, you must limit your loops to a lower number of events.
The discussion about include guards (https://github.com/madgraph5/madgraph4gpu/issues/458) is related, because the arrays dimensioned using NB_PAGE_MAX (or even in the current code using NB_PAGE) are in .inc files where NB_PAGE is used. This means that either vector.inc (defining NB_PAGE) must be included in those .inc, or it must be included in the .f files that include those .inc. In C++ you would include the headers within other headers, but in Fortran without include guards this is complex. For the moment I am just making sure that each .f file includes all relevant .inc files (but I am trying not to include .inc files in other .inc files).
The discussion about get_user_params (https://github.com/oliviermattelaer/mg5amc_test/issues/15) is also related, because if you want to have NB_PAGE_LOOP defined at runtime, then it must be defined very early on. The function that is now used to read from 'madevent < inputfile', get_user_params, is called quite late in driver.f, while I need NB_PAGE_LOOP earlier on. What I am doing for the moment is that I have a separate READ(,) statement in driver.f, outside get_user_params, quite a bit earlier than the call to get_user_params.
Voila... many points to discuss at some point, but at least I try to summarise what I have done! Thanks Andrea
Not that the model does use vector.inc and has hardcoded the value for the coupling/G