we often run out of memory on grid, I think the issue is due to the following bad ordering of tasks
a) we load the json files of SF and stuff like that at config time
b) just loading the config means reaching 1GB
c) then we run the preprocessor, cmsRun, that needs 1-2GB
d) then we really run the heppy part where we need SF and stuff like that
We must implement a deferred loading of the payloads
arizzi
commented
7 years ago
we often run out of memory on grid, I think the issue is due to the following bad ordering of tasks
a) we load the json files of SF and stuff like that at config time b) just loading the config means reaching 1GB c) then we run the preprocessor, cmsRun, that needs 1-2GB d) then we really run the heppy part where we need SF and stuff like that
We must implement a deferred loading of the payloads