Open makortel opened 2 weeks ago
assign core
New categories assigned: core
@Dr15Jones,@makortel,@smuzaffar you have been requested to review this Pull request/Issue and eventually sign? Thanks
cms-bot internal usage
A new Issue was created by @makortel.
@Dr15Jones, @antoniovilela, @makortel, @mandrenguyen, @rappoccio, @sextonkennedy, @smuzaffar can you please review it and eventually sign/assign? Thanks.
cms-bot commands are listed here
Here is an example from https://github.com/cms-sw/cmssw/pull/46204#issuecomment-2392400600
With @Dr15Jones we hypothesized maybe the FileInPath
could lead to this increase?
Another reason could be the PluginManager cache would also include reading from the workspace's lib directory and therefore increase its working memory.
Another example from https://github.com/cms-sw/cmssw/pull/46388#issuecomment-2417639671
Interestingly https://github.com/cms-sw/cmssw/pull/46241#issuecomment-2420668031, that was a test with full_cmssw = true
(I checked both compilation log and unit test report to see that many packages were indeed rebuilt), does not show noticeable increase
I was able to reproduce locally a "rebuild of most of the universe in a developer area" leading to ~30 MB higher max memory used
than "empty developer area".
I ran IgProf for both cases, below are the MEM_LIVE after 10 events
and the cause is in edmplugin::CacheParser::read()
, see baseline vs rebuild. I believe it is this overload
https://github.com/cms-sw/cmssw/blob/b96fd024cd7818d2b3bd5ba01d3a3075943f3233/FWCore/PluginManager/src/CacheParser.cc#L120-L122
So the cost is that the developer area's .edmplugincache
becomes large, and we read that in memory in addition to the .edmplugincache
from the release area?
In my "rebuild" developer area the .edmplugincache
is 1.3 MB, whereas the $CMSSW_RELEASE_BASE/lib/slc7_amd64_gcc12/.edmplugincache
is 1.5 MB.
I wonder if there would be any easy way to exclude the contribution from CacheParser::read()
(other than starting the monitoring later in the job)
Instrumenting some of the PluginManager code, I see (in my rebuild area) that the job reads
biglib/slc7_amd64_gcc12/.edmplugincache
lib/slc7_amd64_gcc12/.edmplugincache
lib/slc7_amd64_gcc12/.poisonededmplugincache
biglib/slc7_amd64_gcc12/.edmplugincache
lib/slc7_amd64_gcc12/.edmplugincache
So with nearly all of CMSSW build, most of the plugins have 3 entries in the CategoryToInfos
: local, local poisoned, and release.
Observed in https://github.com/cms-sw/cmssw/pull/46200#issuecomment-2394356850.