TriggerSimu + picoDst for BEMC data in daq->picoDst production

[starsdong]

• PicoDst EMC collection pointer taken from StEvent directly in case of daq production
• TriggerSimuMaker added to BFC chain
• EmcRawMaker used to obtain BEMC data in case of daq production

[starsdong]

This line is trying to deal with the daq->picoDst production where StEvent should be there. In the MuDst->picoDst production, the EmcCollection will be obtained from MuDst directly (line 861).

[starsdong]

Hi Dmitri, may need your help here. One of the tests from the CI build failed (119). But I tested the same macro/chain which ran OK with my local setup. See log file /star/u/dongx/pwg/Git/20220225/test_21.log Do you have any insight how to resolve this? Thanks

[starsdong]

By comparing the log file, it seems the test job crashed in TriggerSimuMaker Eemc related part when trying to load some local files ... StTriggerSimuMaker:INFO - Using mask file /star/u/pibero/public/StarTrigSimuSetup/mask/eec.05-13-09.dat ... The setup is supposed to be the same as what we have been using in genDst.C for MuDst->picoDst production. Gene and Grigory, do you know whether we had encountered this issue or how was this dealt with? Thanks

[plexoos]

Xin, in your log file I see the following lines which may suggest that the job is trying to access non existing local files. If this is the case, and the files are indeed used they should be placed in the repository or database.

 3491 StTriggerSimuMaker:INFO  - ^[[32mEemc::InitRun()  yyyymmdd=20210624  hhmmss=065455
 3492 ^[[0m
 3493 StTriggerSimuMaker:INFO  - ^[[32mUsing mask file /star/u/pibero/public/StarTrigSimuSetup/mask/eec.05-13-09.dat^[[0m
 3494 StTriggerSimuMaker:INFO  - ^[[32mUsing mask directory /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010^[[0m
 3495 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-1-current_beam_config.dat^[[0m
 3496 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-2-current_beam_config.dat^[[0m
 3497 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-3-current_beam_config.dat^[[0m
 3498 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-4-current_beam_config.dat^[[0m
 3499 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-5-current_beam_config.dat^[[0m
 3500 StTriggerSimuMaker:INFO  - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-6-current_beam_config.dat^[[0m
 3501 StTriggerSimuMaker:INFO  - ^[[32mUsing EEMC OFFLINE pedestals^[[0m

[zlchang]

Hi guys,

This is a known issue. We had thought about put these masks into database. For this task, we need to creat new table structure in the database. I believe I have all the information from EEMC expert regarding the masks. This was a low priority task. I was supposed to do the work, but the work got stalled for a year or so. The issue will prevent you from running jobs in nodes that do not have access to afs file system.

Zilong

On Feb 28, 2022, at 4:13 PM, Dmitri Smirnov @.***> wrote:



Xin, in your log file I see the following lines which may suggest that the job is trying to access non existing local files. If this is the case, and the files are indeed used they should be placed in the repository or database.

3491 StTriggerSimuMaker:INFO - ^[[32mEemc::InitRun() yyyymmdd=20210624 hhmmss=065455 3492 ^[[0m 3493 StTriggerSimuMaker:INFO - ^[[32mUsing mask file /star/u/pibero/public/StarTrigSimuSetup/mask/eec.05-13-09.dat^[[0m 3494 StTriggerSimuMaker:INFO - ^[[32mUsing mask directory /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010^[[0m 3495 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-1-current_beam_config.dat^[[0m 3496 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-2-current_beam_config.dat^[[0m 3497 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-3-current_beam_config.dat^[[0m 3498 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-4-current_beam_config.dat^[[0m 3499 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-5-current_beam_config.dat^[[0m 3500 StTriggerSimuMaker:INFO - ^[[32mScanning mask file /star/u/pibero/public/StarTrigSimuSetup/mask/10.15.2010/tower-6-current_beam_config.dat^[[0m 3501 StTriggerSimuMaker:INFO - ^[[32mUsing EEMC OFFLINE pedestals^[[0m

— Reply to this email directly, view it on GitHubhttps://github.com/star-bnl/star-sw/pull/314#issuecomment-1054667193, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGH7F3JWLVZSLDXYPWCTH4DU5PQPDANCNFSM5PRZ3ZRQ. You are receiving this because your review was requested.Message ID: @.***>

[starsdong]

I see similar lines shown in the production log file for MuDst->picoDst conversion. /star/rcf/prodlog/P20ic.SL21d/log/daq/job_st_physics_adc_18105030_raw_5500050.log.gz Gene, how did we handle it in the current production?

[genevb]

This was happening without us (the Production Team) knowing about it. Of course it isn't OK to access files that are in some private directory because such a directory won't be available to all possibilities for running the job. It seems we've been "lucky" in some sense for the past couple years to be running all these conversion productions on the RACF/SDCC farm.

-Gene

On Feb 28, 2022, at 5:02 PM, Xin Dong @.**@.>> wrote:

I see similar lines shown in the production log file for MuDst->picoDst conversion. /star/rcf/prodlog/P20ic.SL21d/log/daq/job_st_physics_adc_18105030_raw_5500050.log.gz Gene, how did we handle it in the current production?

[starsdong]

1) Zilong, if you can help with the DB structure and this is not a big amount of work, would be very much appreciated if you can help.

2) However, I am also wondering how useful these tables are. It seems all jobs are reading the same mask files which were created in 2010. Are they just for test or indeed can be valid still for current data?

[zlchang]

Hi Xin,

I can try to look into this. The impact is very minimal. If I remember correctly, for run12 and run13, there should be no impact at all whether you read or don’t read those files.

Zilong

On Feb 28, 2022, at 5:25 PM, Xin Dong @.***> wrote:



1. Zilong, if you can help with the DB structure and this is not a big amount of work, would be very much appreciated if you can help.

2. However, I am also wondering how useful these tables are. It seems all jobs are reading the same mask files which were created in 2010. Are they just for test or indeed can be valid still for current data?

— Reply to this email directly, view it on GitHubhttps://github.com/star-bnl/star-sw/pull/314#issuecomment-1054721843, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGH7F3MZTKSG3SKL5PB652LU5PY33ANCNFSM5PRZ3ZRQ. You are receiving this because your review was requested.Message ID: @.***>

[starsdong]

Zilong, do you have any update on this? Would be great if you have some findings on the difference, so maybe tomorrow we can decide the path forward. Specially on the st_hf isobar data production, I believe what matters are only the BHT triggers. These eemc masks will not matter. These masks may have an impact for pp picoDst production as it involves some JP triggers.

[genevb]

Just a reminder that the isobar st_hf dataset has already been processed into MuDsts, and only needs the conversion to picoDsts. I don't need to wait for SL22a for that second step. Just the database content is the concern for that, I guess, and since I'll run it on RACF/SDCC, there will be access to those private files just as has been the case for conversion productions over the past few years (though now I'm wondering if there were any issues for embedding that produced picoDsts at other sites or in containers without AFS).

-Gene

On Mar 1, 2022, at 5:10 PM, Xin Dong @.**@.>> wrote:

Zilong, do you have any update on this? Would be great if you have some findings on the difference, so maybe tomorrow we can decide the path forward. Specially on the st_hf isobar data production, I believe what matters are only the BHT triggers. These eemc masks will not matter. These masks may have an impact for pp picoDst production as it involves some JP triggers.

[starsdong]

Thank you Zilong. I made the suggested cleanup in the new commit. A quick test with the Run22 data looks OK.

[starsdong]

I made two updates (Eemc local files for masking skipped - they are actually not in effect when running on clusters). Zilong is working on the DB solution as discussed on Wed. PicoDstMaker, the sanity check on threshold values now lowered to be just >0 in case some runs with low threshold values. Will wait for the CI Build test outcome. Grigory, if you can have a look at the PicoDstMaker changes, comment or approve it. Much appreciated.

[nigmatkulov]

Looks good to me.

[genevb]

A note on some results from nightly tests since this PR was merged: this has resulted in very significant increases to the size of picoDst, MuDsts, and even event.root files for many of the jobs. I'm particularly concerned about the MuDst increases which are larger than a factor of x2!!! See for example these two nightly tests' DSTs:

ls -ltr /star/rcf/test/dev/daq_sl302.ittf_opt//year_2016/dAu20_production_2016/{event,MuDst,picoDst}.root

-rw-r--r-- 1 starreco stargrid 549987604 Mar 5 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sat/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 43143925 Mar 5 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sat/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 264696 Mar 5 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sat/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 549987252 Mar 6 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sun/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 43143971 Mar 6 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sun/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 264696 Mar 6 03:38 /star/rcf/test/dev/daq_sl302.ittf_opt/Sun/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 549986881 Mar 7 03:40 /star/rcf/test/dev/daq_sl302.ittf_opt/Mon/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 43143966 Mar 7 03:40 /star/rcf/test/dev/daq_sl302.ittf_opt/Mon/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 264696 Mar 7 03:40 /star/rcf/test/dev/daq_sl302.ittf_opt/Mon/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 727813933 Mar 8 04:13 /star/rcf/test/dev/daq_sl302.ittf_opt/Tue/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 109562289 Mar 8 04:13 /star/rcf/test/dev/daq_sl302.ittf_opt/Tue/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 366098 Mar 8 04:13 /star/rcf/test/dev/daq_sl302.ittf_opt/Tue/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 727813492 Mar 9 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Wed/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 109562227 Mar 9 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Wed/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 366098 Mar 9 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Wed/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 727814339 Mar 10 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Thu/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 109562297 Mar 10 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Thu/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 366098 Mar 10 03:42 /star/rcf/test/dev/daq_sl302.ittf_opt/Thu/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root -rw-r--r-- 1 starreco stargrid 726999394 Mar 11 04:08 /star/rcf/test/dev/daq_sl302.ittf_opt/Fri/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.event.root -rw-r--r-- 1 starreco stargrid 109180347 Mar 11 04:08 /star/rcf/test/dev/daq_sl302.ittf_opt/Fri/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.MuDst.root -rw-r--r-- 1 starreco stargrid 363560 Mar 11 04:08 /star/rcf/test/dev/daq_sl302.ittf_opt/Fri/year_2016/dAu20_production_2016/st_physics_17155014_raw_3000009.picoDst.root

ls -ltr /star/rcf/test/dev/daq_sl302.stica//year_2021/production_OO_200GeV_2021/{event,MuDst,picoDst}.root

-rw-r--r-- 1 starreco rhstar 48308496 Mar 5 04:18 /star/rcf/test/dev/daq_sl302.stica/Sat/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 16554537 Mar 5 04:18 /star/rcf/test/dev/daq_sl302.stica/Sat/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 48308519 Mar 6 04:23 /star/rcf/test/dev/daq_sl302.stica/Sun/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 16554537 Mar 6 04:23 /star/rcf/test/dev/daq_sl302.stica/Sun/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 48308490 Mar 7 04:19 /star/rcf/test/dev/daq_sl302.stica/Mon/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 16554537 Mar 7 04:19 /star/rcf/test/dev/daq_sl302.stica/Mon/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 114738936 Mar 8 04:58 /star/rcf/test/dev/daq_sl302.stica/Tue/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 22088351 Mar 8 04:58 /star/rcf/test/dev/daq_sl302.stica/Tue/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 114738987 Mar 9 04:18 /star/rcf/test/dev/daq_sl302.stica/Wed/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 22088351 Mar 9 04:18 /star/rcf/test/dev/daq_sl302.stica/Wed/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 114738937 Mar 10 04:04 /star/rcf/test/dev/daq_sl302.stica/Thu/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 22088351 Mar 10 04:04 /star/rcf/test/dev/daq_sl302.stica/Thu/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root -rw-r--r-- 1 starreco rhstar 114738978 Mar 11 05:16 /star/rcf/test/dev/daq_sl302.stica/Fri/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.MuDst.root -rw-r--r-- 1 starreco rhstar 22088351 Mar 11 05:16 /star/rcf/test/dev/daq_sl302.stica/Fri/year_2021/production_OO_200GeV_2021/st_physics_22134030_raw_4500015.picoDst.root

[starsdong]

I wonder the size increase is probably due to the StEmcRawMaker::SetMode(1) change to enable TriggerSimuMaker working in picoDst production.

I can see the increase in picoDst is primarily due to the new branches EmcTrigger and EmcPidTraits got filled. In MuDst, I see new branches are there: EmcPrs (size of 4800), EmcSmde (size 18000) and EmcSmdp (size 18000).

Zilong, if possible, could you please help check whether we can identify a way to keep the same StEvent/MuDst structure while allowing the TriggerSimuMaker to work OK with EmcCollection retrieved from EmcRawMaker?

[zlchang]

Hi Xin,

I’m a bit surprised that this is due to setting the saveALLStEvent to true in the StEmcRawMaker. The size should be same, unless there are some weird things in the offline table for BSMD and preshowers. I don’t think the gain and pedestals for these detectors are maintained.

We can hack the code so that it won’t save information for BSMD or preshowers. But this is a dangerous thing to do.

Zilong

On Mar 11, 2022, at 5:18 PM, Xin Dong @.***> wrote:



I wonder the size increase is probably due to the StEmcRawMaker::SetMode(1) change to enable TriggerSimuMaker working in picoDst production.

I can see the increase in picoDst is primarily due to the new branches EmcTrigger and EmcPidTraits got filled. In MuDst, I see new branches are there: EmcPrs (size of 4800), EmcSmde (size 18000) and EmcSmdp (size 18000).

Zilong, if possible, could you please help check whether we can identify a way to keep the same StEvent/MuDst structure while allowing the TriggerSimuMaker to work OK with EmcCollection retrieved from EmcRawMaker?

— Reply to this email directly, view it on GitHubhttps://github.com/star-bnl/star-sw/pull/314#issuecomment-1065582638, or unsubscribehttps://github.com/notifications/unsubscribe-auth/AGH7F3KSQFTZXPR62TSGEUTU7PBDLANCNFSM5PRZ3ZRQ. You are receiving this because your review was requested.Message ID: @.***>

[genevb]

I have another request, @zlchang : StTriggerSimuMaker prints a large number of lines of text to the log files (something like 7000 lines of text on the first event). Is all of this text output helpful for every job? I encountered this because some of the lines of text include the word "abort" as part of some trigger name (which I realize is legitimate such as when related to the collider's "abort gap"), and that was tripping up our job-failure detection scripts. We can make the scripts smarter about this, but do all these trigger items really need to be printed?

Thanks, -Gene

[zlchang]

Hi Gene,

It prints out the status and pedestals of barrel and Endcap towers, so they're long. If you prefer, I can make them to be printed when only in debug mode. But I don't see the word "abort" is printed for trigger simulator? Do you have a concrete example?

Zilong

[genevb]

Concrete example:

> grep -i abort /star/rcf/test/dev/daq_sl302.ittf/Tue/year_2016/AuAu200_production_2016/*log
StTriggerSimuMaker:INFO  -                   32                   0                  68Blue-or-Yellow-abort-gap                  -1                   0
StTriggerSimuMaker:INFO  -                   36                   0                   3        trgTestAbort                  -1                   0
StTriggerSimuMaker:INFO  -                   36                   3                   0               abort                  -1                   0
StTriggerSimuMaker:INFO  -                   36                   0                  22           testAbort                  -1                   0
StTriggerSimuMaker:INFO  -                   36                  22                   0               abort                  -1                   0
StTriggerSimuMaker:INFO  -                   36                  27                   0          doAbort-i0                  -1                   0

All of these nightly test datasets showed this, so perhaps it was only used as a name in 2016 and 2017:

AuAu200_production_2016 dAu200_production_2016 dAu20_production_2016 dAu39_production_2016 dAu62_production_2016 54GeV_production_2017

-Gene