zhux97
commented
1 year ago Please find some detail about this PR in the following slides https://drupal.star.bnl.gov/STAR/system/files/embedding_status_20230226_PR498_0.pdf
Closed zhux97 closed 1 year ago
zhux97
commented
1 year ago Please find some detail about this PR in the following slides https://drupal.star.bnl.gov/STAR/system/files/embedding_status_20230226_PR498_0.pdf
zhux97
commented
1 year ago Hi Gene, thanks for bringing up these issues. For my educations, are these issues affecting the official real data productions? or only embedding productions?
On Mar 18, 2023, at 02:32, Gene Van Buren @.***> wrote:
@genevb commented on this pull request.
I've been asked by @starsdong to review this, but I've avoided commenting on other such PRs for reasons that are not best discussed here.
The changes to StTpcRTSHitMaker codes in this commit do two things of note to me:
• The modifications made in February 2020 to address Coverity and compiler issues have been removed as if they were to be ignored. • The entire maxHitsPerSector and maxBinZeroHits portions of code introduced long ago have been removed, which can potentially lead to different physics results as these were designed to exclude problematic events from reconstruction. This is essentially a modification of the reconstruction code from the version used in official productions. — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.
genevb
commented
1 year ago Hi Gene, thanks for bringing up these issues. For my educations, are these issues affecting the official real data productions? or only embedding productions? …
As a generalization, the maxHitsPerSector and maxBinZeroHits cuts are in use in official productions. Specific to SL20c, which was used for official BFC productions of datasets only in Runs 17 and 18, the default cut values are in place and no study was performed to see if there were problematic events that the cuts helped us handle. That does not imply that there were no such problematic events, as the cuts would have done their job without much fanfare (only a minor notice in the production log files), and the potential is there for jobs to get hung on tracking for those events. There is also the possibility that not a single event was affected by the cuts for Runs 17 and 18 as the problematic events were more prominent in low energy collisions.
I don't understand why we would make an effort to specifically modify the code to remove these cuts.
-Gene
zhux97
commented
1 year ago Hi Gene, thanks for the detailed information. This portion of code indeed were removed in the latest version of StTpcRTSHitMaker. But it is still in the StTpcHitMaker. I checked one of the log file from the real data bfc production chain (which reads the online hits), but did not find the StTpcRTSHitMaker in the maker list. So I guess it is safe for real data production, isn't it? For embedding production, this piece of code will reject some high occupancy events for processing, otherwise the embedding chain will crash, right? We indeed find some small fraction of embedding jobs (on some certain daq files, less than 10 out of 1000 files) have higher probability to fail during tracking, which might be related to the missing protection in this StTpcRTSHitMaker code. Thanks!
On Mar 21, 2023, at 00:07, Gene Van Buren @.***> wrote:
Hi Gene, thanks for bringing up these issues. For my educations, are these issues affecting the official real data productions? or only embedding productions? …
As a generalization, the maxHitsPerSector and maxBinZeroHits cuts are in use in official productions. Specific to SL20c, which was used for official BFC productions of datasets only in Runs 17 and 18, the default cut values are in place and no study was performed to see if there were problematic events that the cuts helped us handle. That does not imply that there were no such problematic events, as the cuts would have done their job without much fanfare (only a minor notice in the production log files), and the potential is there for jobs to get hung on tracking for those events. There is also the possibility that not a single event was affected by the cuts for Runs 17 and 18 as the problematic events were more prominent in low energy collisions.
I don't understand why we would make an effort to specifically modify the code to remove these cuts.
-Gene
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.
genevb
commented
1 year ago Hi, Xianglei.
Yes, I think you understand correctly. StTpcHitMaker is generally used in the real data production, and StTpcRTSHitMaker is used in the embedding reconstruction. The two should be configured to skip the same data. And, yes, I agree that you may be seeing symptoms in SL21d where this code was similarly modified in PR #493 , and in SL19b modified by PR #481 .
And, yes, PR #417 from a few months ago did erase the code from the current version of StTpcRTSHitMaker for new libraries (currently only SL23a and DEV, but we're about to freeze SL23b with this mess as well).
-Gene
On Mar 21, 2023, at 2:22 AM, zhux97 @.***> wrote:
Hi Gene, thanks for the detailed information. This portion of code indeed were removed in the latest version of StTpcRTSHitMaker. But it is still in the StTpcHitMaker. I checked one of the log file from the real data bfc production chain (which reads the online hits), but did not find the StTpcRTSHitMaker in the maker list. So I guess it is safe for real data production, isn't it? For embedding production, this piece of code will reject some high occupancy events for processing, otherwise the embedding chain will crash, right? We indeed find some small fraction of embedding jobs (on some certain daq files, less than 10 out of 1000 files) have higher probability to fail during tracking, which might be related to the missing protection in this StTpcRTSHitMaker code. Thanks!
On Mar 21, 2023, at 00:07, Gene Van Buren @.***> wrote: I don't understand why we would make an effort to specifically modify the code to remove these cuts.
klendathu2k
commented
1 year ago Just a random thought / question here. (Or more).
Do we know what the source of the high hit multiplicity is in the problematic events which these cuts are meant to address? In other words, are they from the primary collision or from pileup? I would presume the latter. But if it really is the former, there is a bias introduced in high multiplicity events.
When I see the statement "different reconstruction code for real data and embedding" I get nervous. Is my understanding correct that the StTpcHitMaker creates hits from clusters found online, while the StTpcRTSHitMaker must also run the clustering code on the raw TPC data (plus any simulation "hits")?
If so...
Wouldn't it be better if cluster finding were split into a separate cluster maker, and then we only have one maker to create TPC hits? (Recognizing we may be past the point where the effort is worth it).
Jason
On 2023-03-21 10:46, Gene Van Buren wrote:
Hi, Xianglei.
Yes, I think you understand correctly. StTpcHitMaker is generally used in the real data production, and StTpcRTSHitMaker is used in the embedding reconstruction. The two should be configured to skip the same data. And, yes, I agree that you may be seeing symptoms in SL21d where this code was similarly modified in PR #493 , and in SL19b modified by PR #481 .
And, yes, PR #417 from a few months ago did erase the code from the current version of StTpcRTSHitMaker for new libraries (currently only SL23a and DEV, but we're about to freeze SL23b with this mess as well).
-Gene
On Mar 21, 2023, at 2:22 AM, zhux97 @.***> wrote:
Hi Gene, thanks for the detailed information. This portion of code indeed were removed in the latest version of StTpcRTSHitMaker. But it is still in the StTpcHitMaker. I checked one of the log file from the real data bfc production chain (which reads the online hits), but did not find the StTpcRTSHitMaker in the maker list. So I guess it is safe for real data production, isn't it? For embedding production, this piece of code will reject some high occupancy events for processing, otherwise the embedding chain will crash, right? We indeed find some small fraction of embedding jobs (on some certain daq files, less than 10 out of 1000 files) have higher probability to fail during tracking, which might be related to the missing protection in this StTpcRTSHitMaker code. Thanks!
On Mar 21, 2023, at 00:07, Gene Van Buren @.***> wrote: I don't understand why we would make an effort to specifically modify the code to remove these cuts.
-- Reply to this email directly, view it on GitHub [1], or unsubscribe [2]. You are receiving this because your review was requested.Message ID: @.***>
Links:
[1] https://github.com/star-bnl/star-sw/pull/498#issuecomment-1477972041 [2] https://github.com/notifications/unsubscribe-auth/ANL4LVAHSOVHARE3LUU4AJ3W5G5M3ANCNFSM6AAAAAAVIL2XAI
genevb
commented
1 year ago My last post, responding to @klendathu2k .... this may not be a complete list.
https://drupal.star.bnl.gov/STAR/blog/genevb/2010/aug/26/limiting-production-events-with-large-event-counts) https://drupal.star.bnl.gov/STAR/blog/genevb/2010/aug/31/sequential-excessively-large-tpc-events https://drupal.star.bnl.gov/STAR/blog/genevb/2011/mar/02/rt-ticket-2099 https://drupal.star.bnl.gov/STAR/blog/genevb/2011/jun/08/issue-2131-another-big-event-class https://drupal.star.bnl.gov/STAR/blog/genevb/bes-ii-events-large-hit-counts
-Gene
zhux97
commented
1 year ago Hi Gene, Jason, Dmitri, the StTpcRTSHitMaker.cxx has been updated with Xin's PR #510. Now the high occupancy protection code is restored.
starsdong
commented
1 year ago Dmitri, could you please take a look at the CI build? It seems to be related to the environment setup? Thanks
plexoos
commented
1 year ago Yes, will do
starsdong
commented
1 year ago Hi Dmitri, I wonder whether you have any findings from the CI build why it failed. Would be great this can be solved asap and we can proceed with the embedding production. Thanks
starsdong
commented
1 year ago Hi Dmitri, I am curious whether you have any findings on the failure of the CI build in this PR. The embedding team is eagerly waiting for this to be merged. Thanks
plexoos
commented
1 year ago The changes in this PR submitted by Xianglei pass the tests (as expected). It is somewhat unexpected and worth noting here that the SL20c branch also had to be patched to include the PicoVtxFXT option introduced after SL20c in order for the tests to work on Run 20 data samples.
zhux97
commented
1 year ago Dmitri, thanks a lot for more patches and merging this PR.
Gene, could you help to deploy the latest SL20c? Both CVMFS and AFS are fine. Thanks!
On May 10, 2023, at 20:10, Dmitri Smirnov @.***> wrote:
Merged #498 into SL20c.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you authored the thread.
genevb
commented
1 year ago Once Dmitri tags the library, I'll build it.
Thanks, -Gene
On May 10, 2023, at 11:25 PM, zhux97 @.***> wrote: Gene, could you help to deploy the latest SL20c? Both CVMFS and AFS are fine. Thanks!
plexoos
commented
1 year ago Hi Gene, actually I created the SL20c_3 tag yesterday, just forgot to announce.
genevb
commented
1 year ago Hi, Dmitri.
It looks like the star-mcgen repo wasn't given the SL20c_3 tag yet.
-Gene
On May 11, 2023, at 7:48 AM, Dmitri Smirnov @.***> wrote:
Hi Gene, actually I created the SL20c_3 tag yesterday, just forgot to announce.
plexoos
commented
1 year ago Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
genevb
commented
1 year ago Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k , @zhux97 : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
zhux97
commented
1 year ago Gene, I have no idea about 'star-mcgen'. What codes are stored in this repo ? thanks.
On May 12, 2023, at 06:03, Gene Van Buren @.***> wrote:
Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k , @zhux97 : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
genevb
commented
1 year ago Hi, Xianglei
Event generators like HIJING, PYTHIA, URQMD, etc. https://github.com/star-bnl/star-mcgen/tree/main/StRoot/StarGenerator
-Gene
On May 12, 2023, at 4:43 AM, zhux97 @.***> wrote:
Gene, I have no idea about 'star-mcgen'. What codes are stored in this repo ? thanks.
On May 12, 2023, at 06:03, Gene Van Buren @.***> wrote:
Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k , @zhux97 : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
zhux97
commented
1 year ago Hi Gene,
I see. Currently, we have only single-particle embedding requests for run18 isobar and run17 pp510. So we can keep the star-mcgen as it is for now.
Thanks, Xianglei
On May 12, 2023, at 20:14, Gene Van Buren @.***> wrote:
Hi, Xianglei
Event generators like HIJING, PYTHIA, URQMD, etc. https://github.com/star-bnl/star-mcgen/tree/main/StRoot/StarGenerator
-Gene
On May 12, 2023, at 4:43 AM, zhux97 @.***> wrote:
Gene, I have no idea about 'star-mcgen'. What codes are stored in this repo ? thanks.
On May 12, 2023, at 06:03, Gene Van Buren @.***> wrote:
Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k , @zhux97 : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
klendathu2k
commented
1 year ago Just looking at the differences between the current version of mcgen and SL20c_2...
https://github.com/star-bnl/star-mcgen/compare/SL20c_2...main
1) Pythia 8.3 was added / allows HI simulations using the pythia 8 model(s) 2) Another common block was exposed in the pythia6 interface -- supports upsilon simulations 3) The pygive routine was exposed in pythia6 -- this is a geant4star modification
Unless there is an open request that uses pythia6 to generate upsilons, then I think we are good with SL20c_2.
Jason
On 2023-05-11 18:03, Gene Van Buren wrote:
Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k [1] , @zhux97 [2] : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
-- Reply to this email directly, view it on GitHub [3], or unsubscribe [4]. You are receiving this because you were mentioned.Message ID: @.***>
Links:
[1] https://github.com/klendathu2k [2] https://github.com/zhux97 [3] https://github.com/star-bnl/star-sw/pull/498#issuecomment-1544746202 [4] https://github.com/notifications/unsubscribe-auth/ANL4LVFRJXXMSXFKMTJGVTDXFVO3FANCNFSM6AAAAAAVIL2XAI
genevb
commented
1 year ago @plexoos , the suggestion from Jason was to re-tag SL20c_2 as SL20c_3, but the SL20c_3 tag still points to a more recent version of star-mcgen. I think one of these three things needs to happen...
1) Make an argument for keeping the SL20c_3 tag at the latest version.
2) Remove the current SL20c_3 tag and re-apply it to SL20c_2.
3) Or if that is not straightforward, just create an SL20c_4 tag that is the same as SL20c_3 in star-sw and the same as SL20c_2 in star-mcgen.
Thanks, -Gene
On May 12, 2023, at 9:37 AM, klendathu2k @.***> wrote:
Just looking at the differences between the current version of mcgen and SL20c_2...
https://github.com/star-bnl/star-mcgen/compare/SL20c_2...main
1) Pythia 8.3 was added / allows HI simulations using the pythia 8 model(s) 2) Another common block was exposed in the pythia6 interface -- supports upsilon simulations 3) The pygive routine was exposed in pythia6 -- this is a geant4star modification
Unless there is an open request that uses pythia6 to generate upsilons, then I think we are good with SL20c_2.
Jason
On 2023-05-11 18:03, Gene Van Buren wrote:
Right. I've tagged the latest commit on main with SL20c_3 now. I hope this is what we want.
@klendathu2k [1] , @zhux97 [2] : is the latest version of star-mcgen what we want? Or just continue with what it was before now (tagged as SL20c_2)?
-- Reply to this email directly, view it on GitHub [3], or unsubscribe [4]. You are receiving this because you were mentioned.Message ID: @.***>
Links:
[1] https://github.com/klendathu2k [2] https://github.com/zhux97 [3] https://github.com/star-bnl/star-sw/pull/498#issuecomment-1544746202 [4] https://github.com/notifications/unsubscribe-auth/ANL4LVFRJXXMSXFKMTJGVTDXFVO3FANCNFSM6AAAAAAVIL2XAI — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
plexoos
commented
1 year ago SL20c_3 now points to the same commit as SL20c_2
genevb
commented
1 year ago SZL20 (SL20c_3) has been deployed on AFS. I'll start getting it built on CVMFS next.
-Gene
zhux97
commented
1 year ago Gene, thanks! I have submitted a test.
On May 18, 2023, at 09:11, Gene Van Buren @.***> wrote:
SZL20 (SL20c_3) has been deployed on AFS. I'll start getting it built on CVMFS next.
-Gene — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
zhux97
commented
1 year ago Gene, the embedding script breaks in Sti,
StiMaker:INFO - TAttr::Found : usessd = 1
StiMaker:INFO - We are StiMasterDetectorBuilder::build() and will clone the geometry if exists StiMaker:INFO - Timestamp check - Date/Time from chain is 20180426/83442 StiMaker:INFO - Create a clone of VmcGeometry for reconstruction StiMaker:ERROR - TGeoManager::CloseGeometry : you MUST call SetTopVolume() first ! StiMaker:INFO - StiMasterDetectorBuilder::build() : Started StiMaker:INFO - StiMasterDetectorBuilder::build() : Calling Group Builder named: StarBuilderBuilder StiMaker:INFO - StiStarDetectorBuilder::useVMCGeometry : No PIPC volume
root4star: .sl73_gcc485/obj/StRoot/Sti/Star/StiStarDetectorBuilder.cxx:30: virtual void StiStarDetectorBuilder::useVMCGeometry(): Assertion `0' failed. Abort
The full log file is /star/u/starembd/zhux/embd/P20ic/Proton_Isobar_2018.testAFS/test.log
On May 18, 2023, at 11:12, Xianglei Zhu @.***> wrote:
Gene, thanks! I have submitted a test.
On May 18, 2023, at 09:11, Gene Van Buren @.***> wrote:
SZL20 (SL20c_3) has been deployed on AFS. I'll start getting it built on CVMFS next.
-Gene — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
genevb
commented
1 year ago Thanks for the quick test, Xianglei. Argh - the same issue each time I try to rebuild without building from scratch. That's annoying. OK....rebuilding from scratch, but probably not available until tomorrow then.
-Gene
On May 17, 2023, at 11:58 PM, zhux97 @.***> wrote:
Gene, the embedding script breaks in Sti,
StiMaker:INFO - TAttr::Found : usessd = 1
StiMaker:INFO - We are StiMasterDetectorBuilder::build() and will clone the geometry if exists StiMaker:INFO - Timestamp check - Date/Time from chain is 20180426/83442 StiMaker:INFO - Create a clone of VmcGeometry for reconstruction StiMaker:ERROR - TGeoManager::CloseGeometry : you MUST call SetTopVolume() first ! StiMaker:INFO - StiMasterDetectorBuilder::build() : Started StiMaker:INFO - StiMasterDetectorBuilder::build() : Calling Group Builder named: StarBuilderBuilder StiMaker:INFO - StiStarDetectorBuilder::useVMCGeometry : No PIPC volume
root4star: .sl73_gcc485/obj/StRoot/Sti/Star/StiStarDetectorBuilder.cxx:30: virtual void StiStarDetectorBuilder::useVMCGeometry(): Assertion `0' failed. Abort
The full log file is /star/u/starembd/zhux/embd/P20ic/Proton_Isobar_2018.testAFS/test.log
On May 18, 2023, at 11:12, Xianglei Zhu @.***> wrote:
Gene, thanks! I have submitted a test.
On May 18, 2023, at 09:11, Gene Van Buren @.***> wrote:
SZL20 (SL20c_3) has been deployed on AFS. I'll start getting it built on CVMFS next.
-Gene — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
genevb
commented
1 year ago Hi, Xianglei
Please give your test another try on AFS. I will rebuild from scratch on CVMFS.
Thanks, -Gene
On May 18, 2023, at 12:39 AM, Gene Van Buren @.***> wrote:
Thanks for the quick test, Xianglei. Argh - the same issue each time I try to rebuild without building from scratch. That's annoying. OK....rebuilding from scratch, but probably not available until tomorrow then.
-Gene
On May 17, 2023, at 11:58 PM, zhux97 @.***> wrote:
Gene, the embedding script breaks in Sti,
StiMaker:INFO - TAttr::Found : usessd = 1
StiMaker:INFO - We are StiMasterDetectorBuilder::build() and will clone the geometry if exists StiMaker:INFO - Timestamp check - Date/Time from chain is 20180426/83442 StiMaker:INFO - Create a clone of VmcGeometry for reconstruction StiMaker:ERROR - TGeoManager::CloseGeometry : you MUST call SetTopVolume() first ! StiMaker:INFO - StiMasterDetectorBuilder::build() : Started StiMaker:INFO - StiMasterDetectorBuilder::build() : Calling Group Builder named: StarBuilderBuilder StiMaker:INFO - StiStarDetectorBuilder::useVMCGeometry : No PIPC volume
root4star: .sl73_gcc485/obj/StRoot/Sti/Star/StiStarDetectorBuilder.cxx:30: virtual void StiStarDetectorBuilder::useVMCGeometry(): Assertion `0' failed. Abort
The full log file is /star/u/starembd/zhux/embd/P20ic/Proton_Isobar_2018.testAFS/test.log
zhux97
commented
1 year ago Hi Gene,
Thanks! But now the script breaks at the beginning...
Executing bfcMixer_Tpx(10, "/gpfs01/star/embed/daq/2018/isobar/st_physics_adc_19116012_raw_3000008.daq", "/gpfs01/star/embed/tags/2018/isobar/st_physics_adc_19116012_raw_3000008.tags.root", 0, 2.5, -1.0, 1.0, -35.0, 35.0, 2.0, 14, 0.05, triggers, "P20icIsobar200", "FlatPt", 0, "st_physics_adc_19116012_raw_3000008.fzd"); ...
ROOT 5.34/38 @.***, Mar 12 2018, 15:49:39 on linux)
CINT/ROOT C/C++ Interpreter version 5.18.00, July 2, 2010
Type ? for help. Commands must be C++ statements.
Enclose multiple statements between { }.
Error in
The full log is /star/u/starembd/zhux/embd/P20ic/Proton_Isobar_2018.testAFS/test.log2
On May 18, 2023, at 21:48, Gene Van Buren @.***> wrote:
Hi, Xianglei
Please give your test another try on AFS. I will rebuild from scratch on CVMFS.
Thanks, -Gene
On May 18, 2023, at 12:39 AM, Gene Van Buren @.***> wrote:
Thanks for the quick test, Xianglei. Argh - the same issue each time I try to rebuild without building from scratch. That's annoying. OK....rebuilding from scratch, but probably not available until tomorrow then.
-Gene
On May 17, 2023, at 11:58 PM, zhux97 @.***> wrote:
Gene, the embedding script breaks in Sti,
StiMaker:INFO - TAttr::Found : usessd = 1
StiMaker:INFO - We are StiMasterDetectorBuilder::build() and will clone the geometry if exists StiMaker:INFO - Timestamp check - Date/Time from chain is 20180426/83442 StiMaker:INFO - Create a clone of VmcGeometry for reconstruction StiMaker:ERROR - TGeoManager::CloseGeometry : you MUST call SetTopVolume() first ! StiMaker:INFO - StiMasterDetectorBuilder::build() : Started StiMaker:INFO - StiMasterDetectorBuilder::build() : Calling Group Builder named: StarBuilderBuilder StiMaker:INFO - StiStarDetectorBuilder::useVMCGeometry : No PIPC volume
root4star: .sl73_gcc485/obj/StRoot/Sti/Star/StiStarDetectorBuilder.cxx:30: virtual void StiStarDetectorBuilder::useVMCGeometry(): Assertion `0' failed. Abort
The full log file is /star/u/starembd/zhux/embd/P20ic/Proton_Isobar_2018.testAFS/test.log
— Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned.
genevb
commented
1 year ago Having started from scratch, I'm not sure what is going wrong there on AFS.
I also tried starting from scratch and building it on CVMFS earlier today. Maybe you can try your test there, Xianglei, and see if at least that is working?
Thanks, -Gene
Patches needed for Run18 Isobar (st_physics) embedding are assembled. The patched codes are extracted from PR #466, PR #481.