[NERSC] Need set up for running LSST alerts

[RickKessler]

Description A clear and concise description of what the issue is.

Need help to import these packages on Cori for writing LSST alerts import lsst.alert.packet
from fastavro import writer, reader
(ideally, part of /global/cfs/cdirs/lsst/groups/SN/software/setup_sn_all.sh)

Choose all applicable topics by placing an 'X' between the [ ]:

• [ ] jupyter
• [ ] jupyter terminal
• [ ] Cori interactive command line
• [ ] Batch jobs
• [ ] python
• [ ] CSCRATCH
• [ x] Community File System
• [ ] HPSS (tape)
• [ ] Data transfer and Globus
• [ ] New User Account or account access problems

To Reproduce Steps to reproduce the behavior:

1. Go to '...'
2. Click on '....'
3. Scroll down to '....'
4. See error

Screenshots If applicable, add screenshots to help explain your problem.

[RickKessler]

The import request is from Renee's starterKit, https://github.com/LSSTDESC/plasticc_alerts/tree/main/Examples/starterkit (see write_single_object.py)

[gnarayan]

Tagging @reneehlozek and @alexandergagliano

[RickKessler]

To check if import works,

cd /global/cfs/cdirs/lsst/groups/SN/snana/SURVEYS/LSST/USERS/kessler/makeDataFiles_devel

./RUNTEST_DDF_ALERT Init data_snana_folder class. Base init Read data version = LSST_DDF_MODEL11 from data_foler = /global/cscratch1/sd/kessler/SNANA_LSST_SIM/PLASTICC/LSST_DDF_MODEL11 Read 1382 events from LSST_DDF_NONIa-0001_HEAD.FITS Traceback (most recent call last): File "/global/homes/k/kessler/SNANA/util/makeDataFiles/makeDataFiles_main.py", line 211, in program.read_data_driver() File "/global/u2/k/kessler/SNANA/util/makeDataFiles/makeDataFiles_base.py", line 515, in read_data_driver lsst_alert.write_event_lsst_alert(args, self.config_data, NameError: name 'lsst_alert' is not defined

[heather999]

/global/cfs/cdirs/lsst/groups/SN/software/setup_sn_all.sh sets up a lot of different things, including the sn-py conda environment. To use the DM stack and code like lsst_alert, you need to be in DM's conda environment. I don't know that setting up one environment with all of the software you need for your full workflow can function in this case. I think we have to define what software you need for each step of your processing chain or at least define the necessary software for this particular step. Is there a good way to figure that out?
Is having just the DM stack weekly available enough? Is there a particular DM weekly you would need? Right now, stack-weekly is "stuck" on w_2021_19 (due to changes in the availability of lsst_sims), is that recent enough? Or would you prefer a more recent DM stack? There are other ways to set up the DM stack - so we can get whatever version you feel is best suited to the work you want to do.

So I'm looking for a few answers..

• version of DM stack you need
• Can we define the software you need for this step?
• Is it time to break down setup_sn_all a bit to set up just the software necessary for various steps in your workflow?

[heather999]

I'm working through the example Rick posted above trying to run RUNTEST_DDF_ALERT I'll see what I can come up with :)

[RickKessler]

Unfortunately I cannot answer the questions about, but hopefully on Monday (first day of sprint) we can get input from Renee and other DM experts. I would naively assume that we need only a very small part of DM (that writes alerts in Avro format), but no idea which part.

[heather999]

I hacked until I could make it work..now for some comments and then a recipe

• It looks like we need both snana and DM stack to be available to get the scripts referenced by RUNTEST_DDF_ALERT to run
• DM stack has its own copy of cfitsio 3.470 installed via conda - it's the same version that is available on Cori that SNANA was built against. For now I'm leaving the NERSC cfitsio in the environment so SNANA will be happy - it is possible to override that and point to DM stack's copy. It would likely be best to clean this up. Given that it's the same version of cfitsio, I'm going to blindly assume this is ok.
• SNANA needs ROOT to be available, so I left that set up.
• The only package from sn-py that seems necessary is coloredlogs, so I installed it in a temporary shared location for now (/global/common/software/lsst/common/miniconda/sn-sprint-oct21 against the DM stack conda environment and updated PYTHONPATH to point to it.

Here are some more details:

• copied setup_sn_all.sh and renamed it setup_sn_some.sh and just commented out a bunch of stuff except what was absolutely necessary for things to run. Primarily I added a call to set up a recent DM stack weekly, w_2021_40 which is the version targeted for the upcoming v23 release.
• Copied /global/cfs/cdirs/lsst/groups/SN/snana/SURVEYS/LSST/USERS/kessler/makeDataFiles_devel into my own area
• Copied ~kessler/SNANA to my own area - primarily because I didn't want to risk running in your $HOME.
• Updated ./RUNTEST_DDF_ALERT to point to my copy of the scripts
• Setup the DM stack w_2021_40 environment and then installed coloredlogs via pip install --root <myPath> coloredLogs
• Updated my setup_sn_some.sh to set PYTHONPATH pointing to my copy of coloredlogs

• ran source setup_sn_some.sh and then RUNTEST_DDF_ALERT and received the following output:

  (lsst-scipipe-0.7.0-ext) heatherk@cori02:/global/cscratch1/sd/heatherk/sprint-sn
  /makeDataFiles_devel> ./RUNTEST_DDF_ALERT
  Init data_snana_folder class.
  Base init
  Read data version = LSST_DDF_MODEL11
  from data_foler = /global/cscratch1/sd/kessler/SNANA_LSST_SIM/PLASTICC/LSST_DDF_MODEL11
   Read 1382 events from LSST_DDF_NONIa-0001_HEAD.FITS
  xxx SNID -> diaObjectId 
  xxx RA -> ra 
  xxx DEC -> decl 
  xxx MWEBV -> mwebv 
  xxx MWEBV_ERR -> mwebv_err 
  xxx REDSHIFT_HELIO -> z_final 
  xxx REDSHIFT_HELIO_ERR -> z_final_err 
  xxx NOBS -> nobs 
  
  xxx NOBS=351  diasrc = 
  {'diaObjectId': '199387', 'ra': 352.711273, 'decl': -63.823658, 'mwebv': 0.023682913, 'mwebv_err': 0.0011841457, 'z_final': 0.32167563, 'z_final_err': 0.001, 'nobs': 351, 'midPointTai': 60624.2132, 'filterName': 'Y', 'apFlux': -7.7372093, 'apFluxErr': 7.4942503}

My copy of setup_sn_some.sh is in: /global/cscratch1/sd/heatherk/sprint-sn/setup_sn_some.sh I think you guys could just take, clean it up as desired and it might work as you need it to.

Longer term, we should think about if this use case of setting up a SN python environment should generally include the DM stack. Or if you need two environments, one with and another without DM stack? The downside of pulling in the DM stack is that we are then tied to their versions of some packages and that can be just fine, and in other cases, groups may find thatrestrictive. It may depend on the various steps of the processing chain but we really need to talk about that so I can offer a useful suggestion.

[RickKessler]

I ran "setup_sn_some.sh" and was able to run the above test script. I think this temporary setup will be ok for this broker test to write alerts. Longer term, however, we will need "data-access" tools that work with SNANA and other analysis codes.

[heather999]

That's good news. We should plan to talk in more detail about how we can set up "data-access" tools that work with SNANA & other analysis codes. Pinging @JoanneBogart. We should arrange a discussion about that to try to start planning next steps.

[RickKessler]

the "sn_some" setup worked well this week, and we were able to integrate writing Avro alerts into a submit_batch framework. One minor issue is that it's easy to forget doing the special setup, so I am looking for a stable ENV to automatically check before launching jobs. I examined "printenv" and found $SETUP_LSST_DISTRIB, but wondering if there is a more reliable ENV to check.

[heather999]

We can add our own ENV variable. You're right the *_LSST_DISTRIB variables are not going to be reliable enough. It could be something as simple as DESC_SN_ALERT_ENV and it could store a version number so you have some sense of which "version" of the environment you're working with. If it is useful to have different environments for different portions of your processing - we could define some naming & versioning convention for the different environments.

[RickKessler]

Is there something more generic that says DM stack is loaded? The issue isn't really about knowing which version, but remembering to setup DM stack for processes like making alerts, and eventually reading the data base. I'd like to add a module in the codes to immediately remind when DM stack is not setup. In short, trying to avoid a separate ENV for each DM-related task.

[heather999]

ah ok you could try checking either LSST_CONDA_ENV_NAME or LSST_STACK_VERSION either should indicate that the DM stack has been set up.

[heather999]

@RickKessler do we want to try again to have you attend an upcoming DESC CO meeting - we meet this Wednesday, Nov 3rd at 9:10 AM PT and then every other Wednesday? Or we could try to chat about environments and data access at an upcoming TD session?

[RickKessler]

yes, I can attend the Wed CO meeting.