examples/imsim-user-instcat.yaml crashes

[esheldon]

(imsim) [esheldon@rivendell examples (main)] galsim imsim-user-instcat.yaml 
Reading config file imsim-user-instcat.yaml
Overriding default configuration file with /home/esheldon/miniconda3/envs/imsim/share/eups/Linux64/dustmaps_cachedata/gbb0a0c949e+81bc2a20b4/config/.dustmapsrc
Reading config file /home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/imsim/config/imsim-config-instcat.yaml
Reading config file /home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/imsim/config/imsim-config.yaml
TreeRing file /home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/imsim/data/tree_ring_data/tree_ring_parameters_2018-04-26.txt will be used.
Traceback (most recent call last):
  File "/home/esheldon/miniconda3/envs/imsim/bin/galsim", line 10, in <module>
    sys.exit(run_main())
             ^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/main.py", line 227, in run_main
    main(sys.argv[1:])
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/main.py", line 222, in main
    process_config(all_config, args, logger)
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/main.py", line 200, in process_config
    Process(config, logger, njobs=args.njobs, job=args.job, new_params=new_params,
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/config/process.py", line 295, in Process
    config_out = BuildFiles(nfiles, config, file_num=start, logger=logger,
                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/config/output.py", line 108, in BuildFiles
    builder.setup(output, config, file_num, logger)
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/imsim/ccd.py", line 30, in setup
    seed = galsim.config.SetupConfigRNG(base, logger=logger)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/config/util.py", line 538, in SetupConfigRNG
    seed, rng = ParseRandomSeed(image, 'random_seed', config, seed_offset)
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/config/util.py", line 411, in ParseRandomSeed
    seed = ParseValue(config, param_name, base, int)[0]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/galsim/config/value.py", line 143, in ParseValue
    val_safe = generate_func(param, base, value_type)
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/esheldon/miniconda3/envs/imsim/lib/python3.11/site-packages/imsim/opsim_data.py", line 373, in OpsimData
    val = value_type(meta.get(field))
                     ^^^^^^^^
AttributeError: 'NoneType' object has no attribute 'get'

[cwwalter]

This I haven't seen. I just tried on my laptop after doing a git pull and this is also done as part of our CI so it should always be checked before we merge.

Did you make any changes to the file?

Has anyone else seen this and can offer advice?

The closest thing I have seen to this in the past is when people had different versions of galsim etc that they had in .local that were being used instead.

(p.s. I will be in and out today)

[esheldon]

latest imsim main, no modifications to file

[esheldon]

I followed the conda installation instructions exactly, using a new environment.

The crash occurs with main and v1.6.0rc2 of skyCatalogs

[cwwalter]

By "conda" I assume you mean method #2 with @beckermr 's stackvana.

Has anyone using stackvana also seen this?

It's what the CI uses (https://github.com/LSSTDESC/imSim/blob/main/.github/workflows/ci.yml) and I can't see what might be going on unless maybe something changed in the conda distribution?

Erin can you double check that your python path etc doesn't have any other versions of galsim or imsim in it?

[esheldon]

Its using the galsim/imsim from the environment I created following the instructions

[cwwalter]

Yes, understood. The reason I asked is that I have personally seen situations where even though parts of GalsIm are being used as I expected, other parts were being run by other GalSim code in my python path. It doesn't look like that is what is happening here, but I thought I would ask.

@rmjarvis The code is complaining about this function:

https://github.com/LSSTDESC/imSim/blob/345a7cc3307ce44b0e560efc0a68344523c18dbe/imsim/opsim_data.py#L363

Does the complaint from the galsim config system mean it can't find an opsim_data entry?

[rmjarvis]

It looks like it's not reading the instcat file. When I run that example file, here is the output:

$ galsim imsim-user-instcat.yaml 
Reading config file imsim-user-instcat.yaml
Overriding default configuration file with /Users/Mike/miniforge3/envs/lsst-scipipe-7.0.1/share/eups/Darwin/dustmaps_cachedata/gbb0a0c949e+81bc2a20b4/config/.dustmapsrc
Reading config file /Users/Mike/LSSTDESC/imSim/imsim/config/imsim-config-instcat.yaml
Reading config file /Users/Mike/LSSTDESC/imSim/imsim/config/imsim-config.yaml
Reading visit info from instance catalog /Users/Mike/LSSTDESC//imSim/examples/example_instance_catalog.txt
Done reading meta information from instance catalog
Building atmospheric PSF
TreeRing file /Users/Mike/LSSTDESC/imSim/imsim/data/tree_ring_data/tree_ring_parameters_2018-04-26.txt will be used.
Reading instance catalog /Users/Mike/LSSTDESC//imSim/examples/example_instance_catalog.txt
Total objects in file = 1998
Found 1079 objects potentially on image
Start file 0 = output/eimage_00398414-0-r-R22_S11-det094.fits
Start batch 1/10 with 1 objects [0, 1)
Start batch 2/10 with 1 objects [1, 2)
Start batch 3/10 with 1 objects [2, 3)
Start batch 4/10 with 1 objects [3, 4)
Start batch 5/10 with 1 objects [4, 5)
Start batch 6/10 with 1 objects [5, 6)
Start batch 7/10 with 1 objects [6, 7)
Start batch 8/10 with 1 objects [7, 8)
Start batch 9/10 with 1 objects [8, 9)
Start batch 10/10 with 1 objects [9, 10)
Making amplifier images
Writing amplifier images to output/amp_00398414-0-r-R22_S11-det094.fits.fz
File 0 = output/eimage_00398414-0-r-R22_S11-det094.fits: time = 18.883040 sec
Done building files

Note particularly the lines starting "Reading visit info..." and "Done reading meta information". These are missing from Erin's output. I can't quite trace through the code path that causes this to get skipped. But maybe if the file is missing or corrupt, it would do that? (Obviously we should emit an error in that case, but it's possible we missed this edge case.)

[cwwalter]

Erin do you have IMSIM_HOME set as in the instructions? That example file uses it so it can be run from anywhere. If it wasn't set it might not be able find the file.

[esheldon]

That is not in the instructions for the conda based install

Our system is down for maintenance, I can try this when its back up

[esheldon]

I was able to get this to work for the conda based install by setting IMSIM_HOME to the directory that contains the imSim git repo

Note that the conda instructions do not indicate that IMSIM_HOME must be set. I made a separate issue for the docs

[cwwalter]

Thanks, Erin. Apologies, I have it in the CMVFS section and it is used in the bottom where it tells how to test it, but it looks like I forgot to change the section for stackvana when I merged that previous text in.

I'll update it. Thanks for the other issue. Closing this one.