Dithering

[mcclearyj]

Implemented several changes to mock_superBIT_data.py, most notably adding a random dither (between 0-100 pixeks) to each exposure about the nominal RA/Dec center of the observation. A real star catalog is also sampled instead of the previous ~semi-analytic flux distributions.

Also, a small change to medsmaker_mocks.py was added to cut out entries in the MEDS files with zero cutouts.

[mcclearyj]

I've gone @sweverett's comments in the PR and made a few suggestions for resolving the issues noted in make_annular_catalog.py/annular_jmac.py.

[mcclearyj]

The latest commits (d9970d4 & 1165c49) have introduced a bug so now pipe_test.py doesn't run to completion. See the error message below. Any thoughts?

Running module metacal

config:
{'end': 2500, 'meds_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outfile': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits'}

python /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits -outdir=/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test -end=2500 -n=8 --vb

--plots=False; no per-object diagnostic plots will be generated
MEDS file: /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits
index start, end: None, 2500
outfile: /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits
make_plots: False
im_savedir: /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/metacal-plots/
nproc: 8
vb: True
seed: 1655289126
Warning: index_end=2500 larger than catalog size of 1820; running over full catalog
Starting metacal fitting with 8 cores
/users/jmcclear/.local/lib/python3.7/site-packages/pyfits/__init__.py:22: PyFITSDeprecationWarning: PyFITS is deprecated, please use astropy.io.fits
  PyFITSDeprecationWarning)  # noqa
Traceback (most recent call last):
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py", line 825, in <module>
    main()
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py", line 796, in main
    ) for i in range(index_start, index_end)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py", line 796, in <listcomp>
    ) for i in range(index_start, index_end)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py", line 206, in _get_jacobians
    jlist = self.medsObj.get_jacobian_list(source_id)
  File "/users/jmcclear/anaconda3/lib/python3.7/site-packages/meds/meds.py", line 921, in get_jacobian_list
    self._check_indices(iobj)
  File "/users/jmcclear/anaconda3/lib/python3.7/site-packages/meds/meds.py", line 1067, in _check_indices
    raise ValueError("object %s has no cutouts" % iobj)
ValueError: object 14 has no cutouts

Traceback (most recent call last):
  File "superbit_lensing/pipe_test.py", line 45, in <module>
    rc = main()
  File "superbit_lensing/pipe_test.py", line 40, in main
    rc = pipe.run()
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 289, in run
    rc = module.run(self._config['run_options'], self.logprint)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 437, in run
    rc = self._run_command(cmd, logprint)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 159, in _run_command
    raise err
subprocess.CalledProcessError: Command '['python', '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/metacalibration/ngmix_fit_superbit3.py', '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits', '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits', '-outdir=/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', '-end=2500', '-n=8', '--vb']' returned non-zero exit status 1.

I think the root of the problem lies in

[sweverett]

The latest commits (d9970d4 & 1165c49) have introduced a bug so now pipe_test.py doesn't run to completion. See the error message below. Any thoughts?

Sorry for being unable to do a full pipe test before committing these. I believe a quick error catch in get_jacobian_list() to return None in this case will work, as we want it to fail during the flag check function which will print out the root cause of the failure. I will add this.

[mcclearyj]

We forgot to take out the Sextractor galaxy source selection from medsmaker_mocks.py, but I can just do that after the merge with master.