search

superbit-collaboration / superbit-metacal

Contains a collection of routines used to perform gmix/metacalibration on simulated SuperBIT images
4 stars 1 forks source link

Automated job meta config generation #63

Closed sweverett closed 2 years ago

sweverett commented 2 years ago

See #62.

First test runs have finished (by running python prep_jobs.py {meta-config}, though I haven't run it through the full pipeline yet. Here is what the new top-level meta-config looks like:

run_name: 'jobs_manager_test'
base_dir: '/Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/'
nfw_dir: '/Users/sweveret/repos/superbit-metacal/runs/nfw_truth/'
gs_base_config: '/Users/sweveret/repos/superbit-metacal/superbit_lensing/tests/jobs/spencer_gs_pipe_test.yaml'
mass_bins: [3.0e14, 8.0e14] # Msol / h; must have decimal!
z_bins: [0.2, 0.3]
realizations: 2
ncores_per_job: 2
memory_per_job: 4 # GB

Right now the memory_per_job does nothing, but in the future I'd like to use it to help automate the job submission as well

Here is what the outputs look like for 2 masses, redshifts, and realizations:

> ls {base_dir}
cl_m3.0e14_z0.2 cl_m3.0e14_z0.3 cl_m8.0e14_z0.2 cl_m8.0e14_z0.3
> ls {base_dir}/cl_m3.0e14_z0.2
r0 r1
> ls {base_dir}/cl_m3.0e14_z0.2/r0
{run_name}_cl_m3.0e14_z0.2.yaml   {run_name}_gs_config.yaml   ngmix_test_config.yaml

The first config is the pipe config for that cluster, the second is the GalSim config for the image simulation generation, and the final is an example config for the ngmix_fit module. The truth files are also assumed to have the same naming convention as the cluster run names, but all are stored in the same nfw_truth directory.

Here is an example output pipeline run config for one of the cluster realizations:

galsim:
  clobber: false
  config_dir: /Users/sweveret/repos/superbit-metacal/superbit_lensing/galsim/config_files
  config_file: /Users/sweveret/repos/superbitmetacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_gs_config.yaml
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
medsmaker:
  fname_base: jobs_manager_test
  mock_dir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  outfile: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_meds.fits
  run_name: jobs_manager_test
metacal:
  meds_file: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_meds.fits
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  outfile: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_mcal.fits
shear_profile:
  mcal_file: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_mcal.fits
  nfw_file: /Users/sweveret/repos/superbit-metacal/runs/nfw_truth/cl_m8.0e14_z0.2/nfw_cl_m8.0e14_z0.2.fits
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  outfile: jobs_manager_test_annular.fits
  overwrite: false
  run_name: jobs_manager_test
  se_file: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_mock_coadd_cat.ldac
ngmix_fit:
  config: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/ngmix_test_config.yaml
  meds_file: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0/jobs_manager_test_meds.fits
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  outfile: jobs_manager_test_ngmix.fits
run_options:
  ncores: 8
  order:
  - galsim
  - medsmaker
  - metacal
  - shear_profile
  - ngmix_fit
  outdir: /Users/sweveret/repos/superbit-metacal/runs/jobs_manager_test/cl_m8.0e14_z0.2/r0
  run_diagnostics: true
  run_name: jobs_manager_test
  vb: true

While this all seems to work correctly, we should wait to merge the PR until I've tested

mcclearyj commented 2 years ago

Is it possible that there's a version of pipe.py that hasn't been committed to this branch yet? I ran a pipe_test.py and got the following error:

[jmcclear@login006 superbit-metacal]$ python superbit_lensing/pipe_test.py --fresh
/users/jmcclear/.local/lib/python3.7/site-packages/pyfits/__init__.py:22: PyFITSDeprecationWarning: PyFITS is deprecated, please use astropy.io.fits
  PyFITSDeprecationWarning)  # noqa
Deleting old test directory /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test...
config =
{'galsim': {'clobber': True, 'config_dir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/config_files', 'config_file': 'pipe_test.yaml', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test'}, 'medsmaker': {'fname_base': 'pipe_test', 'mock_dir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outfile': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits', 'run_name': 'pipe_test'}, 'metacal': {'end': 2500, 'meds_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outfile': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits'}, 'ngmix_fit': {'config': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/ngmix_test.yaml', 'end': 100, 'meds_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_meds.fits', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outfile': 'pipe_test_ngmix.fits'}, 'run_options': {'ncores': 8, 'order': ['galsim', 'medsmaker', 'metacal', 'shear_profile', 'ngmix_fit'], 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'run_diagnostics': True, 'run_name': 'pipe_test', 'vb': True}, 'shear_profile': {'mcal_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mcal.fits', 'nfw_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/runs/truth/cl3_nfwonly_truth_cat.fits', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', 'outfile': 'pipe_test_annular.fits', 'overwrite': True, 'run_name': 'pipe_test', 'se_file': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test/pipe_test_mock_coadd_cat.ldac'}}

Starting pipeline run

Running module galsim

config:
{'clobber': True, 'config_dir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/config_files', 'config_file': 'pipe_test.yaml', 'outdir': '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test'}

python /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/config_files/pipe_test.yaml -outdir=/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test -run_name=pipe_test --clobber --vb -ncores=8

Loading parameters from /gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/config_files/pipe_test.yaml
Processing command line args
Traceback (most recent call last):
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py", line 1180, in <module>
    main()
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py", line 774, in main
    sbparams = SuperBITParameters(config_file, logprint, args=args)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py", line 452, in __init__
    self._set_seeds()
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py", line 636, in _set_seeds
    for seed_name in seeds.keys():
AttributeError: 'zip' object has no attribute 'keys'

Traceback (most recent call last):
  File "superbit_lensing/pipe_test.py", line 45, in <module>
    rc = main()
  File "superbit_lensing/pipe_test.py", line 40, in main
    rc = pipe.run()
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 289, in run
    rc = module.run(self._config['run_options'], self.logprint)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 330, in run
    rc = self._run_command(cmd, logprint)
  File "/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/pipe.py", line 159, in _run_command
    raise err
subprocess.CalledProcessError: Command '['python', '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/mock_superBIT_data.py', '/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/superbit_lensing/galsim/config_files/pipe_test.yaml', '-outdir=/gpfs/data/idellant/jmcclear/superbit/superbit-metacal/tests/pipe_test', '-run_name=pipe_test', '--clobber', '--vb', '-ncores=8']' returned non-zero exit status 1.

The superbit_lensing/prep_jobs.py code ran just fine, though.

mcclearyj commented 2 years ago

It looks great, merging in now.