search

LSSTDESC / desc-gen3-prod

Desc-prod wrapper for pipeline production using gen3_workflow.
BSD 3-Clause "New" or "Revised" License
0 stars 1 forks source link

Scratch repo does not work for ccd jobs #8

Open dladams opened 8 months ago

dladams commented 8 months ago

I cannot use my run 1085 scratch-based repository for ccd jobs. Here is the log from the QG generation in failed job 1498:

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/bin/pipetask --long-log --log-level=VERBOSE qgraph --butler-config /pscratch/sd/d/dladams/repo-1085 -i LSSTCam-imSim/defaults -o u/dladams/ccd --output-run u/dladams/ccd/20240105T140350Z --pipeline /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/drp_pipe/gf4d25a888e+cc957198ff/pipelines/_ingredients/LSSTCam-imSim/DRP.yaml#isr,characterizeImage,calibrate --save-qgraph /pscratch/sd/d/dladams/descprod-out/jobs/job001498/submit/u/dladams/ccd/20240105T140350Z/u_dladams_ccd_20240105T140350Z.qgraph --qgraph-datastore-records -d "visit>0 and skymap='DC2' and visit in (250..260)"

ERROR 2024-01-05T06:04:03.639-08:00 lsst.daf.butler.cli.utils ()(utils.py:1133) - Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 189, in qgraph
    if script.qgraph(pipelineObj=pipeline, **kwargs, show=show) is None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 221, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 650, in makeGraph
    qgraph = graphBuilder.makeGraph(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/graphBuilder.py", line 163, in makeGraph
    qgb = AllDimensionsQuantumGraphBuilder(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/all_dimensions_quantum_graph_builder.py", line 112, in __init__
    super().__init__(pipeline_graph, butler, **kwargs)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/quantum_graph_builder.py", line 183, in __init__
    self._pipeline_graph.resolve(self.butler.registry)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/pipeline_graph/_pipeline_graph.py", line 518, in resolve
    new_dataset_type_node = DatasetTypeNode._from_edges(
                            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/pipeline_graph/_dataset_types.py", line 152, in _from_edges
    dataset_type, is_initial_query_constraint, is_prerequisite = consuming_edge._resolve_dataset_type(
                                                                 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/pipeline_graph/_edges.py", line 491, in _resolve_dataset_type
    raise MissingDatasetTypeError(
lsst.daf.butler.registry._exceptions.MissingDatasetTypeError: "DatasetType 'cal_ref_cat_2_2' referenced by 'calibrate' uses 'skypix' as a dimension placeholder, but has not been registered with the data repository.  Note that reference catalog names are now used as the dataset type name instead of 'ref_cat'."
dladams commented 8 months ago

I try building a new repo from scratch with

 lsst:w_2023_44> ~/desc/dev/desc-gen3-prod/scripts/g3wf-create-qg-repo -c dc2 1505 1505
g3wf-create-qg-repo:     QG run: 1505
g3wf-create-qg-repo:       Repo: /pscratch/sd/d/dladams/repo-1505
g3wf-create-qg-repo:  Calib opt: dc2
Directory already exists: /pscratch/sd/d/dladams/repo-1505
Do you wish to recreate (y/n)> y
Removing /pscratch/sd/d/dladams/repo-1505

Mon Jan  8 10:30:58 PST 2024
g3wf-create-qg-repo: Creating empty repository /pscratch/sd/d/dladams/repo-1505
total 749
  1 -rw-r----- 1 dladams dladams    756 Jan  8 10:30 butler.yaml
748 -rw-r----- 1 dladams dladams 765952 Jan  8 10:30 gen3.sqlite3
g3wf-create-qg-repo: Registering instrument lsst.obs.lsst.LsstCamImSim

Mon Jan  8 10:31:05 PST 2024
g3wf-create-qg-repo: Importing calibs.
lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/cli/cmd/commands.py", line 687, in collection_chain
    chain = script.collectionChain(**kwargs)
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/script/collectionChain.py", line 137, in collectionChain
    butler.registry.setCollectionChain(parent, children, flatten=flatten)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/_registry_shim.py", line 139, in setCollectionChain
    self._registry.setCollectionChain(parent, children, flatten=flatten)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/utils.py", line 61, in inner
    return func(self, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/registry/sql_registry.py", line 606, in setCollectionChain
    record.update(self._managers.collections, children, flatten=flatten)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/registry/interfaces/_collections.py", line 210, in update
    for record in manager.resolve_wildcard(
                  ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/registry/collections/_base.py", line 506, in resolve_wildcard
    result.extend(resolve_nested(self.find(name), done))
                                 ^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/daf_butler/ge89626a060+c4141ea9c4/python/lsst/daf/butler/registry/collections/_base.py", line 460, in find
    raise MissingCollectionError(f"No collection with name '{name}' found.")
lsst.daf.butler.registry._exceptions.MissingCollectionError: No collection with name 'LSSTCam-imSim/raw/all' found.

and then

Mon Jan  8 10:39:01 PST 2024
g3wf-create-qg-repo: Processing QG
g3wf-create-qg-repo: python g3wf-create-qg-repo.py 1505 /pscratch/sd/d/dladams/repo-1505 0 copy
qgread: Input repo has 184 collections:
Loading quantum graph...
Graph ID: 1704732514.190567-2114415
qgread: Ignoring unknown dataset type isrOverscanCorrected
qgread: Ignoring unknown dataset type bfKernel
qgread: Ignoring unknown dataset type fringe
qgread: Ignoring unknown dataset type crosstalk
qgread: Found 17 of 21 dataset types:
       src: DatasetType('src', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
cal_ref_cat_2_2: DatasetType('cal_ref_cat_2_2', {htm7}, SimpleCatalog)
srcMatchFull: DatasetType('srcMatchFull', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
    calexp: DatasetType('calexp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
icExpBackground: DatasetType('icExpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
    camera: DatasetType('camera', {instrument}, Camera, isCalibration=True)
calexpBackground: DatasetType('calexpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
      dark: DatasetType('dark', {instrument, detector}, ExposureF, isCalibration=True)
       raw: DatasetType('raw', {band, instrument, detector, physical_filter, exposure}, Exposure)
      bias: DatasetType('bias', {instrument, detector}, ExposureF, isCalibration=True)
  srcMatch: DatasetType('srcMatch', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
     icExp: DatasetType('icExp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
     icSrc: DatasetType('icSrc', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
       bfk: DatasetType('bfk', {instrument, detector}, BrighterFatterKernel, isCalibration=True)
postISRCCD: DatasetType('postISRCCD', {band, instrument, detector, physical_filter, exposure}, Exposure)
icSrc_schema: DatasetType('icSrc_schema', {}, SourceCatalog)
      flat: DatasetType('flat', {band, instrument, detector, physical_filter}, ExposureF, isCalibration=True)

qgread: Looping over 995 inputquanta.
       1: Inserting bias@{instrument: 'LSSTCam-imSim', detector: 122} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=11ff2b5b-d924-46c7-8809-55c5a4112311)
         /pscratch/sd/d/dladams/repo-1505/2.2i/calib/gen2/20220101T000000Z/bias/bias_LSSTCam-imSim_R30_S12_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting dark@{instrument: 'LSSTCam-imSim', detector: 122} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=b40461d2-fde1-4c2b-855c-cc90edd69e05)
         /pscratch/sd/d/dladams/repo-1505/2.2i/calib/gen2/20220101T000000Z/dark/dark_LSSTCam-imSim_R30_S12_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting flat@{instrument: 'LSSTCam-imSim', detector: 122, physical_filter: 'g_sim_1.4', ...} [sc=ExposureF] (run=2.2i/calib/gen2/20220806T000000Z id=b39d0844-5106-471d-ab85-2052128a8556)
         /pscratch/sd/d/dladams/repo-1505/2.2i/calib/gen2/20220806T000000Z/flat/g/g_sim_1.4/flat_LSSTCam-imSim_g_g_sim_1_4_R30_S12_2_2i_calib_gen2_20220806T000000Z.fits
.
.
.
     993: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 175, exposure: 260, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=8348fa3b-2ed0-5360-a893-3c653a0e3f3b)
     994: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 88, exposure: 258, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=dc37a044-f44c-5a75-9145-a9eef33eed85)
     995: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 53, exposure: 258, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=4aa8b205-7943-5654-a889-a8ee52534226)
qgread: Processed 995 quanta with
qgread:     1580 datasets
qgread:      995 recorded datasets
qgread:     4390 duplicate datasets
Bye
Repository /pscratch/sd/d/dladams/repo-1505 holds 44766 MB

Mon Jan  8 11:25:10 PST 2024
g3wf-create-qg-repo: Creating skymap.
/global/homes/d/dladams/desc/dev/desc-gen3-prod/scripts/g3wf-create-qg-repo: line 147: wget: command not found

I modified the script to use curl instead of wget and ran it for a single visit:

 lsst:w_2023_44> ~/desc/dev/desc-gen3-prod/scripts/g3wf-create-qg-repo -c qg 1514 1514
g3wf-create-qg-repo:     QG run: 1514
g3wf-create-qg-repo:       Repo: /pscratch/sd/d/dladams/repo-1514
g3wf-create-qg-repo:  Calib opt: qg

Mon Jan  8 12:09:48 PST 2024
g3wf-create-qg-repo: Creating empty repository /pscratch/sd/d/dladams/repo-1514
total 749
  1 -rw-r----- 1 dladams dladams    756 Jan  8 12:09 butler.yaml
748 -rw-r----- 1 dladams dladams 765952 Jan  8 12:09 gen3.sqlite3
g3wf-create-qg-repo: Registering instrument lsst.obs.lsst.LsstCamImSim

Mon Jan  8 12:09:56 PST 2024
g3wf-create-qg-repo: Processing QG
g3wf-create-qg-repo: python g3wf-create-qg-repo.py 1514 /pscratch/sd/d/dladams/repo-1514 0 copy
qgread: Input repo has 184 collections:
Loading quantum graph...
Graph ID: 1704744262.008478-507998
qgread: Ignoring unknown dataset type bfKernel
qgread: Ignoring unknown dataset type fringe
qgread: Ignoring unknown dataset type crosstalk
qgread: Ignoring unknown dataset type isrOverscanCorrected
qgread: Found 17 of 21 dataset types:
       raw: DatasetType('raw', {band, instrument, detector, physical_filter, exposure}, Exposure)
cal_ref_cat_2_2: DatasetType('cal_ref_cat_2_2', {htm7}, SimpleCatalog)
calexpBackground: DatasetType('calexpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
    camera: DatasetType('camera', {instrument}, Camera, isCalibration=True)
    calexp: DatasetType('calexp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
srcMatchFull: DatasetType('srcMatchFull', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
      bias: DatasetType('bias', {instrument, detector}, ExposureF, isCalibration=True)
  srcMatch: DatasetType('srcMatch', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
       src: DatasetType('src', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
      flat: DatasetType('flat', {band, instrument, detector, physical_filter}, ExposureF, isCalibration=True)
     icExp: DatasetType('icExp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
postISRCCD: DatasetType('postISRCCD', {band, instrument, detector, physical_filter, exposure}, Exposure)
       bfk: DatasetType('bfk', {instrument, detector}, BrighterFatterKernel, isCalibration=True)
icSrc_schema: DatasetType('icSrc_schema', {}, SourceCatalog)
icExpBackground: DatasetType('icExpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
     icSrc: DatasetType('icSrc', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
      dark: DatasetType('dark', {instrument, detector}, ExposureF, isCalibration=True)

qgread: Looping over 94 inputquanta.
       1: Inserting flat@{instrument: 'LSSTCam-imSim', detector: 72, physical_filter: 'g_sim_1.4', ...} [sc=ExposureF] (run=2.2i/calib/gen2/20220806T000000Z id=c10e27e0-a861-4a2d-ae63-4ff2841ee375)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220806T000000Z/flat/g/g_sim_1.4/flat_LSSTCam-imSim_g_g_sim_1_4_R20_S00_2_2i_calib_gen2_20220806T000000Z.fits
       1: Inserting dark@{instrument: 'LSSTCam-imSim', detector: 72} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=246b0b81-ffc2-4ccc-bad8-cd8ef5120344)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220101T000000Z/dark/dark_LSSTCam-imSim_R20_S00_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting camera@{instrument: 'LSSTCam-imSim'} [sc=Camera] (run=2.2i/calib/DM-30694/unbounded id=b7e109f0-764c-492a-9cfc-04471a7171e3)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/DM-30694/unbounded/camera/camera_LSSTCam-imSim_2_2i_calib_DM-30694_unbounded.fits
       1: Inserting bias@{instrument: 'LSSTCam-imSim', detector: 72} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=0ced0daf-f7ef-4c1c-8f3b-a650c42e4363)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220101T000000Z/bias/bias_LSSTCam-imSim_R20_S00_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting bfk@{instrument: 'LSSTCam-imSim', detector: 72} [sc=BrighterFatterKernel] (run=2.2i/calib/DM-30694/curated/19700101T000000Z id=ca33c78b-7994-486b-9a04-c11a261fb8c3)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/DM-30694/curated/19700101T000000Z/bfk/bfk_LSSTCam-imSim_R20_S00_2_2i_calib_DM-30694_curated_19700101T000000Z.fits
       1: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 72, exposure: 250, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=2c75a640-d3b4-5cc3-9965-849c3d99d0b7)
       2: Inserting flat@{instrument: 'LSSTCam-imSim', detector: 51, physical_filter: 'g_sim_1.4', ...} [sc=ExposureF] (run=2.2i/calib/gen2/20220806T000000Z id=8cccb020-4738-4650-93e6-85fc0ff50483)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220806T000000Z/flat/g/g_sim_1.4/flat_LSSTCam-imSim_g_g_sim_1_4_R12_S20_2_2i_calib_gen2_20220806T000000Z.fits
       2: Inserting dark@{instrument: 'LSSTCam-imSim', detector: 51} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=c441208a-bb04-4362-bd18-1f4783228759)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220101T000000Z/dark/dark_LSSTCam-imSim_R12_S20_2_2i_calib_gen2_20220101T000000Z.fits
.
.
.
      93: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 86, exposure: 250, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=f612a108-4d16-53b1-bf86-97eb16c76a1d)
      94: Inserting flat@{instrument: 'LSSTCam-imSim', detector: 117, physical_filter: 'g_sim_1.4', ...} [sc=ExposureF] (run=2.2i/calib/gen2/20220806T000000Z id=1f4ac2bc-a015-48ba-a888-9ec360f7d5bf)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220806T000000Z/flat/g/g_sim_1.4/flat_LSSTCam-imSim_g_g_sim_1_4_R30_S00_2_2i_calib_gen2_20220806T000000Z.fits
      94: Inserting dark@{instrument: 'LSSTCam-imSim', detector: 117} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=d95b8a61-bc24-408d-8780-3fbf76da4168)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220101T000000Z/dark/dark_LSSTCam-imSim_R30_S00_2_2i_calib_gen2_20220101T000000Z.fits
      94: Inserting bias@{instrument: 'LSSTCam-imSim', detector: 117} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=dd8e6663-b2df-4a38-8c93-61b5c687c8f9)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/gen2/20220101T000000Z/bias/bias_LSSTCam-imSim_R30_S00_2_2i_calib_gen2_20220101T000000Z.fits
      94: Inserting bfk@{instrument: 'LSSTCam-imSim', detector: 117} [sc=BrighterFatterKernel] (run=2.2i/calib/DM-30694/curated/19700101T000000Z id=09d3e630-639b-40fd-a042-cfa3ed977f25)
         /pscratch/sd/d/dladams/repo-1514/2.2i/calib/DM-30694/curated/19700101T000000Z/bfk/bfk_LSSTCam-imSim_R30_S00_2_2i_calib_DM-30694_curated_19700101T000000Z.fits
      94: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 117, exposure: 250, ...} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=1b23cf2c-e511-53b1-8827-347026bfedcb)
qgread: Processed 94 quanta with
qgread:      471 datasets
qgread:       94 recorded datasets
qgread:       93 duplicate datasets
Bye
Repository /pscratch/sd/d/dladams/repo-1514 holds 28207 MB

Mon Jan  8 12:34:52 PST 2024
g3wf-create-qg-repo: Creating skymap.
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  1262  100  1262    0     0   8857      0 --:--:-- --:--:-- --:--:--  8887
Repository /pscratch/sd/d/dladams/repo-1514 holds 28789 MB

Mon Jan  8 12:37:02 PST 2024
g3wf-create-qg-repo: Inserting raw data.
/global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim/y1-wfd/00000250/lsst_a_250_R20_S00_g.fits
    /pscratch/sd/d/dladams/repo-1514/raw/lsst_a_250_R20_S00_g.fits
lsst.ingest INFO: Successfully extracted metadata from 1 file with 0 failures
lsst.ingest INFO: Exposure LSSTCam-imSim:250 ingested successfully
lsst.ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest.
lsst.ingest INFO: Ingested 1 distinct Butler dataset
/global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim/y1-wfd/00000250/lsst_a_250_R12_S20_g.fits
    /pscratch/sd/d/dladams/repo-1514/raw/lsst_a_250_R12_S20_g.fits
lsst.ingest INFO: Successfully extracted metadata from 1 file with 0 failures
lsst.ingest INFO: Exposure LSSTCam-imSim:250 ingested successfully
.
.
.
    /pscratch/sd/d/dladams/repo-1514/raw/lsst_a_250_R21_S12_g.fits
lsst.ingest INFO: Successfully extracted metadata from 1 file with 0 failures
lsst.ingest INFO: Exposure LSSTCam-imSim:250 ingested successfully
lsst.ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest.
lsst.ingest INFO: Ingested 1 distinct Butler dataset
/global/cfs/cdirs/lsst/shared/DC2-prod/Run2.2i/sim/y1-wfd/00000250/lsst_a_250_R30_S00_g.fits
    /pscratch/sd/d/dladams/repo-1514/raw/lsst_a_250_R30_S00_g.fits
lsst.ingest INFO: Successfully extracted metadata from 1 file with 0 failures
lsst.ingest INFO: Exposure LSSTCam-imSim:250 ingested successfully
lsst.ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest.
lsst.ingest INFO: Ingested 1 distinct Butler dataset
lsst.defineVisits INFO: Preprocessing data IDs.
lsst.defineVisits INFO: Registering visit_system 2: by-seq-start-end.
lsst.defineVisits INFO: Registering visit_system 0: one-to-one.
lsst.defineVisits INFO: Grouping 1 exposure(s) into visits.
lsst.defineVisits INFO: Computing regions and other metadata for 1 visit(s).
Repository /pscratch/sd/d/dladams/repo-1514 holds 30322 MB

Mon Jan  8 12:44:30 PST 2024
g3wf-create-qg-repo: Creating default chain LSSTCam-imSim/defaults
2.2i/calib/DM-30694/curated/19700101T000000Z,2.2i/calib/DM-30694/unbounded,2.2i/calib/gen2/20220101T000000Z,2.2i/calib/gen2/20220806T000000Z,LSSTCam-imSim/raw/all,skymaps

Creating chain LSSTCam-imSim/defaults
[2.2i/calib/DM-30694/curated/19700101T000000Z, 2.2i/calib/DM-30694/unbounded, 2.2i/calib/gen2/20220101T000000Z, 2.2i/calib/gen2/20220806T000000Z, LSSTCam-imSim/raw/all, skymaps]
                     Name                        Type 
---------------------------------------------- -------
2.2i/calib/DM-30694/curated/19700101T000000Z   RUN    
2.2i/calib/DM-30694/unbounded                  RUN    
2.2i/calib/gen2/20220101T000000Z               RUN    
2.2i/calib/gen2/20220806T000000Z               RUN    
LSSTCam-imSim/defaults                         CHAINED
  2.2i/calib/DM-30694/curated/19700101T000000Z RUN    
  2.2i/calib/DM-30694/unbounded                RUN    
  2.2i/calib/gen2/20220101T000000Z             RUN    
  2.2i/calib/gen2/20220806T000000Z             RUN    
  LSSTCam-imSim/raw/all                        RUN    
  skymaps                                      RUN    
LSSTCam-imSim/raw/all                          RUN    
skymaps                                        RUN    

Mon Jan  8 12:44:34 PST 2024
 lsst:w_2023_44>
dladams commented 8 months ago

But when I tried to run with the new repo, I still had problems:

/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/bin/pipetask --long-log --log-level=VERBOSE qgraph --butler-config /pscratch/sd/d/dladams/repo-1514 -i LSSTCam-imSim/defaults -o u/dladams/ccd --output-run u/dladams/ccd/20240108T205701Z --pipeline /opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/drp_pipe/gf4d25a888e+cc957198ff/pipelines/_ingredients/LSSTCam-imSim/DRP.yaml#isr,characterizeImage,calibrate --save-qgraph /pscratch/sd/d/dladams/descprod-out/jobs/job001516/submit/u/dladams/ccd/20240108T205701Z/u_dladams_ccd_20240108T205701Z.qgraph --qgraph-datastore-records -d "visit>0 and skymap='DC2' and visit=250"

INFO 2024-01-08T12:57:12.998-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:347) - Processing pipeline subgraph 1 of 1 with 3 task(s).
VERBOSE 2024-01-08T12:57:12.998-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:353) - Subgraph tasks: [isr, characterizeImage, calibrate]
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:491) - Querying for data IDs with arguments:
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:492) -   dimensions=['band', 'instrument', 'detector', 'physical_filter', 'exposure', 'visit'],
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:493) -   dataId={'instrument': 'LSSTCam-imSim'},
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:495) -   where="visit>0 and skymap='DC2' and visit=250",
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:497) -   datasets=['raw'],
VERBOSE 2024-01-08T12:57:13.000-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:499) -   collections=['2.2i/calib/DM-30694/curated/19700101T000000Z', '2.2i/calib/DM-30694/unbounded', '2.2i/calib/gen2/20220101T000000Z', '2.2i/calib/gen2/20220806T000000Z', 'LSSTCam-imSim/raw/all', 'skymaps'],
INFO 2024-01-08T12:57:13.102-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:159) - Iterating over query results to associate quanta with datasets.
INFO 2024-01-08T12:57:13.279-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:201) - Initial bipartite graph has 282 quanta, 1512 dataset nodes, and 1880 edges from 94 query row(s).
VERBOSE 2024-01-08T12:57:13.303-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:238) - Found 94 overall-input dataset(s) of type 'raw'.
VERBOSE 2024-01-08T12:57:13.330-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 94 prerequisite input edge(s) from dataset type 'bias' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.354-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 94 prerequisite input edge(s) from dataset type 'camera' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.354-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 0 prerequisite input edge(s) from dataset type 'crosstalk' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.354-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 0 prerequisite input edge(s) from dataset type 'fringe' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.354-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 0 prerequisite input edge(s) from dataset type 'bfKernel' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.380-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 94 prerequisite input edge(s) from dataset type 'bfk' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.405-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 94 prerequisite input edge(s) from dataset type 'dark' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.433-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 94 prerequisite input edge(s) from dataset type 'flat' to task 'isr'.
VERBOSE 2024-01-08T12:57:13.433-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:290) - Deferring prerequisite input 'isrOverscanCorrected' of task 'isr' to per-quantum processing (lookup function provided).
VERBOSE 2024-01-08T12:57:13.439-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 0 prerequisite input edge(s) from dataset type 'cal_ref_cat_2_2' to task 'calibrate'.
VERBOSE 2024-01-08T12:57:13.441-08:00 lsst.pipe.base.quantum_graph_builder ()(all_dimensions_quantum_graph_builder.py:363) - Added 0 prerequisite input edge(s) from dataset type 'cal_ref_cat_2_2' to task 'calibrate'.
INFO 2024-01-08T12:57:13.493-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:575) - Generated 94 quanta for task isr.
INFO 2024-01-08T12:57:13.513-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:575) - Generated 94 quanta for task characterizeImage.
ERROR 2024-01-08T12:57:13.514-08:00 lsst.daf.butler.cli.utils ()(utils.py:1133) - Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 189, in qgraph
    if script.qgraph(pipelineObj=pipeline, **kwargs, show=show) is None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 221, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/ctrl_mpexec/gef1ac55fff+9dabee4918/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 650, in makeGraph
    qgraph = graphBuilder.makeGraph(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/graphBuilder.py", line 174, in makeGraph
    return qgb.build(metadata)
           ^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/quantum_graph_builder.py", line 360, in build
    self._resolve_task_quanta(task_node, full_skeleton)
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/utils/g384e8880d6+81bc2a20b4/python/lsst/utils/timer.py", line 295, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/quantum_graph_builder.py", line 510, in _resolve_task_quanta
    helper.adjust_in_place(
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/connections.py", line 1079, in adjust_in_place
    adjusted_inputs_by_connection, adjusted_outputs_by_connection = connections.adjustQuantum(
                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/opt/lsst/software/stack/stack/miniconda3-py38_4.9.2-7.0.1/Linux64/pipe_base/gfacbe7f57e+607ede7504/python/lsst/pipe/base/connections.py", line 908, in adjustQuantum
    raise FileNotFoundError(
FileNotFoundError: Not enough datasets (0) found for non-optional connection calibrate.photoRefCat (cal_ref_cat_2_2) with minimum=1 for quantum data ID {instrument: 'LSSTCam-imSim', detector: 76, visit: 250, ...}.
dladams commented 8 months ago

I try again with w_2024_01:

 cvmfs g3wf-w2401-parsl> ./g3wf-create-qg-repo -c qg 1555 1555
g3wf-create-qg-repo:     QG run: 1555
g3wf-create-qg-repo:       Repo: /pscratch/sd/d/dladams/repo-1555
g3wf-create-qg-repo:  Calib opt: qg

Wed 10 Jan 2024 12:25:20 PM PST
g3wf-create-qg-repo: Creating empty repository /pscratch/sd/d/dladams/repo-1555
total 749
  1 -rw-r----- 1 dladams dladams    756 Jan 10 12:26 butler.yaml
748 -rw-r----- 1 dladams dladams 765952 Jan 10 12:26 gen3.sqlite3
g3wf-create-qg-repo: Registering instrument lsst.obs.lsst.LsstCamImSim

Wed 10 Jan 2024 12:28:57 PM PST
g3wf-create-qg-repo: Processing QG
g3wf-create-qg-repo: python g3wf-create-qg-repo.py 1555 /pscratch/sd/d/dladams/repo-1555 0 copy
qgread: Input repo has 184 collections:
Loading quantum graph...
Graph ID: 1704917720.97608-171539
qgread: Ignoring unknown dataset type fringe
qgread: Ignoring unknown dataset type crosstalk
qgread: Ignoring unknown dataset type bfKernel
qgread: Ignoring unknown dataset type isrOverscanCorrected
qgread: Found 17 of 21 dataset types:
    calexp: DatasetType('calexp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
srcMatchFull: DatasetType('srcMatchFull', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
      bias: DatasetType('bias', {instrument, detector}, ExposureF, isCalibration=True)
icSrc_schema: DatasetType('icSrc_schema', {}, SourceCatalog)
cal_ref_cat_2_2: DatasetType('cal_ref_cat_2_2', {htm7}, SimpleCatalog)
       bfk: DatasetType('bfk', {instrument, detector}, BrighterFatterKernel, isCalibration=True)
postISRCCD: DatasetType('postISRCCD', {band, instrument, detector, physical_filter, exposure}, Exposure)
calexpBackground: DatasetType('calexpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
     icSrc: DatasetType('icSrc', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
      flat: DatasetType('flat', {band, instrument, detector, physical_filter}, ExposureF, isCalibration=True)
      dark: DatasetType('dark', {instrument, detector}, ExposureF, isCalibration=True)
     icExp: DatasetType('icExp', {band, instrument, detector, physical_filter, visit_system, visit}, ExposureF)
       src: DatasetType('src', {band, instrument, detector, physical_filter, visit_system, visit}, SourceCatalog)
       raw: DatasetType('raw', {band, instrument, detector, physical_filter, exposure}, Exposure)
icExpBackground: DatasetType('icExpBackground', {band, instrument, detector, physical_filter, visit_system, visit}, Background)
  srcMatch: DatasetType('srcMatch', {band, instrument, detector, physical_filter, visit_system, visit}, Catalog)
    camera: DatasetType('camera', {instrument}, Camera, isCalibration=True)

qgread: Looping over 94 inputquanta.
       1: Inserting camera@{instrument: 'LSSTCam-imSim'} [sc=Camera] (run=2.2i/calib/DM-30694/unbounded id=b7e109f0-764c-492a-9cfc-04471a7171e3)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/DM-30694/unbounded/camera/camera_LSSTCam-imSim_2_2i_calib_DM-30694_unbounded.fits
       1: Inserting bfk@{instrument: 'LSSTCam-imSim', detector: 142} [sc=BrighterFatterKernel] (run=2.2i/calib/DM-30694/curated/19700101T000000Z id=120a325b-3828-426c-8291-bbc39ab821f8)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/DM-30694/curated/19700101T000000Z/bfk/bfk_LSSTCam-imSim_R32_S21_2_2i_calib_DM-30694_curated_19700101T000000Z.fits
       1: Inserting dark@{instrument: 'LSSTCam-imSim', detector: 142} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=92d3a5c8-8050-462f-8f24-90f2309820d2)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/gen2/20220101T000000Z/dark/dark_LSSTCam-imSim_R32_S21_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting bias@{instrument: 'LSSTCam-imSim', detector: 142} [sc=ExposureF] (run=2.2i/calib/gen2/20220101T000000Z id=e766e9f8-a74f-4f2b-93eb-d04eb504db4e)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/gen2/20220101T000000Z/bias/bias_LSSTCam-imSim_R32_S21_2_2i_calib_gen2_20220101T000000Z.fits
       1: Inserting flat@{instrument: 'LSSTCam-imSim', detector: 142, physical_filter: 'g_sim_1.4', band: 'g'} [sc=ExposureF] (run=2.2i/calib/gen2/20220806T000000Z id=433c568b-9f64-40cc-9d01-85228b6f6ed5)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/gen2/20220806T000000Z/flat/g/g_sim_1.4/flat_LSSTCam-imSim_g_g_sim_1_4_R32_S21_2_2i_calib_gen2_20220806T000000Z.fits
       1: Recording raw raw@{instrument: 'LSSTCam-imSim', detector: 142, exposure: 250, band: 'g', physical_filter: 'g_sim_1.4'} [sc=Exposure] (run=LSSTCam-imSim/raw/all id=e3068579-0edc-582f-9ebb-aeb058aece34)
       2: Inserting bfk@{instrument: 'LSSTCam-imSim', detector: 166} [sc=BrighterFatterKernel] (run=2.2i/calib/DM-30694/curated/19700101T000000Z id=048c6f3f-e3a1-4a96-a884-ec617748933a)
         /pscratch/sd/d/dladams/repo-1555/2.2i/calib/DM-30694/curated/19700101T000000Z/bfk/bfk_LSSTCam-imSim_R41_S11_2_2i_calib_DM-30694_curated_19700101T000000Z.fits
.
.
.
    /pscratch/sd/d/dladams/repo-1555/raw/lsst_a_250_R20_S11_g.fits
lsst.ingest INFO: Successfully extracted metadata from 1 file with 0 failures
lsst.ingest INFO: Exposure LSSTCam-imSim:250 ingested successfully
lsst.ingest INFO: Successfully processed data from 1 exposure with 0 failures from exposure registration and 0 failures from file ingest.
lsst.ingest INFO: Ingested 1 distinct Butler dataset
lsst.defineVisits INFO: Preprocessing data IDs.
lsst.defineVisits INFO: Registering visit_system 0: one-to-one.
lsst.defineVisits INFO: Registering visit_system 2: by-seq-start-end.
lsst.defineVisits INFO: Grouping 1 exposure(s) into visits.
lsst.defineVisits INFO: Computing regions and other metadata for 1 visit(s).
Repository /pscratch/sd/d/dladams/repo-1555 holds 30340 MB

Wed 10 Jan 2024 01:03:03 PM PST
g3wf-create-qg-repo: Creating default chain LSSTCam-imSim/defaults
2.2i/calib/DM-30694/curated/19700101T000000Z,2.2i/calib/DM-30694/unbounded,2.2i/calib/gen2/20220101T000000Z,2.2i/calib/gen2/20220806T000000Z,LSSTCam-imSim/raw/all,skymaps

Creating chain LSSTCam-imSim/defaults
[2.2i/calib/DM-30694/curated/19700101T000000Z, 2.2i/calib/DM-30694/unbounded, 2.2i/calib/gen2/20220101T000000Z, 2.2i/calib/gen2/20220806T000000Z, LSSTCam-imSim/raw/all, skymaps]
                     Name                        Type 
---------------------------------------------- -------
2.2i/calib/DM-30694/curated/19700101T000000Z   RUN    
2.2i/calib/DM-30694/unbounded                  RUN    
2.2i/calib/gen2/20220101T000000Z               RUN    
2.2i/calib/gen2/20220806T000000Z               RUN    
LSSTCam-imSim/defaults                         CHAINED
  2.2i/calib/DM-30694/curated/19700101T000000Z RUN    
  2.2i/calib/DM-30694/unbounded                RUN    
  2.2i/calib/gen2/20220101T000000Z             RUN    
  2.2i/calib/gen2/20220806T000000Z             RUN    
  LSSTCam-imSim/raw/all                        RUN    
  skymaps                                      RUN    
LSSTCam-imSim/raw/all                          RUN    
skymaps                                        RUN    

Wed 10 Jan 2024 01:03:08 PM PST
 cvmfs g3wf-w2401-parsl>

My attempt to use that repository with the same release still fails:

INFO 2024-01-11T06:53:14.900-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:576) - Generated 94 quanta for task isr.
INFO 2024-01-11T06:53:14.919-08:00 lsst.pipe.base.quantum_graph_builder ()(quantum_graph_builder.py:576) - Generated 94 quanta for task characterizeImage.
ERROR 2024-01-11T06:53:14.920-08:00 lsst.daf.butler.cli.utils ()(utils.py:1223) - Caught an exception, details are in traceback:
Traceback (most recent call last):
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/gc579f04acb+9768e77abd/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 189, in qgraph
    if script.qgraph(pipelineObj=pipeline, **kwargs, show=show) is None:
       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/gc579f04acb+9768e77abd/python/lsst/ctrl/mpexec/cli/script/qgraph.py", line 221, in qgraph
    qgraph = f.makeGraph(pipelineObj, args)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/ctrl_mpexec/gc579f04acb+9768e77abd/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 651, in makeGraph
    qgraph = graphBuilder.makeGraph(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/pipe_base/g9d3cecc6b4+308e6d8964/python/lsst/pipe/base/graphBuilder.py", line 175, in makeGraph
    return qgb.build(metadata)
           ^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/utils/gd9c83c13ba+81bc2a20b4/python/lsst/utils/timer.py", line 300, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/pipe_base/g9d3cecc6b4+308e6d8964/python/lsst/pipe/base/quantum_graph_builder.py", line 360, in build
    self._resolve_task_quanta(task_node, full_skeleton)
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/utils/gd9c83c13ba+81bc2a20b4/python/lsst/utils/timer.py", line 300, in timeMethod_wrapper
    res = func(self, *args, **keyArgs)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/pipe_base/g9d3cecc6b4+308e6d8964/python/lsst/pipe/base/quantum_graph_builder.py", line 511, in _resolve_task_quanta
    helper.adjust_in_place(
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/pipe_base/g9d3cecc6b4+308e6d8964/python/lsst/pipe/base/connections.py", line 1103, in adjust_in_place
    adjusted_inputs_by_connection, adjusted_outputs_by_connection = connections.adjustQuantum(
                                                                    ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2024_01/conda/envs/lsst-scipipe-8.0.0-exact-ext/share/eups/Linux64/pipe_base/g9d3cecc6b4+308e6d8964/python/lsst/pipe/base/connections.py", line 932, in adjustQuantum
    raise FileNotFoundError(
FileNotFoundError: Not enough datasets (0) found for non-optional connection calibrate.photoRefCat (cal_ref_cat_2_2) with minimum=1 for quantum data ID {instrument: 'LSSTCam-imSim', detector: 48, visit: 250, band: 'g', physical_filter: 'g_sim_1.4'}.