IOSvc.Input property can't be set from CLI

m-fila commented 1 month ago

I cannot pass to k4run the input files to process via --IOSvc.input blah.root. On the other hand IOSvc.output foo.root works like a charm. The difference between the two is that output is a string property while input is a vector property, so the problem seems to be in IOSvc parsing of vector<..>. Is there a special syntax to be used in the CLI that I am unaware of or is it a parsing problem in IOSvc?

Originally posted by @giovannimarchiori in https://github.com/key4hep/k4FWCore/issues/238#issuecomment-2376461994

m-fila commented 1 month ago

The IOSvc.Input is not listed in the acceptable arguments (unless it's assigned in a steering file):

  --IOSvc.OutputLevel [IOSVC.OUTPUTLEVEL], --OutputLevel.IOSvc [IOSVC.OUTPUTLEVEL]
                        output level [Service]
  --IOSvc.output [IOSVC.OUTPUT], --output.IOSvc [IOSVC.OUTPUT]
                        List of files to write output to [IOSvc]
  --IOSvc.Output [IOSVC.OUTPUT], --Output.IOSvc [IOSVC.OUTPUT]
                        List of files to write output to [IOSvc]
  --IOSvc.outputCommands IOSVC.OUTPUTCOMMANDS [IOSVC.OUTPUTCOMMANDS ...], --outputCommands.IOSvc IOSVC.OUTPUTCOMMANDS [IOSVC.OUTPUTCOMMANDS ...]
                        A set of commands to declare which collections to keep or drop. [IOSvc]
  --IOSvc.IOType [IOSVC.IOTYPE], --IOType.IOSvc [IOSVC.IOTYPE]
                        Type of input file (ROOT, RNTuple) [IOSvc]
  --IOSvc.ImportedFromk4FWCore [IOSVC.IMPORTEDFROMK4FWCORE], --ImportedFromk4FWCore.IOSvc [IOSVC.IMPORTEDFROMK4FWCORE]
                        This is set to true when IOSvc is imported from k4FWCore instead of Configurables in python [IOSvc]
  --IOSvc.FirstEventEntry [IOSVC.FIRSTEVENTENTRY], --FirstEventEntry.IOSvc [IOSVC.FIRSTEVENTENTRY]
                        First event entry to read [IOSvc]

I'm not sure if the problem here is that it's a vector of string because outputCommands are also vector of strings and they appear here just fine. Could it be something with the python wrappers (IOSvc.py, ApplicationMgr.py?)?

jmcarcell commented 1 month ago

This will be fixed tomorrow in the nightlies @giovannimarchiori. Passing IOSvc.Input file1.root or IOSvc.Input file1.root file2.root to read multiple files will be possible.

giovannimarchiori commented 1 month ago

Hi @jmcarcell , thanks a lot! Does this require some update in other packages other than k4FWCore? I tried syncing my fork of k4FWCore and pulling these changes (plus making a clean build) and the code does not work:

if the Input property of the iosvc service is defined in blah.py, doing k4run blah.py IOSvc.Input foo.root just ignores the CLI option

if I unset the input property of the iosvc service in blah.py and run again, I get the following error:

IOSvc               ERROR Property::fromString   Cannot convert '[['r', 'o', 'o', 't', '/', 'a', 'l', 'l', 'e', 'g', 'r', 'o', '_', 'v', '0', '3', '_', 'e', 'v', 't', 's', '_', '1', '0', '_', 'p', 'd', 'g', '_', '2', '2', '_', 'M', 'o', 'm', 'e', 'n', 't', 'u', 'm', 'M', 'i', 'n', 'M', 'a', 'x', '_', '2', '0', '_', '2', '0', '_', 'G', 'e', 'V', '_', 'T', 'h', 'e', 't', 'a', 'M', 'i', 'n', 'M', 'a', 'x', '_', '9', '0', '_', '9', '0', '_', 'P', 'h', 'i', 'M', 'i', 'n', 'M', 'a', 'x', '_', '1', '.', '5', '7', '0', '7', '9', '6', '_', '1', '.', '5', '7', '0', '7', '9', '6', '_', 's', 'i', 'm', '.', 'r', 'o', 'o', 't']]' for property 'Input' in class 'IOSvc': std::invalid_argument, cannot parse '[['r', 'o', 'o', 't', '/', 'a', 'l', 'l', 'e', 'g', 'r', 'o', '_', 'v', '0', '3', '_', 'e', 'v', 't', 's', '_', '1', '0', '_', 'p', 'd', 'g', '_', '2', '2', '_', 'M', 'o', 'm', 'e', 'n', 't', 'u', 'm', 'M', 'i', 'n', 'M', 'a', 'x', '_', '2', '0', '_', '2', '0', '_', 'G', 'e', 'V', '_', 'T', 'h', 'e', 't', 'a', 'M', 'i', 'n', 'M', 'a', 'x', '_', '9', '0', '_', '9', '0', '_', 'P', 'h', 'i', 'M', 'i', 'n', 'M', 'a', 'x', '_', '1', '.', '5', '7', '0', '7', '9', '6', '_', '1', '.', '5', '7', '0', '7', '9', '6', '_', 's', 'i', 'm', '.', 'r', 'o', 'o', 't']]' to std::vector<std::string,std::allocator<std::string> >  StatusCode=FAILURE

jmcarcell commented 1 month ago

Only k4FWCore, and that's weird, I can run it fine with multiple files or overriding the file from the CLI. Did you use k4_local_repo? which k4run should correctly point to the local version. How are you passing the arguments?

giovannimarchiori commented 1 month ago

I am setting up the working area like this:

git clone git@github.com:HEP-FCC/FCC-config.git
git clone git@github.com:key4hep/k4FWCore.git
git clone git@github.com:key4hep/k4geo.git
git clone git@github.com:HEP-FCC/k4RecCalorimeter.git

I setup the environment with

source /cvmfs/sw-nightlies.hsf.org/key4hep/setup.sh
cd k4RecCalorimeter && k4_local_repo && cd ..
cd k4geo && k4_local_repo && cd ..
cd k4FWCore && k4_local_repo && cd ..

I compile all packages

I run with something like

k4run python/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_HCal_ON_digi_reco.py --IOSvc.Input root/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_sim.root --IOSvc.Output root/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_HCal_ON_digi_reco.root > log/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_HCal_ON_reco.log

where the steering script contains:

...
from k4FWCore import IOSvc
from Configurables import EventDataSvc
io_svc = IOSvc("IOSvc")
io_svc.Input = inputfile
io_svc.Output = outputfile
ExtSvc += [EventDataSvc("EventDataSvc")]
...

from k4FWCore import ApplicationMgr
applicationMgr = ApplicationMgr(
    TopAlg=TopAlg,
    EvtSel='NONE',
    EvtMax=Nevts,
    ExtSvc=ExtSvc,
    StopOnSignal=True,
)

giovannimarchiori commented 1 month ago

I am quite lost... now the script seems to work, but a similar one fails with the same errors... working script: https://gitlab.cern.ch/gmarchio/FCC-scripts/-/blob/main/run_ALLEGRO_reco.py?ref_type=heads invoked with:

k4run python/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_HCal_ON_digi_reco.py --IOSvc.Input root/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_sim.root --IOSvc.Output root/allegro_v03_evts_10_pdg_22_MomentumMinMax_20_20_GeV_ThetaMinMax_90_90_PhiMinMax_1.570796_1.570796_HCal_ON_digi_reco.root

nonworking script: https://gitlab.cern.ch/gmarchio/FCC-scripts/-/blob/main/fcc_ee_samplingFraction_inclinedEcal.py?ref_type=heads invoked with:

k4run ../FCC-scripts/fcc_ee_samplingFraction_inclinedEcal.py -n 10 --hists.energyAxis 10.0 --IOSvc.Input test/sampling/root/sampling_output_energy_10000_theta_-1_particle_e-_jobid_1_sim.root --Output.THistSvc "rec DATAFILE='test/sampling/root/calibration_sampling_output_energy_10000_theta_-1_particle_e-_jobid_1.root' TYP='ROOT' OPT='RECREATE'" --IOSvc.Output test/sampling/root/sampling_output_energy_10000_theta_-1_particle_e-_jobid_1_rec.root

To me IOSvc is configured the same way in the two scripts.

Error message showing that IOSvc.Input is ignored:

Error in <TFile::TFile>: file /home/gmarchio/work/fcc/allegro/fullsim/run/./ALLEGRO_calibration_sim.root does not exist
Traceback (most recent call last):
  File "/home/gmarchio/work/fcc/allegro/fullsim/k4FWCore/install/bin/k4run", line 246, in <module>
    main()
  File "/home/gmarchio/work/fcc/allegro/fullsim/k4FWCore/install/bin/k4run", line 178, in main
    load_file(file)
  File "/home/gmarchio/work/fcc/allegro/fullsim/k4FWCore/install/python/k4FWCore/utils.py", line 85, in load_file
    exec(code, globals())
  File "<string>", line 72, in <module>
  File "/home/gmarchio/work/fcc/allegro/fullsim/k4FWCore/install/python/k4FWCore/ApplicationMgr.py", line 92, in __init__
    podio_reader = PodioReader(path)
                   ^^^^^^^^^^^^^^^^^
  File "/cvmfs/sw-nightlies.hsf.org/key4hep/releases/2024-09-29/x86_64-almalinux9-gcc14.2.0-opt/podio/c43bdf25e3ab4a2f30f64c4fffd5126af0c62e3b_develop-ux5pj4/lib/python3.11/site-packages/podio/root_io.py", line 26, in __init__
    self._reader.openFiles(filenames)
cppyy.gbl.std.runtime_error: void podio::ROOTReader::openFiles(const vector<string>& filenames) =>
    runtime_error: File /home/gmarchio/work/fcc/allegro/fullsim/run/./ALLEGRO_calibration_sim.root couldn't be found or the "podio_metadata" tree couldn't be read.

jmcarcell commented 1 month ago

That should be fixed in https://github.com/key4hep/k4FWCore/pull/243.

key4hep / k4FWCore

IOSvc.Input property can't be set from CLI #241