search

art-framework-suite / art

The implementation of the art physics event processing framework
Other
2 stars 7 forks source link

SAM Process ID in metadata #35

Open knoepfel opened 2 years ago

knoepfel commented 2 years ago

This issue has been migrated from https://cdcvs.fnal.gov/redmine/issues/7081 (FNAL account required) Originally created by @rocconnick on 2014-09-29 14:54:59

Robert Illingworth pointed out that it would be helpful to store the SAM process ID in the file metadata. It would be nice if ART could pick this up and add it. NOvA could build this into it's own metadata module, but Robert thought it is an appropriate all-experiment feature. The art_sam_wrap.sh shell script stores the process id in a variable called $consumer_id, so the art code could try to key off that. Otherwise, it seems appropriate to pass it in as an environment variable, art_sam_wrap.sh could be modified to pass it in.

knoepfel commented 2 years ago

Comment by @chissg on 2014-09-29 16:51:46

According to the code, the services.FileCatalogMetadata.processID parameter (settable via the --sam-process-id option) is already added to the file catalog metadata as process_id. Please let us know whether this item is present in the output of sam_metadata_dumper for an output file that you know used SAM as input, as the absence of this item would indicate a bug which should be found and fixed.

knoepfel commented 2 years ago

Comment by @rocconnick on 2014-10-01 14:35:53

It appears that the --sam-process-id flag can only be passed in the presence of --sam-web-uri. In our case, art_sam_wrap.sh does the fetching of the file, so we don't pass in --sam-web-uri. I think that the desired behavior is to require the process ID in the presence of a web URI, but not vice versa. Is that a sensible alternative? Here's the output I see when --sam-web-uri is not included. 

$ nova -c /grid/fermiapp/nova/novaart/novasvn/releases/development/ProductionScripts/fhicl/prod_reco_numi_job.fcl /nova/ana/users/rocco/tmp/neardet_genie_fhc_nonswap_geantonly_1000_r01000095_s54_S14-08-01_v1_20140620_170832_fnpc2058.fnal.gov_1404203105_4746_0_sim.pid.root --sam-process-id 99
OptionsHandler caught a cet::exception calling art::FileCatalogOptionsHandler::doProcessOptions()
---- Configuration BEGIN
  configurations services.user.CatalogInterface.webURI (--sam-web-uri) and
  services.FileCatalogMetadata.processID (--sam-process-id) must be specified
  together or not at all.
---- Configuration END

Art has completed and will exit with status 7002.

When I do pass them together, I get a different exception, but it may be simply because of a poorly formed URI when trying to run on a local file. 


$ nova -c /grid/fermiapp/nova/novaart/novasvn/releases/development/ProductionScripts/fhicl/prod_reco_numi_job.fcl /nova/ana/users/rocco/tmp/neardet_genie_fhc_nonswap_geantonly_1000_r01000095_s54_S14-08-01_v1_20140620_170832_fnpc2058.fnal.gov_1404203105_4746_0_sim.pid.root --sam-process-id 99 --sam-web-uri stuff --sam-data-tier=out1:pid --sam-stream-name=out1:0
%MSG-s ArtException:  nova 01-Oct-2014 09:27:25 CDT JobSetup
cet::exception caught in art
---- OtherArt BEGIN
  ServiceCreation
  ---- Type mismatch BEGIN
    processID
    ---- Type mismatch BEGIN
      can't obtain atom from sequence
    ---- Type mismatch END
  ---- Type mismatch END
  cet::exception caught during construction of service type art::FileCatalogMetadata:
---- OtherArt END
%MSG
Art has completed and will exit with status 1.