search

DUNE / data-mgmt-ops

3 stars 3 forks source link

Minerva metadata conversion crash in decladd #670

Closed dougbenjamin closed 3 months ago

dougbenjamin commented 4 months ago

07/09/2024 09:01:05.764: Mover: Mover failed: TS1_00001698_0027_pdstl_v09_2405141423_RawData.dat status: declaring to SAM error: Traceback (most recent call last): File "/exp/dune/app/home/dunepro/declad_py311/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run result = task.run() ^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/mover.py", line 326, in run s_metadata = sam_metadata(self.FileDesc, metadata, self.Config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/init.py", line 104, in sam_metadata out = mc.convert_all_mc_sam(metadata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/metadata_converter.py", line 379, in convert_all_mc_sam res[ck] = converter(v, md) ^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/metadata_converter.py", line 11, in convert_date_mc_sam dt = datetime.datetime.fromtimestamp(date) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'str' object cannot be interpreted as an integer

StevenCTimm commented 4 months ago

OK it is crashing on the created_timestamp field in the json which is a quoted numeric (float) string. Just grep -v created_timestamp out of the existing json files and they will be good to go. And change the minerva script to not put it in.

It will be generated automatically.

StevenCTimm commented 4 months ago

(Thanks Matt Kramer for catching the above).

StevenCTimm commented 4 months ago

There's a second problem with that metadata:

         [50017, 1]

[dunepro@duneingestgpvm01 dropbox]$ more TS1_00050017_0001_numip_v09_2407081843_RawData.dat.json { "checksums": { "adler32": "cd344547" }, "created_timestamp": "1720472915.193868", "metadata": { "core.data_stream": "numip", "core.data_tier": "binary-raw", "core.end_time": 1720472914, "core.event_count": 9000, "core.file_format": "binary", "core.file_type": "importedDetector", "core.first_event_number": 26392630, "core.group": "minerva", "core.last_event_number": 26401629, "core.lum_block_ranges": [ [ 26392630, 26401629 ] ], "core.run_type": "teststand", "core.runs": [ [50017, 1] ], "core.runs_subruns": [ 706732705 ], "core.start_time": 1720464241, "dune.daq_test": "True", "misc.file_partition": 1, "online.triggerconfig": "NOFILE", "online.triggertype": "oneshot", "retention.status": "active", "retention.class": "commissioning" }, "name": "TS1_00050017_0001_numip_v09_2407081843_RawData.dat", "namespace": "neardet-2x2-minerva", "size": 1224641778 }

Note the core.runs list it is not right This is one run and one subrun, core.runs should have only the run number in it \

in stead of "core.runs": [ [50017, 1] ],

it should be "core.runs": [ [50017] ],

StevenCTimm commented 4 months ago

data was moved to minerva subdirectory of the dropbox until we can deal with it. and dunedecladgpvm01 restarted.

StevenCTimm commented 4 months ago

also noted in slack that the core.data_stream contained some new values numil and numib that we did not know about before, we had to add those.. also some of them had core.run_type of test stand rather than neardet-2x2-minerva as it was supposed to be, that was also fixed.

StevenCTimm commented 3 months ago

This is all fixed, closing