Closed dougbenjamin closed 1 month ago
OK it is crashing on the created_timestamp field in the json which is a quoted numeric (float) string. Just grep -v created_timestamp out of the existing json files and they will be good to go. And change the minerva script to not put it in.
It will be generated automatically.
(Thanks Matt Kramer for catching the above).
There's a second problem with that metadata:
[50017, 1]
[dunepro@duneingestgpvm01 dropbox]$ more TS1_00050017_0001_numip_v09_2407081843_RawData.dat.json { "checksums": { "adler32": "cd344547" }, "created_timestamp": "1720472915.193868", "metadata": { "core.data_stream": "numip", "core.data_tier": "binary-raw", "core.end_time": 1720472914, "core.event_count": 9000, "core.file_format": "binary", "core.file_type": "importedDetector", "core.first_event_number": 26392630, "core.group": "minerva", "core.last_event_number": 26401629, "core.lum_block_ranges": [ [ 26392630, 26401629 ] ], "core.run_type": "teststand", "core.runs": [ [50017, 1] ], "core.runs_subruns": [ 706732705 ], "core.start_time": 1720464241, "dune.daq_test": "True", "misc.file_partition": 1, "online.triggerconfig": "NOFILE", "online.triggertype": "oneshot", "retention.status": "active", "retention.class": "commissioning" }, "name": "TS1_00050017_0001_numip_v09_2407081843_RawData.dat", "namespace": "neardet-2x2-minerva", "size": 1224641778 }
Note the core.runs list it is not right This is one run and one subrun, core.runs should have only the run number in it \
in stead of "core.runs": [ [50017, 1] ],
it should be "core.runs": [ [50017] ],
data was moved to minerva subdirectory of the dropbox until we can deal with it. and dunedecladgpvm01 restarted.
also noted in slack that the core.data_stream contained some new values numil and numib that we did not know about before, we had to add those.. also some of them had core.run_type of test stand rather than neardet-2x2-minerva as it was supposed to be, that was also fixed.
This is all fixed, closing
07/09/2024 09:01:05.764: Mover: Mover failed: TS1_00001698_0027_pdstl_v09_2405141423_RawData.dat status: declaring to SAM error: Traceback (most recent call last): File "/exp/dune/app/home/dunepro/declad_py311/lib64/python3.11/site-packages/pythreader/task_queue.py", line 210, in run result = task.run() ^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/mover.py", line 326, in run s_metadata = sam_metadata(self.FileDesc, metadata, self.Config) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/init.py", line 104, in sam_metadata out = mc.convert_all_mc_sam(metadata) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/metadata_converter.py", line 379, in convert_all_mc_sam res[ck] = converter(v, md) ^^^^^^^^^^^^^^^^ File "/exp/dune/app/home/dunepro/declad_py311/declad/declad/custom/metadata_converter.py", line 11, in convert_date_mc_sam dt = datetime.datetime.fromtimestamp(date) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ TypeError: 'str' object cannot be interpreted as an integer