legend-exp / legend-dataflow

LEGEND data flow management
Other
2 stars 11 forks source link

`tcm.array_id` is a double-precision float in v02.00, but was an integer in v01.06 #26

Closed gipert closed 10 months ago

gipert commented 11 months ago

...and this breaks the DataLoader:

File /opt/anaconda3/lib/python3.11/site-packages/pygama/flow/data_loader.py:1010, in DataLoader.load(self, entry_list, in_memory, output_file, orientation, tcm_level)
   1007     raise ValueError("need to set output columns to load data")
   1009 if orientation == "hit":
-> 1010     self.data = self.load_hits(entry_list, in_memory, output_file, tcm_level)
   1011 elif orientation == "evt":
   1012     if tcm_level is None:

File /opt/anaconda3/lib/python3.11/site-packages/pygama/flow/data_loader.py:1097, in DataLoader.load_hits(self, entry_list, in_memory, output_file, tcm_level)
   1094 if tb not in col_tiers[tier]:
   1095     continue
-> 1097 tb_name = self.filedb.get_table_name(tier, tb)
   1098 tier_paths = [
   1099     os.path.join(
   1100         self.data_dir,
   (...)
   1104     for file in files
   1105 ]
   1107 tier_table, _ = sto.read_object(
   1108     name=tb_name,
   1109     lh5_file=tier_paths,
   1110     idx=idx_mask,
   1111     field_mask=field_mask,
   1112 )

File /opt/anaconda3/lib/python3.11/site-packages/pygama/flow/file_db.py:700, in FileDB.get_table_name(self, tier, tb)
    698     keyword = names[0]
    699     args = {keyword: tb}
--> 700     table_name = template.format(**args)
    701 else:
    702     table_name = template

ValueError: Unknown format code 'd' for object of type 'float'

Should we blame https://github.com/legend-exp/legend-dataflow/commit/07f7103e7645cbbf6528d88d59d01cd0a905bcd6?

CC @jasondet

gipert commented 11 months ago

I guess this is the problem:

https://github.com/legend-exp/pygama/blob/7467d987cf3fa03cd5b470f9fc4e18ad8e611b6f/src/pygama/evt/tcm.py#L84

Fix is maybe to just pass dtype=int to numpy.full_like()?

gipert commented 10 months ago

@ggmarshall could you re-generate the TCM files with the fixed pygama?