search

AutoDQM / AutoDQM_ML

0 stars 5 forks source link

reject files not ending in .root when fetching #31

Open chosila opened 1 year ago

chosila commented 1 year ago

Currently run 2022 seems to have many files that end in .root.dqminfo, which when loaded by datafetcher throws and error such as 

WARNING  [DataFetcher : load_data] Problem loading file                                                                                                                                        data_fetcher.py:265
     'root://eoscms.cern.ch//eos/cms/store/group/comm_dqm/DQMGUI_data/Run2022/SingleMuon/0003541xx/DQM_V0001_R000354113__SingleMuon__Run2022A-PromptReco-v1__DQMIO.root.dqminfo', it might                    
     be corrupted. We will just skip this file.

It does not halt the fetching as the fetcher just skips the file, but it may prevent user from knowing if one of the files really was corrupted and needs to be reported.

Perhaps we can make sure that the file ends in .root before loading the files into fetcher.

GluonicPenguin commented 1 year ago

I've added a few lines in the data fetcher here to handle these (you might have done sth similar): https://github.com/AutoDQM/AutoDQM_ML/pull/34

Specifically these lines: https://github.com/AutoDQM/AutoDQM_ML/blob/e28989dd78ee716532ecc469db56fed32a8287f8/autodqm_ml/data_prep/data_fetcher.py#L140-L142