rjones30
commented
3 years ago I narrowed down the differences in the reconstruction output between repeated identical runs over the same raw data to the way that dEdx information is being reported for hits in the CDC. The dEdx hits lists in my debug printouts are identical for the two trials for most events, but not all. A quick scan reveals that the differences are seen in the CDC dEdxHits lists, where about one track out of 100 shows some difference somewhere in the list. Using vimdiff I went through and made a detailed comparison between the two lists. This is what I see.
- All of the differences lie in the CDC hits lists; the FDC hits lists are all identical in this sample of 923 events.
- Most of the differences can be explained as insertions/deletions: some hits included in one list are not found in the other.
- Sometimes, but rarely, the two dE values for a hit agree but the dx values are different.
- Most of the time the dE and dE_amp values are similar in magnitude, but sometimes the dE value is much larger, eg. dE=9.323973e-01, dE_amp=1.788133e-06, dx=1.635120.
- Most of the time the dx value is reasonable, but sometimes it is too large to be realistic, eg. dE=6.554818e-06, dE_amp=5.198256e-06, dx=8494.518240.
Regarding the large values of dE, this appears to not be so unusual. Here is a plot of the distributions of dE and dE_amp values from this sample of 923 events. I have plotted the x axis on a log scale to make the difference in scales between the two parts of the distribution visible.
The dE_amp distribution looks normal, but what are these dE values way up at one? They only comprise 1% of the total statistics on the dE variable, but they should be understood. Even more interesting, they do not appear in the dE_amp distribution. Looking at the debug listing, these exceptional dE values always appear at the end of the CDC hist list for their track, as if the hits list had been ordered in increasing dE, although this is clearly not the case because the ordering is not exact. The corresponding dE_amp values for the same hit look normal.
Repeated runs with the standard libraries and plugins for the Summer 2020 raw data reconstruction campaign do not produce the same number of entries in the histograms in hd_root.root. Here are some repeated tests with the full complement of plugins (29 of them) running with the standard version_4.24.0.xml binaries and a single thread over hd_rawdata_071728_000.evio block number 336. The results from each run show the total number of entries in hd_root.root in histograms of a given type, as reported by my rootmem.py utility. All of these repeats were done under identical conditions, one worker thread, one EVIO decoding thread.
Trial 1:
Trial 2:
Trial 3:
Notice that the differences in the numbers of entries appear in histograms of all types, so there should be no difficulty finding a specific example to drill down on.