Andreas was right, dictionaries are not for big data. I implemented (literally) everything using arrays. Now the results are incredible. For npileup = 200, the runtime of the efficiency rate dropped from 1-2 years to 1 minute. Amazing. Here are the results:
The Efficiency looks nice. The duplicate/fake rates are an issue. The purification algorithm takes a lot to compute, since basically it runs the rz Hough Transform for each bin. I may need to think of a smarter solution to overcome this problem. Maybe NNs are the way to go, so I will invest most of my time there.
asalzburger
commented
2 years ago
Andreas was right, dictionaries are not for big data. I implemented (literally) everything using arrays. Now the results are incredible. For
npileup = 200, the runtime of the efficiency rate dropped from 1-2 years to 1 minute. Amazing. Here are the results:The Efficiency looks nice. The duplicate/fake rates are an issue. The purification algorithm takes a lot to compute, since basically it runs the rz Hough Transform for each bin. I may need to think of a smarter solution to overcome this problem. Maybe NNs are the way to go, so I will invest most of my time there.