Pixel masking and other features

[laborleben]

I changed the code but the main loop is still the same, though. The following points were changed/added:

1. HitClusterizer parameters can be set during creation.
2. Changed default parameters to be more generic.
3. Removed HitClusterizer.create_cluster_hit_info_array() since it does nothing.
4. Added min_hit_charge parameter for correction factor during calculation of charge weighted hit center.
5. Added noisy_pixels parameter to prevent cluster building (single hit) from noisy pixel.
6. Added disabled_pixels parameter to ignore hits from given pixels completely.
7. Reduced memory footprint a little.
8. Simplified cluster loop and separate cluster building and event building from cluster loop.
9. Added unit tests.

The performance decrease is ~5% but memory usage is less because the new code avoids the temporary arrays.

The new code is more flexible when certain changes are needed.

[coveralls]

Coverage increased (+0.5%) to 96.491% when pulling 0b85d1148b7a455cb50e11b916a71bc33ae4ce43 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[DavidLP]

Was the speed change tested with Mimosa26 data?

[coveralls]

Coverage increased (+0.5%) to 96.482% when pulling d56520eaf4350fa66e0da18e87c404ae77d0982d on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.5%) to 96.482% when pulling d56520eaf4350fa66e0da18e87c404ae77d0982d on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.693% when pulling a17ffe8784a293ec72421b737d05ac0d3ca4b6e7 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.693% when pulling a17ffe8784a293ec72421b737d05ac0d3ca4b6e7 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[laborleben]

Using threaded clusterizer (8 cores, 6x MIMOSA, 2x FEI4) the speed is the same if not marginally faster. Memory footprint is ~100MB less.

The test was without masking and/or disabling pixels while clustering.

[laborleben]

Ran also pyBAR unit tests and all test were OK.

[laborleben]

Compared the cluster files of some larger test beam runs an they are the same before and after changing the code.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 73f02e2989e8ee96842037a1fe34f05dad2baea0 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 73f02e2989e8ee96842037a1fe34f05dad2baea0 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 3db85dc1af9749f475b186dd4f13f4c55afad28b on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 2d8addc4a42246b3d92a3b3ebb911d4bd4b98294 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.3%) to 96.245% when pulling 18a7b11a0ae822f1e87484390521b4bd73de66b7 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 18a7b11a0ae822f1e87484390521b4bd73de66b7 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling 18a7b11a0ae822f1e87484390521b4bd73de66b7 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling b297fbf64758533a6e14eea2704b8d7cde4ac2fa on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.705% when pulling b297fbf64758533a6e14eea2704b8d7cde4ac2fa on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.693% when pulling 8e40fedf32bbabb7558d064f34bf1bf610be7c91 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.693% when pulling 8e40fedf32bbabb7558d064f34bf1bf610be7c91 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[laborleben]

Why is HitClusterizer.get_hit_cluster() and HitClusterizer.get_cluster() necessary?

[coveralls]

Coverage decreased (-0.3%) to 95.676% when pulling 5f1bcddde87a48d7eed226db70d081f243ce3221 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage decreased (-0.3%) to 95.676% when pulling 5f1bcddde87a48d7eed226db70d081f243ce3221 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[DavidLP]

To my mind I can merge, anything else you want to push to this branch?

[laborleben]

Let me optimize the pixel mask check.

[DavidLP]

Why is HitClusterizer.get_hit_cluster() and HitClusterizer.get_cluster() necessary?

In the past the cluster hit array was not always created/returned. I guess now these methods are not needed anymore and can be deleted.

[coveralls]

Coverage increased (+1.006%) to 96.974% when pulling 26dd36901b795b4aa98a99af38dd29ea3d9b7462 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.006%) to 96.974% when pulling 428097847d8c6e2c234ca4a2f0f56a2b6130e89f on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.006%) to 96.974% when pulling 428097847d8c6e2c234ca4a2f0f56a2b6130e89f on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.006%) to 96.974% when pulling 4301b2253724f9642f9333a86793af1b7f1511dd on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.006%) to 96.974% when pulling 4301b2253724f9642f9333a86793af1b7f1511dd on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.835% when pulling 7c91933edb529d4357fbcdfc1375ea7aa909e4ed on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.844% when pulling 7aa29872c18e1894e7ae94acaa19eba77dc713c5 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.844% when pulling 7aa29872c18e1894e7ae94acaa19eba77dc713c5 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.844% when pulling 3bcc7d563eda9b1c175ce93ebaaf0c3a2558f9d5 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[laborleben]

All the changes are done, tests and examples fully working. The code needs a review.

Some open questions and comments:

1. The pixel masking cannot be accelerated (matrix method) as long as the maximum/minimum value of the column and row index is unknown. Additional checks have to be implemented. The implementation as of now requires only little setup effort for the clusterizer which is an advantage. The disadvantage is, when less information is given less sanity checks can be made and some speed improvements are not possible.
2. The arrays are allocated outside the jitted functions (pure python). The size of the result arrays is dynamically adjusted and depends on the input hits array size. Additional code prevents the generation and assignment of the arrays each time the clusterizer.cluster_hits() is called. I don't see any increase of the runtime, and the array size does not need to be adjusted anymore.

[coveralls]

Coverage increased (+0.9%) to 96.875% when pulling 3425e9588d40f5b502ff6914a7d5076f65a0a1ed on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.875% when pulling 3425e9588d40f5b502ff6914a7d5076f65a0a1ed on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.884% when pulling 257b6cf1c9983d9b25158ac97df0560ce4a428ee on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.884% when pulling 29385b729ab1b56eafb1e32c312a791699d795f2 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.879% when pulling d0faaecf1f71f1313cbc0fb6c06b1d3d9d8a9e02 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.879% when pulling d0faaecf1f71f1313cbc0fb6c06b1d3d9d8a9e02 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.879% when pulling cff77f8d654173c972229781f1defc81b1aec38c on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.871% when pulling d77fa6202a55128d6c194d5be4ab79dc50ba4bc3 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+0.9%) to 96.871% when pulling d77fa6202a55128d6c194d5be4ab79dc50ba4bc3 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.5%) to 97.445% when pulling 74bbf291227080022ff08faa08c10291fb35ba50 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.5%) to 97.445% when pulling 74bbf291227080022ff08faa08c10291fb35ba50 on pixel_masking into 31057d45291e69f769c932215e40c001e9767542 on master.

[coveralls]

Coverage increased (+1.3%) to 97.445% when pulling 36f0332c7c195502216b38bb4434f394bd7853ba on pixel_masking into b641fa326dcffd3f319a15513f0ef55af29aa853 on development.

[laborleben]

All is done!

[DavidLP]

Can you deactivate python 3.4 unit tests? If anaconda cannot provide working libraries for this version, lets just skip this version, no?