zinger removal ?

[marcocamma]

Hi,

long before pyfai existed I had my own python azimuthal averaging script. I am very happy not to use it anymore as pyfai is far superior.

There is only one feature I missed: the zinger removal. While calculating the average (for a given q-ring) my program removed any pixel that was > THRESHOLD*STD where THRESHOLD was an external parameter and STD was the STD of all pixels in the q-ring (after mask & polarization correction). It is very well possible that there are better ways ...

Would it be possible to add such a feature to pyfai ? or does it impact only a small fraction of the user community and as such it does not make much sense ?

keep up with the great work, marco

[CJ-Wright]

Have you seen this paper? http://scripts.iucr.org/cgi-bin/paper?fv5059 It uses pyFAI and is currently in both scikit-beam and xpdAn

[marcocamma]

Thanks for the answer, no i hadn't (it is indeed pretty new!) indeed it is similar to the approach I described. I did a quick test with the skbeam 0.10 but it is a bit too slow (~11 sec on a 1920x1920 image). pyfai needs 0.09 sec on the same image.

It seems that mask.binned_outlier is not actually using pyfai, am i right ? could this be the reason for the relatively slow speed ?

anyway, i can also write a small function script but i still the point is still valid, would it make sense to have it directly in pyfai ? marco

[kif]

On Wed, 01 Mar 2017 13:35:53 -0800 marco cammarata notifications@github.com wrote:

Thanks for the answer, no i hadn't (it is indeed pretty new!) indeed it is similar to the approach I described. I did a quick test with the skbeam 0.10 but it is a bit too slow (~11 sec on a 1920x1920 image). pyfai needs 0.09 sec on the same image.

It seems that mask.binned_outlier is not actually using pyfai, am i right ? could this be the reason for the relatively slow speed ?

anyway, i can also write a small function script but i still the point is still valid, would it make sense to have it directly in pyfai ?

Indeed, it may be worth to add such a function in pyFAI.utils.

I don't know how you intent to use this function, so I let you suggest the signature (the function name and its parameters).

Cheers,

Jerome

[marcocamma]

Thanks @kif for your answer.

In my old version it was done at the same time as the 1d integration (it is probably better in terms of performance). in this case it would be a keyword argument of integrate1d

I don't know enough about the inner workings of pyfai to understand if this is possible or how easy it is

As a standalone routine a dezinger like methods could actually be very useful anyway: it is indeed used to find bragg peaks in (serial) protein crystallography (see NanoPeakCell).

For this reason as standalone function a more general name like 'azimuthal_smoothing' might be better than 'dezinger'.

In terms of signature I would say something like: pyFAI.utils.azimuthal_smoothing(img,aimuthalIntegratorInstance,npt,method=np.median,mask=None, threshold=5,error_model='azimuthal')

The idea of passing the method is to have some flexibility. For example while using the difference between a given pixel and the median of all pixels that contribute to a given ring might work in most cases, it might fail if there are too zingers.

I put together a running example you can find (together with real data) here: https://cloud.ipr.univ-rennes1.fr/index.php/s/DZs6jnfsZcNfipD

just open ipython and type %run dezinger.py or (import dezinger; dezinger.test()) It is very slow (~7.5 sec for a 1920x1920 image) but at least is self contained and the logic behind clear (I hope)

thanks very much, marco