search

European-XFEL / EXtra-foam

Fast Online Analysis Monitor at EuXFEL
https://extra-foam.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
9 stars 10 forks source link

Further features #290

Open jo-moeller opened 3 years ago

jo-moeller commented 3 years ago

Hi, as previously discusssed with @dallanto and James, we would need a few additions for online analysis with extra-foam at MID.

Some good to haves: 1) log-colorscale on the detector images 2) Instead of a running average in the overview tab, display a running std/var/min/max as well. 3) Saving AGIPD masks generated in extra-foam in a (16,512,128) format, without applying the geometry to it. (Already discussed in #116, but not solved yet)

JamesWrigley commented 3 years ago

Finding the beam center, displaying a running average, and thresholding the incoming data are implemented. Support for circular ROI's is nearly done, all that's left to do is actually compute and apply the mask (which should be easy, but it's late :stuck_out_tongue: ). I also want to clean up the GUI controls for it, but I think that would take a bit of refactoring, which probably belongs in another PR.

WRT point 2., do you mean displaying the running std/var/min/max over an ROI, i.e. scalar values? Or computing them for each pixel and displaying them as images?

Demo: foo

The code is in the mid_special_suite branch if you want to have a look. It may be completely broken on systems other than my own though.

ebadkamil commented 3 years ago

Thanks @JamesWrigley for the nice work. I have few questions.

dallanto commented 3 years ago

Impressive work, James!

jo-moeller commented 3 years ago

Cool, looks very nice. I'll try to give it a closer look before the weekend.

JamesWrigley commented 3 years ago
  • Will the possibility of Circular ROI, be available only in this special suite or do you plan to provide it in core ImageTool too? It will be nice to have both Circular and RectROI.

Just in this special suite for now, but I could certainly extend it to the ImageTool as the next step. I think the nicest way would be to add an extra widget to the ROI controls to allow the user to select the shape of the ROI they want. That would need some modification to the _SingleRoiCtrlWidget class, perhaps to make it 'own' the ROI object instead of letting it be passed in.

  • You are using AzimuthalIntegrator from pyFAI and setFit2D to evaluate beam center. Did you test the performance with the functionality of beam center available in EXtra-foam core.

Ah, no. I actually wasn't aware that EXtra-foam was capable of finding the beam center, where can I find the code for that? I tried grepping the codebase for 'beam' and 'center' but couldn't find anything.

  • what is the rate of processing for SpeckleContrastProcessor since it also assembles the image using extra-geom AGIPD assembler.

Absolutely terrible :stuck_out_tongue: Like, 0.5 - 1 Hz on my decently powerful i7. Is there another way to assemble the image that would be faster?

And finding the beam center is also really slow, on my machine it takes ~25s. On the plus side it should be easy to parallelize parts of it and run it in a separate process so it doesn't block the GUI. Another option would be to implement it in the C++ core or use Cython (not sure how complicated either of those would be).

jo-moeller commented 3 years ago

And finding the beam center is also really slow, on my machine it takes ~25s. On the plus side it should be easy to parallelize parts of it and run it in a separate process so it doesn't block the GUI. Another option would be to implement it in the C++ core or use Cython (not sure how complicated either of those would be).

Finding the beamcenter is something you would do once at the beginning of each shift. No problem if that is not super fast.

ebadkamil commented 3 years ago

Absolutely terrible stuck_out_tongue Like, 0.5 - 1 Hz on my decently powerful i7. Is there another way to assemble the image that would be faster?

Something like this where the assembler is implemented in C++. Take a look at foamalgo

https://github.com/European-XFEL/EXtra-foam/blob/976bce5458aa974637bf9c89c9adeecc0d9054be/extra_foam/geometries/__init__.py#L336

Ah, no. I actually wasn't aware that EXtra-foam was capable of finding the beam center, where can I find the code for that? I tried grepping the codebase for 'beam' and 'center' but couldn't find anything.

I think foamalgo implements ConcentricRingFinder

zhujun98 commented 3 years ago

Detecting beam center is already implemented https://extra-foam.readthedocs.io/en/latest/image_tool.html#concentric-rings, which takes a few hundreds of milliseconds, depending on the image size. I don't see the point of reinventing the wheel here.

zhujun98 commented 3 years ago

Adding the circular ROI is a good progress :+1: . But I would suggest to think about corner cases (e.g. on the edge of an image) and performance.

JamesWrigley commented 3 years ago

Something like this where the assembler is implemented in C++. Take a look at foamalgo

Ah thanks, will look into that.

WRT finding the beam center, I believe the reason for this different method is to handle images that don't have well-defined concentric rings. I played around a little with the concentric ring feature on the dataset I'm testing with, and the calculated beam center isn't that stable if you play around with the settings for prominence/min. count etc (though this could also be because of my inexperience with the tool).

zhujun98 commented 3 years ago

The reason for the hyperparameters prominence and min count is because the behavior of detectors and experiments are very different. The user should roughly know their data. Of course, this is the first implementation and I had no chance to collect user feedback. But I tested with all the detectors and different datasets that I have. They all worked.

zhujun98 commented 3 years ago

I believe the reason for this different method is to handle images that don't have well-defined concentric rings

If there are no rings, one could simply find the center of mass of the spot, which is how people do in experiments.

JamesWrigley commented 3 years ago

Ah, I see. @jo-moeller, has that already been tried? I could test it if not.

jo-moeller commented 3 years ago

Hey, concerning finding the beam center there are basically two different types of data which have radial symmetry. Crystalline material can have well defined Bragg-rings, that's where the "concentric ring" feature would work well. And obviously it does. I can give it a try as well.

The other type of samples are unordered or non-crystalline. They might have some oscillating features, like the example data set we used. Or only very broad rings. They might also just have a I(q) ~ q^-4 power law all over the range of the detector. Here the beam center finding that I suggested normally works quite well. Center of mass could also work, but I think the one that James now implemented works better if there are some features like oscillations.

Some example plots are in our beamline paper, figure 10: https://journals.iucr.org/s/issues/2021/02/00/ay5570/ay5570.pdf

jo-moeller commented 3 years ago

Maybe it would be worth converging both approaches into one feature?

JamesWrigley commented 3 years ago

Yep, can look into that :)