Improve Bruker format readers

[dagewa]

The dxtbx handling of Bruker file formats is limited. FormatBruker relies on iotbx.detectors.BrukerImage. This makes some restrictive assumptions, such as 1024*1024 pixel images, which are not correct for detectors such as the new PHOTON-II. FormatBrukerPhotonII avoids using BrukerImage to work around the restrictions, but fails on datasets using Bruker's own compression scheme.

Arnaud Basle has been in touch with Bruker representatives, and obtained the latest frame file description: BISFrameFileFormats.zip. We may also be able to see the source for FrmUtility, which is Bruker's own reference example for how to read their .sfrm format.

Ideally, this knowledge should be incorporated into FormatBruker, so that this can read any Bruker image, without unwarranted assumptions.

[dagewa]

Here are some additional questions and answers from Arnaud and Bruker, specifically about the Photon II:

How is the beam center dealt with? It seems that in the header only the offsets of the beam center at 2theta=0 are saved. Could you explain?

The CENTER is the pixel value that corresponds to the values on ANGLES. RAW and UNWARP values are on this line. For a detector that doesn’t need a spatial correction, he values are h same.

Could you explain as well how works the "Adaptive oversampling"? I guess this is to increase the dynamic range to 163810?

I believe that "Adaptive oversampling" refers to the retake images that was done on CCD detectors. Doesn’t apply to CMOS. CMOS frames have 16381 dynamic range and we read ten images per frame.

If the pixel values are coded in 2 bytes how come the max value is 163810?

With 10 images of 16-bits, the final frame is 19-bits. Round it up to 2 bytes.

[graeme-winter]

Did Arnaud send an example set?

[dagewa]

We have the sources for FrmUtility now from Bruker (email to dials-support). From the readme:

Getting Started:

A good place to start is with these source files and routines:

FrmRoutines.cpp: see ReadFrameFile() and SaveFrameFile() routines. Frm2Frm\frm_lib.c & frm_lib.h: see basically the entire files.

And some general documentation for the Bruker Frame File format is:

help\BISFrameFileFormats.rtf

[dagewa]

"One caveat. FrmUtility was written before Bruker supported double frame data. Double frame data is either multiplied by 10 or 1000 and written as integers. I would convert all frames to doubles (apply LINEAR and baseline). See PDF file sent earlier."

[dagewa]

I have some more example images from a Photon II detector from Andrew Purkiss, shared here with permission. PhotonII_images.tar.gz

The image matrix_01_0001.sfrm hangs dials.image_viewer or dxtbx.print_header, whilst LT1_CRUK006_10_01_0001.sfrm is fine. It seems this is due to the BrukerImage class, which sets the detectorbase for this format: https://github.com/cctbx/cctbx_project/blob/master/dxtbx/format/FormatBrukerPhotonII.py#L70

FormatBrukerPhotonII works around deficiencies in FormatBruker (as discussed above), in such a way that the BrukerImage code is not needed (at least not required for get_raw_data). If the detectorbase line is removed, images from this detector can be processed.

However, that would be masking the problem. The real issue is that FormatBruker and BrukerImage cannot support the Photon II, at the very least because they hardcode the assumption of 1024*1024 pixel images. I had a look at the code, but haven't worked out how to fix it yet. This might need input from @nksauter. I think we have enough information now about the Bruker format (see zip file linked to above) to implement complete support for Bruker detectors, in either "86" or "100" format.

[dagewa]

I should note, this is not time critical, but we do want to keep this on the radar. It seems to be possible to process most Photon II images fine. These matrix_*sfrm images are collected as part of the indexing procedure rather than full data collection.

[dagewa]

NB https://github.com/cctbx/dxtbx/pull/637 extended support to the Photon-III detector