search

dials / dials

Diffraction Integration for Advanced Light Sources
https://dials.github.io
BSD 3-Clause "New" or "Revised" License
70 stars 47 forks source link

DIALS fails to import cbf from Pilatus3 1M at BL02B1, SPRING-8 #2488

Open ndevenish opened 1 year ago

ndevenish commented 1 year ago

Raised in dials-support. Fails at:

  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/FormatCBFMiniPilatus.py", line 28, in __init__
    super().__init__(image_file, **kwargs)
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/FormatCBFMini.py", line 74, in __init__
    super().__init__(image_file, **kwargs)
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/FormatCBF.py", line 59, in __init__
    super().__init__(str(image_file), **kwargs)
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/Format.py", line 168, in __init__
    self.setup()
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/Format.py", line 184, in setup
    detector_instance = self._detector()
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/FormatCBFMiniPilatus.py", line 66, in _detector
    for f0, f1, s0, s1 in determine_pilatus_mask(detector):
  File "/usr/local/dials-dev20230526/modules/dxtbx/src/dxtbx/format/FormatPilatusHelpers.py", line 152, in determine_pilatus_mask
    assert (n_fast - 1) * detector.gap_fast == remainder
AssertionError: Please report this error to dials-support@lists.sourceforge.net:

In the CBF:

X-Binary-Size-Fastest-Dimension: 1043
X-Binary-Size-Second-Dimension: 981

This is inverted from what we would expect with any normal Pilatus - it looks like the images have been purposefully rotated or transposed. Unfortunately, I can't see an obvious, clean way to fix this without treating it as a separate detector.

I have the start of a basic WIP attempt at https://github.com/dials/dxtbx/compare/main...cctbx:dxtbx:cbf_inverted_pilatus that just transposes the data; this loads into image viewer, but falls over trying to read/generate masks (which is already required for spotfinding. Unfortunately, the C++ Detector objects seem to be initialised differently from the python geometry models, so it is not so easily overridden.

graeme-winter commented 1 year ago

For reference, via Kunio Hirata

https://www.jasri.jp/content/files/beamlines/02b1.pdf

graeme-winter commented 1 year ago

Bottom left image, shows detector in conventional orientation, so they have modified the pixels in an interesting way

biochem-fan commented 1 year ago

XRDa just got a new entry from SPring-8 BL02B1. Is this useful?

https://xrda.pdbj.org/entry/155

graeme-winter commented 1 year ago

Very much so thank you

graeme-winter commented 1 year ago

OK, I think I can see how Crysalis works now, because lots of inf files like

{
HEADER_BYTES= 4096;
BYTE_ORDER=little_endian;
CCD_DETECTOR_ADC_OFFSET=0;
CCD_DETECTOR_DESCRIPTION=Mercury2 (1x1 bin mode);
CCD_DETECTOR_DIMENSIONS=981 1043;
CCD_DETECTOR_IDENTIFICATION=DECTRIS;
CCD_DETECTOR_OPTIONS=trigger:off;
CCD_DETECTOR_SIZE=168.7 179.4;
CCD_DETECTOR_TEMPERATURE=23.0 deg C;
CCD_DETECTOR_VECTORS=1 0 0 0 1 0;
CCD_FIELD_OF_VIEW=Round;
CCD_GONIO_DESCRIPTION=AFC8: 2theta arm;
CCD_GONIO_NAMES= RotAboutBeam 2Theta RotY XShift YShift Distance;
CCD_GONIO_NUM_VALUES=6;
CCD_GONIO_UNITS=deg deg deg mm mm mm;
CCD_GONIO_VALUES=0.0000 0.000 0.0000 0.0000 0.0000   130.00;
CCD_GONIO_VECTORS=0.0000 0.0000 1.0000 1.0000 0.0000 0.0000 0.0000 1.0000 0.0000 1.0000 0.0000 0.0000 0.0000 1.0000 0.0000 0.0000 0.0000 -1.0000;
CCD_NONUNF_INFO=None;
CCD_NONUNF_TYPE=None;
CCD_NONUNF_TYPE_=None;
CCD_SERIAL_NUMBER=999;
CCD_SPATIAL_BEAM_POSITION=501.16 527.65;
CCD_SPATIAL_DISTORTION_INFO=501.16 527.65 0.17200 0.17200;
CCD_SPATIAL_DISTORTION_TYPE=Simple_spatial;
CCD_SPATIAL_DISTORTION_VECTORS=0 -1 1 0;
CCD_TAPER_ORIENTATION=+x+y;
CCD_UNBINNED_BEAM_POSITION=501.16 527.65;
CCD_UNBINNED_DIMENSIONS=981 1043;
COMMENT=Cintegrate refinement results;
COMPRESSION=None;
CRYSTAL_GONIO_DESCRIPTION=AFC8: Eulerian 3-circle;
CRYSTAL_GONIO_NAMES= Omega Chi Phi;
CRYSTAL_GONIO_NUM_VALUES=3;
CRYSTAL_GONIO_UNITS= deg deg deg;
CRYSTAL_GONIO_VALUES=179.5 45.000 270.000;
CRYSTAL_GONIO_VECTORS=1.0000 0.0000 0.0000 0.0000 -1.0000 0.0000 1.0000 0.0000 0.0000;
DARK_PEDESTAL=20;
DETECTOR_NAMES=PILATUS_CdTe_;
DETECTOR_NUMBER=1;
DIM=2;
DTDISPLAY_ORIENTATION=+X+Y;
Data_type=unsigned short int;
FILENAME=ZNTPP_full_08001.img;
OPTICS_COLLIMATOR=Unknown;
OPTICS_TYPE=Synchrotron;
ROTATION=179.5 180.0 0.5 1.0 0.000 0.000 0.000 100.000 0.000 0.000;
ROTATION_AXIS_NAME=Omega;
ROTATION_VECTOR=1.000 0.000 0.000;
SATURATED_VALUE=1048575;
SCAN_CRYS_RELZERO=3.000 0.000 45.000 0.000;
SCAN_DETECTOR_OPTIONS=;
SCAN_DET_RELZERO=2.000 -30.000   130.00;
SCAN_MODE=Scan_Open;
SCAN_ROTATION=179.5 180.0 0.5 1.0 0.000 0.000 0.000 100.000 0.000 0.000;
SCAN_ROTATION_AXIS_NAME=Omega;
SCAN_ROTATION_VECTOR=1.000 0.000 0.000;
SCAN_SEQ_INFO=1 1       360;
SCAN_TEMPLATE=test;
SCAN_WAVELENGTH=0.3100;
SIZE1=981;
SIZE2=1043;
SOURCE_AMPERAGE=100.0000 mA;
SOURCE_CROSSFIRE=0.0002 0.0002 0.0000 0.0000;
SOURCE_FOCUS=1.00000;
SOURCE_POLARZ=1.0000 1.0000 0.0000 0.0000;
SOURCE_REFINE_FLAGS=0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0;
SOURCE_SIZE=0.0000 0.0000 0.0000 0.0000;
SOURCE_SPECTRAL_DISPERSION=0.00020 0.00020;
SOURCE_VALUES=0.0000 0.0000;
SOURCE_VALUES_SIGMA=0.0000 0.0000;
SOURCE_VECTORS=0.000 0.000 1.000 0.000 1.000 0.000 1.000 0.000 0.000;
SOURCE_VOLTAGE=8.0000 GeV;
SOURCE_WAVELENGTH=1.0000 0.3100;
SOURCE_WAVELENGTH_SIGMA=0.000100;
}
graeme-winter commented 1 year ago

Header also somewhat incomplete

_array_data.header_convention "PILATUS_1.2"
_array_data.header_contents
;
# Detector: PILATUS3 1M, S/N 10-0163
# 2023-06-27T10:50:14.622
# Pixel_size 172e-6 m x 172e-6 m
# CdTe sensor, thickness 0.001000 m
# Exposure_time 0.9990000 s
# Exposure_period 1.0000000 s
# Tau = 0 s
# Count_cutoff 1169254 counts
# Threshold_setting: 19996 eV
# Gain_setting: autog (vrf = 1.000)
# N_excluded_pixels = 1157
# Excluded_pixels: badpix_mask.tif
# Flat_field: FF_p0_E39993_T19996_vrf_m0p100.tif
# Trim_file: p0_E39993_T19996.bin
# Image_path: /ramdisk/
# Ratecorr_lut_directory: ContinuousStandard_v1.1
# Retrigger_mode: 1
# Wavelength 0.31003 A
;

I can probably kludge a custom format class but would not be happy

ndevenish commented 1 year ago

This one has a timestamp at least

biochem-fan commented 1 year ago

The inf file looks like what FormatSMVRigakuSaturn.py expects.

What happens if I read the pixel values from CBF and concatenate it below the inf file (with offset 4096)? Will FormatSMVRigakuSaturn read it? Let me try...

biochem-fan commented 1 year ago

What happens if I read the pixel values from CBF and concatenate it below the inf file (with offset 4096)? Will FormatSMVRigakuSaturn read it? Let me try...

After some tweaks to header items (DETECTOR_NAMES, DTREK_DATE_TIME, DETECTOR_IDENTIFICATION, DETECTOR_DESCRIPTION), image reading and spot finding are OK but indexing is not successful yet.

biochem-fan commented 1 year ago

Is the origin of PILATUS/EIGER detectors at the top left corner, looking from the crystal to the detector? The RIGAKU detectors (e.g. HyPix) have the origin at the lower left corner. The SPring-8 people might have rotated pixels to match the latter convention.

biochem-fan commented 1 year ago

What happens if I read the pixel values from CBF and concatenate it below the inf file (with offset 4096)? Will FormatSMVRigakuSaturn read it? Let me try...

This worked. My ad-hoc converter and processing log are in https://github.com/dials/dials_scratch/tree/master/tn/SPring-8-BL02B1.

biochem-fan commented 1 year ago

Multi-sweep indexing didn't work. Clearly the fixed rotation and/or the setting rotation are wrong...

biochem-fan commented 1 year ago

Multi-sweep indexing worked (https://github.com/dials/dials_scratch/commit/850c36cfc25967d28ade1a00d5d33045d31a6662). The CHI axis in the INF file was wrong.

Well, not completely yet. Sweep 1+4 is OK but not others.

biochem-fan commented 1 year ago

I asked the depositor of the dataset (Dr. Sasaki). Sweeps 5 to 7 are from the same crystal but with the detector two theta at -30, even though the header says 0!

Indeed sweeps 1 to 4 can be jointly indexed.

biochem-fan commented 1 year ago

@graeme-winter I am in contact with Dr. Nakamura at JASRI/SPring-8, whose program writes these INF files. Once I figure out all geometries, I will tell them to use NeXus or full CBF, instead of their own metadata format (or I will write a converter). Another CX beamline at SPring-8, BL40XU, uses EIGER, but with a non-NeXus compliant format. Two theta, chi etc are stored in a non-NeXus way using Dectris SIMPLON API. For example, the file uses the NXtransformations class but lacks depends_on fields.

Do you have an example of full CBF and/or NeXus files with multiple axes (e.g. detector two theta, phi, chi, omega)? The formal specification is hard to read. Apparently our FormatHDF5EigerNearlyNexus supports only one axis. Are there existing attempts to read other axes from a master H5?

biochem-fan commented 1 year ago

I figured out all axes (phi, chi, omega, two_theta).

All sweeps indexed simultaneously https://github.com/dials/dials_scratch/commit/625d10c7b2d1985e46887928700b4ceb4299d803:)

ndevenish commented 1 year ago

btw @graeme-winter is on leave for a couple of weeks

Is the origin of PILATUS/EIGER detectors at the top left corner, looking from the crystal to the detector? The RIGAKU detectors (e.g. HyPix) have the origin at the lower left corner. The SPring-8 people might have rotated pixels to match the latter convention.

You probably worked this out, but yes - the PILATUS/EIGER read from the top-left corner - this is why this set was specifically confusing because the image data fast/slow were inverted in size, which indicated they were doing "something" to the data - and it wasn't obvious if that meant that they had transposed or rotated the image.

The test data in XRDa apparently didn't undergo the swapping of the fast and slow axes unlike the dataset @ndevenish got

Right, that makes it somewhat easier to read. My guess is that without knowing what happened to that data (and with no timestamp to tell the time range that it was happening), that we're probably just not going to be able to process it?

biochem-fan commented 1 year ago

@ndevenish

Yes, Dr. Nakamura explained me what happens at the beam line.

First, they have "raw" miniCBF as written by the PILATUS software, together with metadata in the beam line specific format (the INF files). XRDa-155 contains these. Next, they run a conversion program, which reads miniCBF and INF and writes updated CBF files that matches CrysAlisPro's convention. Here, the 2D image array is transposed and flipped along the fast axis. Probably what you got are CBF files after this conversion.

I told them that this transposition and flipping are very bad for DIALS (and in general). I recommended them to use full CBF or NeXus to properly store all metadata. Now that I understood their geometry, I will write a conversion program for them.

ndevenish commented 1 year ago

Ah! Thanks for the explanation, that does indeed sound like an... interesting transformation to apply to the data.

A conversion program sounds like the best approach. I'm not sure how easy it would be to put a special-case version of pilatus just for this beamline.

I would also encourage them writing a timestamp into the files they do this to!