MiniCBF spec for Pilatus detectors

[jordiandreu]

Dear Lima users,

Some time ago we start using Lima to save the images coming from a Pilatus2 Dectris detector.

Now, the header is written via LimaCCDs DS using the attribute "saving_header_delimiter" and the command "setImageHeader".

However, I find some difficulties to properly format the header according to CBF/CIF specs for pilatus.

1) The _array.header_convention section is always added just before the _array.data. Should this header convention entry be placed just before the _array.header_contents entry? However, the official dectris parsing python script runs ok when parsing the header file were the concention entry is not before the header_contents entry.

2) We need to properly add and place the convention value "PILATUS_1.2" (keeping the double quotes for campitibility to some processing software, even the quotes are not mandatory according the the CBF/CIF specs)

This is the current Lima configuration for the image header in a python script:

headers = list()
header = "........whatever here will be placed in the contents section......"
headers.append('%d : array_data/header_convention|%s;' % (i, '"PILATUS_1.2"'))
headers.append("%d : array_data/header_contents|%s;" % (i, header))
det.saving_header_delimiter = "|",";",":"
det.setImageHeader(headers)

Can someone help or provide some feedback about how setting the attribute and format the headers string to match these requirements?

[laurent-claustre]

Hi Jordi, could you provide us two examples of data file, one generated by pilatus and one by lima, in order to understand the difference in between them.

[jordiandreu]

Here you are:

Header from image saved by dectris Pilatus2-6M:

###CBF: VERSION 1.5, CBFlib v0.7.8 - PILATUS detectors

data_nobeam_1_0001

_array_data.header_convention "PILATUS_1.2"
_array_data.header_contents
;
# Detector: PILATUS 6M, S/N 60-0108, Alba
# 2016-08-24T18:17:59.859

Header saved using Lima:

###CBF: VERSION 1.5
# CBF file written by CBFlib v0.8.0

data_image_0

_array_data.header_contents
;\
_array_data.header_convention "PILATUS_1.2"
# Detector: PILATUS 6M, S/N 60-0108, Alba
# 2018/Feb/27 13:18:19
# Pixel_size 172e-6 m x 172e-6 m

The problem is due to inserting the convention entry after the header contents entry and the slash after the semicolon. The Dectris documentation comes with a parsing script for the image headers which fails for the later but not for the former.

[laurent-claustre]

Hi, Now I understand. I suspect a mistake in the tango server setImageHeader command. I gonna test it.

[laurent-claustre]

Which Lima version version are you using? doing a test with latest Lima 1.8.2 and conda packages I got:

###CBF: VERSION 1.7.10
# CBF file written by CBFlib v0.9.5

data_image_0

_Misc.acq_autoexpo_mode OFF
_Misc.acq_expo_time 1
_Misc.acq_latency_time 0
_Misc.acq_mode Single
_Misc.acq_nb_frames 1
_Misc.acq_trigger_mode IntTrig
_Misc.image_bin <1x1>
_Misc.image_flip '<flip x : False,flip y : False>'
_Misc.image_roi <0,0>-<1024x1024>
_Misc.image_rotation Rotation_0

_array_data.header_contents
;
# Detector: PILATUS 300K, 3-0101
# 2011-07-22T17:33:22.529
;

_array_data.header_convention PILATUS_1.2
_array_data.data
;
--CIF-BINARY-FORMAT-SECTION--
Content-Type: application/octet-stream;
     conversions="x-CBF_BYTE_OFFSET"
Content-Transfer-Encoding: BINARY

And my python test is that script:

import PyTango
dev=PyTango.DeviceProxy('id00/limaccds/bcu_simulator1')
headers = list()
header = "# Detector: PILATUS 300K, 3-0101\n# 2011-07-22T17:33:22.529"
i=0
headers.append('%d : array_data/header_convention|%s;' % (i, 'PILATUS_1.2'))
headers.append("%d : array_data/header_contents|%s;" % (i, header))
dev.saving_header_delimiter = "|",";",":"
dev.setImageHeader(headers)
dev.saving_directory='/tmp/'
dev.saving_prefix='test_cbfheader_tango_'
dev.saving_format='cbf'
dev.saving_mode='auto_frame'
dev.saving_mode
dev.saving_suffix
dev.prepareAcq()
dev.startAcq()

The "problem" comes from the C++ std:Map we use to pass the headers, entries are sorted by alphabetic order. So convention will be always inserted after contents. I've not simple fix for you, could you tell me which dectris software you are using to read the image file?

[jordiandreu]

Hi @laurent-claustre ,

We are using LimaCore = 1.7.0 and Pilatus = 1.7.2. For testing the header format, we are using the python script attached as Appendix A in the official Pilatus CBF header specification (version 1.4) from Dectris.

The scientist complain that if the header does not follow this specifications, some post processing software fail o open the images.

[laurent-claustre]

Hi @jordiandreu, i did fix the issue, using C++11 unordered_map, Could you download the image file here and test with your script or scientist software if it fits with specifications: https://cloud.esrf.fr/s/myrFid54cjYq6G8

If that's ok, I will patch the current version of Lima 1.8 where we provide conda packages for an easier installation, or up to you to recompile or patch your local Lima 1.7 installation.

[jordiandreu]

Hi @laurent-claustre, The script works fine with this image header so you can go ahead with the fix. Thank you very much for your support.

[jordiandreu]

Hi @laurent-claustre, Has been already applied this change to the lima1.8 version?

[laurent-claustre]

Hi @jordiandreu, it is now availaible but within version 1.9, we just push on github.com and anaconda for conda packages.

[jordiandreu]

Many thanks, I've received the notification!

[jordiandreu]

Hi @laurent-claustre ,

Many thanks for fixing this issue regarding the header convention for Pilatus.

Here at ALBA, the Pilatus2 detector server has an old OS openSuSE 10.3. This means that the c++ compiler is also quite old (4.2.1) and I think this has no support for c++11 standard. You commented the changes made to implement the support to mini cbf header make use of the C++11 unordered_map.

I guess this force us to update the g++ compiler to some newer version (In addition, cmake requirements makes also us compile the cmake library for glib6, which I still have to compile the newer version).

Are you using the new lima 1.9 library in old OS? Have you tried to migrate Dectris camserver to some new OS to overcome the complatio issues?

[sdebionne]

On older OSes, we use the toolsets that comes with Conda (gcc, cmake and also some libraries). This is probably the easiest option if your platform supports conda:

conda create -n lima-dev python=3 cmake gxx_linux-64
source activate lima-dev

I'll let Laurent comment on the Dectris camserver migration...

[jordiandreu]

I guess conda is not available for openSuSE 10.3...

[sdebionne]

I think conda requires glibc 2.12 (2010) so you might be out of luck indeed...

[sdebionne]

It might actually be easier for you to backport the fix to 1.7... Could you try to cherry pick 62cbf23a20e0b6d0edfbd1c0abad31c8305600e0?

[jordiandreu]

I tried to cherry pick the code and compile but failed. I guess the compiler is to old (gcc version 4.2.1). ..

[laurent-claustre]

Hi @jordiandreu, yes I did try the same, but the code uses some C++ features like for example using vector_of_int = typename std::vector<int>; which are not supported by gcc 4.2.1 I will prepare a compatible patch for you on branch core-1.7

[jordiandreu]

@laurent-claustre , merci beaucoup! I'll appreciate it so much.

[laurent-claustre]

Dear @jordiandreu, never late, I've just pushed on branch 1.7 the fix. Could you test on your side, please?

[jordiandreu]

Dear @laurent-claustre , I have compiled the new version without problems, but I will not have access to the detector until next week (monday, probably). I let you know ASAP!

[jordiandreu]

Dear @laurent-claustre,

I could test the changes today ( I had a small slot for it) and the changes work fine, creating a valid miniCBF header (according to Dectris parser).

So, many thanks for your help and dedication.

[laurent-claustre]

Good news, thanks for the report.