K2 IS binary data reader

[sk1p]

• [x] figure out guarantees (is block ordering deterministic?, are frame IDs for blocks really monotonic or can they "interlace" so to speak?, ...)
• [x] figure out what data we need to discard ("desynced" frames at the beginning and end)
• [x] integrate into LiberTEM as DataSet implementation
• [x] split sectors / binary files into smaller partitions
• [x] don't require direct mapping of
  • [x] sectors to partitions
  • [x] blocks to tiles
• [ ] crop frames to user specified area (see comment by @woozey below, and #28 for the general case)
• [x] optimize conversion uint12 -> uint16 (see #108)
• [x] read tags from gtg file (for us most important: scan dimensions, see #109)

[woozey]

@sk1p

1) that the data was acquired in IS mode (the gtg file says it was, any other way to find out?)

It is in IS mode. The K2 camera outputs into binary format only when it is in IS mode. The data we are looking at is however "STEMx" data which means that K2 is in IS mode and synchronised with scanning. I'm not sure if that would change the version in the header. My guess that it should not, but we can double check it looking at regular IS data...

2) what firmware version is/was installed on the K2

Is: 1.01.315

[sk1p]

@woozey thanks! Comparison with non-IS data would be useful indeed.

In 509f86ac3bc7335bd62f5218bf8d3a143154229b I added synchronization using the shutter_active flag (first bit of byte 9 in the header). This is the result, again using the 02_0V dataset:

As you can see, the synchronization is almost perfect. It looks like it skips one frame more than with skip_frames=57.

[sk1p]

Regarding the version, this is a complete hexdump of a header of one of the blocks:

['0xff', '0xff', '0x00', '0x55', '0x00', '0x00', '0x00', '0x00', '0x01', '0x01', '0x00', '0x00', '0x00', '0x00', '0x00', '0x0c', '0x07', '0xe9', '0x0d', '0x20', '0x01', '0x00', '0x07', '0x44', '0x00', '0x00', '0x00', '0x3c', '0x00', '0xf0', '0x00', '0x00', '0x00', '0xff', '0x03', '0xa1', '0x00', '0x00', '0x57', '0x58']

Now, the 9th byte should contain the version. It is 0x1. Even a off-by-one error would result in 0x1 or 0, I don't see how I can arrive at 2 here.

[uellue]

Cropping frames in an UDF is probably more or less as good as cropping in the IO layer since frame cropping will usually give a strided data structure with small pieces that is not efficient to read, in contrast to ROIs in navigation space.

Cropping can be added to an existing UDF through a mixin using multiple inheritance. See https://github.com/LiberTEM/LiberTEM/blob/master/src/libertem/udf/blobfinder.py for an example of this pattern, in that case for postprocessing!

For that reason I am closing this issue now.