Karabo bridge detector data format does not match the format at the SPB beamline

[valmar]

The format of the detector data streamed by the simulator is:

(16, 128, 512, 64)

That is: (panel, y, x, frames)

However the format has been recently changed at SPB to:

(64, 16, 512, 128)

To match the format of the offline data.

The problem can be corrected using the numpy.moveaxis function:

array1=numpy.moveaxis(array, -1, 0) array2=numpy.moveaxis(array1, -2, -1)

[antarcticrainforest]

Sorry for the late response @valmar. The karabo-bridge is rather agnostic in sending data. The shape is related to the calibration pipeline. Yet, users at SPB requested some shape changes of the data which we decided not to implement the recent update of the simulator as it should also be consistent with the shapes at other instruments. In fact the we're planning to reverse the special shape at SPB at some point rather than cater for those in the simulator. I hope that answers your questions and makes somehow sense to you :).

[takluyver]

We're also pushing for being able to send an array with labelled axes, so you can look at e.g. the 'frames' dimension rather than assuming that's the last dimension.

[valmar]

@antarcticrainforest I am actually the person who requested the shape change. Please do not revert it. It makes the shape of the online data match the shape of the offline data, allowing the online analysis people to share bad pixel maps, and especially geometry files with the offline analysis people.... It is one of the biggest improvement to the data processing pipeline that I have seen in recent times. Please reconsider your decision!

[valmar]

If you are going to revert this, can you please share the code that you use to change the format layout, so we can implement this client side?

[tmichela]

@valmar We decided for now to support both data layout.

I plan to have as default the shape coming from the calibration, but can be changed on the fly to set the shape as stored in files. The reason not to have this as default is that it takes ~1second for AGIPD to be reshaped. If you want to use the the file shape, you can warn us before the experiment, and it can also be changed on the fly during an experiment.

If you want to do the reordering on your side, you can simply use numpy:

# (mod, fs, ss, pulse) -> (pulse, mod, ss, fs)
array = np.transpose(array, (3, 0, 2, 1))
# if you need contiguous array in memory
array = np.ascontiguousarray(array)

[valmar]

Thank you very much.

Should I then close this?

On Thu, Apr 18, 2019 at 1:51 PM, Thomas Michelat notifications@github.com wrote:

We decided for now to support both data layout.

I plan to have as default the shape coming from the calibration, but can be changed on the fly to set the shape as stored in files. The reason not to have this as default is that it takes ~1second for AGIPD to be reshaped. If you want to use the the file shape, you can warn us before the experiment, and it can also be changed on the fly during an experiment.

If you want to do the reordering on your side, you can simply use numpy:

(mod, fs, ss, pulse) -> (pulse, mod, ss, fs)

array = np.transpose(array, (3, 0, 2, 1))

if you need contiguous array in memory

array = np.ascontiguousarray(array) — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/European-XFEL/karabo-bridge-py/issues/48#issuecomment-484471922, or mute the thread https://github.com/notifications/unsubscribe-auth/AAEILBKERJ6S7R65DJS4PVLPRBOCJANCNFSM4HC6EXBQ.

{"api_version":"1.0","publisher":{"api_key":"05dde50f1d1a384dd78767c55493e4bb","name":"GitHub"},"entity":{"external_key":"github/European-XFEL/karabo-bridge-py","title":"European-XFEL/karabo-bridge-py","subtitle":"GitHub repository","main_image_url":"https://github.githubassets.com/images/email/message_cards/header.png","avatar_image_url":"https://github.githubassets.com/images/email/message_cards/avatar.png","action":{"name":"Open in GitHub","url":"https://github.com/European-XFEL/karabo-bridge-py"}},"updates":{"snippets":[{"icon":"PERSON","message":"@tmichela in #48: We decided for now to support both data layout.\r\n\r\nI plan to have as default the shape coming from the calibration, but can be changed on the fly to set the shape as stored in files. The reason not to have this as default is that it takes ~1second for AGIPD to be reshaped.\r\nIf you want to use the the file shape, you can warn us before the experiment, and it can also be changed on the fly during an experiment.\r\n\r\nIf you want to do the reordering on your side, you can simply use numpy:\r\npython\r\n# (mod, fs, ss, pulse) -\u003e (pulse, mod, ss, fs)\r\narray = np.transpose(array, (3, 0, 2, 1))\r\n# if you need contiguous array in memory\r\narray = np.ascontiguousarray(array)\r\n"}],"action":{"name":"View Issue","url":"https://github.com/European-XFEL/karabo-bridge-py/issues/48#issuecomment-484471922"}}} [ { "@context": "http://schema.org", "@type": "EmailMessage", "potentialAction": { "@type": "ViewAction", "target": "https://github.com/European-XFEL/karabo-bridge-py/issues/48#issuecomment-484471922", "url": "https://github.com/European-XFEL/karabo-bridge-py/issues/48#issuecomment-484471922", "name": "View Issue" }, "description": "View this Issue on GitHub", "publisher": { "@type": "Organization", "name": "GitHub", "url": "https://github.com" } } ]

[tmichela]

I keep it open until we add this in the simulation.