Find a reliable way to match FESA wr-events and picoscope triggers #28

Open alexxcons opened 3 years ago

In GitLab by @alexxcons on Feb 5, 2019, 16:56

Major problem: On lower sampling rates the delay between two streaming_work calls on the DigitizerBlock can be ~100ms. (On high sampling rates it goes down to ~15ms, which still is not that nice)

Because of that, the timestamp used for the trigger tag is not very precise. TODO: Check how the precision of that stamp can be increased.

Check how the precision of the stamps can be increased E.g. /usr/include/picoscope/ps3000aApi.h provides some methods, like ps3000aGetTimebase and others which sound promising

There are as well methods in order to find triggerers

Problem: On lower sample rates, like 500k, time_realligment requires 80ms matching tolerance Where for 2MS sampling rate 40ms matching tolerance are fine. (TODO: time_realligment not uased any more .. check if fesa event matching behaves the same way)

Currently there seems to be a fixed precision of \~10µs (1MS samp_rate, 1kHz Sink)

Gaps of \~10µs can be observed whenever a new ref-event is used.

Theoretically the precision should be increased when using higher sampling rates. Practically this seems not to be the case. Theoretically the precision for 1MS should be 1µs (10MS --> 100ns)

The precision (\~10µs) correlates with the length of the cycle !!!

Find out why the precision does not behave like expected.

Find a reliable way to match FESA wr-events and utc_stamps !

In GitLab by @alexxcons on Feb 5, 2019, 16:56

changed weight to 20

In GitLab by @alexxcons on Feb 6, 2019, 09:43

changed title from {-Increase precision of utc_timestamp used to match WR-events with trigger tags in time_realligment-} to {+Find a reliable way to match FESA wr-events and picoscope triggers+}

In GitLab by @alexxcons on Feb 6, 2019, 09:43

changed the description

In GitLab by @alexxcons on Mar 1, 2019, 15:27

mentioned in commit DigitizerClass2@25011d9e8024446b459f9c5462bb9633424f59ca

In GitLab by @alexxcons on Mar 1, 2019, 15:27

mentioned in commit 7691a5c741ed5782cf1f460669debe822357ad42

In GitLab by @alexxcons on Mar 1, 2019, 15:27

mentioned in commit gr-flowgraph@7d76a064e6f64f9c6fa53b15d7f8bbbe8c23b47e

In GitLab by @alexxcons on Mar 1, 2019, 16:37

mentioned in commit 4b9086aa1f25ca9564866b88818effc4ab72e2a1

In GitLab by @alexxcons on Mar 1, 2019, 16:45

mentioned in commit b33e19d526138d16e0ffe2b598ce837b9f6e4cae

In GitLab by @alexxcons on Mar 1, 2019, 16:47

mentioned in commit c77954ac2a2a56039f1a2c203e46647348d32868

In GitLab by @alexxcons on Sep 6, 2019, 17:59

This patch should fix the described problems by taking Timestamps directly on the Digitizer Callback, instead of taking them in the gnuradio work_stream method ... to be tested !!

0001-Find-a-reliable-way-to-match-FESA-wr-events-and-pico.patch

In GitLab by @alexxcons on Sep 6, 2019, 17:59

changed the description

In GitLab by @alexxcons on Feb 26, 2020, 14:20

changed the description

In GitLab by @alexxcons on Feb 26, 2020, 14:21

changed the description

In GitLab by @RalphSteinhagen on May 3, 2020, 07:57

the time-realignment has been defunct but is crucially needed to add user-delays prior to further processing in other modules. This is crucially needed e.g. for the actual-vs-reference comparisons, \chit^2-fit modules, kicker delay re-trigger module etc.

In order to fulfill this day-0 specified functionality, the WR integration into the GR graph needs to be reworked. If the WR events cannot be injected within the time-realignment block, than one should consider these as optional block prior to the GR digitizer sources.

In GitLab by @alexxcons on May 3, 2020, 23:59

changed the description

In GitLab by @alexxcons on Sep 25, 2020, 19:30

Still trouble with events which are not matched (E.g. missing FULL_SEQ updates) Matching tolerance up to 20ms required.

While it works better than before, event matching still is far from good and very complex. I would call the "matching tolerance approach" to be failed.

Instead of fixing the problem via "matching tolerance" in the lockfree circularbuffer, it should be fixed where it arises, which is inside the DigitizerBlock. (Already discussed this with R.Steinhagen)

The source of the problem: d_streaming_callback, which is passed to the picoscope drivers is called at random/unknown times. The UTC stamp taken from there currently is used to align data in time. We should not do that any more. Best dont rely on that stamp at all.

Here a concrete proposal for a first fix:

method "InjectEvent" on Digitizer Block to inject stamps and payloads via FESA (can be moved out to separate block in a second iteration)
Digitizer Block uses passed stamps as timestamp for trigger-events and adds payload to tags
Digitizer only takes own UTC stamp if no external stamp is available ... as soon as there is an external stamp, use sample-counting to tag data with timestamps.
Digitizer Block counts received triggers and received events, reports error if both do not match ( to many received triggers in queue should result in a restart)
Digitizer reports error if distance between trigger tags[count samples] does not match distance between used WRstamps (to much deviation should result in a restart)
Fesa class directly passes stamp into circularBufferLibrary via addTriggerMetaData .. no need to mofify/allign that stamp in FESA or circBuffer any more
Remove triggerMatchingTolerance, n_matching_triggerevents, delayWRStampToTriggerStamp, etc from lockfreeCircularBuffer

In GitLab by @RalphSteinhagen on Sep 25, 2020, 21:48

What about using PMT messages to model the raw timing events, to inject them via a PDU input port to and merge them within the digitizer block?

This way the digitizer block could be kept a bit more generic and more easily reusable by systems w/o WR timing card, or FESA server, or those who want to synthesis the timing events.

The idea would be to have the following timing source blocks (i.e. with a PDU/PMT message output port) that are either fed by: a) programmatic internal API (e.g. call-back function triggered by FESA) b) block that implements a generic WR timing receiver (via saftlib) c) block that receives SW-based (i.e. via UDP) timing events and forwards them d) programmed event table that is synced to the UTC second (host clock).

----------------                       -----------------
|               |-----        ---------|               |------
| timing source | PDU| | ---- | timing |  gr-digitizer | out1 |----- etc.
|               |-----        ---------|               |......
----------------                       |               |
                                       |               |------
                                       |               | out2 |----- etc.
                                       |               |------
                                       |               |
                                       ----------------

In GitLab by @alexxcons on Sep 28, 2020, 10:29

Thanks you for the details !

Yes, would be ok for me to take a try for PMT messages for that service. Though I would split the issue: first a quick fix of the current situation, using a simple callback, later on support for an deddicated block which replaces the callback.

I opened #60 for the second step now.

fair-acc / gr-digitizers

Find a reliable way to match FESA wr-events and picoscope triggers #28