Add flower wave option

[cozzyd]

This (so-far untested) PR adds the option to take FLOWER data before or after running, in compressed json format. Similar to running flower-wave before or after a run, but automated.

Doing it during running is a bit more complex because there is no way to effectively synchronize between the RADIANT and FLOWER, though it could be done via best effort via timestamps. But that's way more complicated, has to deal with some concurrency issues (since the same SPI link is used for data taking and slow control), and would benefit from a better defined binary output.

You can ask for a number of force triggers as well as a number of seconds of RF triggers. The force triggers are taken first. Note that the force triggers could actually be RF triggers if an RF trigger happens in the window between clearing the buffer and sending the force trigger. Avoiding this would require temporarily disabling the trigger, which I didn't do yet.

This PR also includes a few miscellaneous changes, changing where the output directories are created so that they can be written to!

[fschlueter]

@cozzyd I have a few questions:

• Is it correct that the chance to have an RF trigger while taking forced trigger is reasonable small? Since we clear the buffer just before we send the "command" to record a force trigger or is the latency large enough that this could happen on a regular basis.
• Is it somehow possible to get the trigger type from the incoming data stream and store it with the waveforms?
• Can we store a timestamp with the waveforms?
• Is the opposite case also possible: That we take RF trigger and a force trigger smuggles in?
• Can we differentiate between LT and RADIANT triggers?

[cozzyd]

@cozzyd I have a few questions:

* Is it correct that the chance to have an RF trigger while taking forced trigger is reasonable small? Since we clear the buffer just before we send the "command" to record a force trigger or is the latency large enough that this could happen on a regular basis.

I would expect that most of the time the two commands should be executed very close to each other (a few microseconds? can check), but there's no guarantee that there is no context switch in between, which can take an unbounded amount of time (but likely ms?). I can execute them as the same system call which would guarantee that there is no context switch in between, but we'd have to test that the FLOWER can handle that...

At a ~Hz RF trigger rate for the FLOWER you can estimate how often you accidentally get a coincidence.

* Is it somehow possible to get the trigger type from the incoming data stream and store it with the waveforms?

Yes, it looks like the trigger type is stored on the FLOWER. I'm not sure if we've tested this on this firmware, but I will add the readout and modify accordingly.

* Can we store a timestamp with the waveforms?

Yes. Both readout time and an attempt at the clock on the FLOWER, though I don't remember if that's the "right" clock or not. Adding that...

* Is the opposite case also possible: That we take RF trigger and a force trigger smuggles in?

That shouldn't be possible, unless somehow a force trigger command was sent to the FLOWER from a different process, but that would be quite hard to do without breaking everything.

* Can we differentiate between LT and RADIANT triggers?

RADIANT triggers don't go to the FLOWER board so everything is a FLOWER trigger.

[cozzyd]

ok, now implemented reading FLOWER metadata as well as optionally clearing the trigger in the same syscall as the force trigger in libflower.so (e.g. https://github.com/RNO-G/librno-g/commit/547bcef909bde14cfb318ba39201ddf2af291f2e, https://github.com/RNO-G/librno-g/commit/5e692f010ee212941004bd1eccd448039e8bd1d0 ) and propagated those (untested) changes here. I just went ahead and committed those changes in librno-g without a branch/PR since they shouldn't break anything (except for maybe flower-wave...).

[fschlueter]

@cozzyd great work! This will be super useful. Have you thought about the possibility to take regular forced trigger with the flower board during the run? would that be possible (to the cost of some deadtime)? I could imagine it is not easily possible because we would need to tell the radiant to not do something during that time? But I thought I bring it here up briefly.

[cozzyd]

I don't know if a flower force trigger sends a trigger to the radiant (though would be easy to find out and also easy to add in firmware if it doesn't). If we don't insist they are coincident in time, then we could do this in the servo thread without much thinking.

[fschlueter]

I would be fine with that (not coincident in time). I will test what you already have today or tomorrow

[cozzyd]

You'll have to update librno-g and make install-daq. Just do you know one of the branches has large test files in it that I need to remove (since too annoying in Greenland) but for now you can just git pull the master branch to avoid that.

[fschlueter]

I have been running the flower-wave option for the end of the run with the default settings. I have not looked at the data yet but I am getting the following errors in the stdout/err. This might be a librno-g problem but I post it here.

[cozzyd]

that's printed out by https://github.com/RNO-G/librno-g/blob/master/src/flower.c#L615

So it's possible I remembered something wrong and that we don't expect the register to be echoed back in the first byte. It would be helpful to print out the values of all the tested words (e.g. p/x ev_counter.bytes in gdb, after breaking on that line).