Timer Block (not planned) and pause mechanism

[occoder]

Hi It seems to be important to introduce a Timer block into Crappy. Because it's very often that an experiment needs to be operated according to the time counting, say dwelling at operating point A for X min then dwelling at operating point B for Y min. Also sometimes, an experiment may need to be paused. Then the remaining time needs to be remembered and resumed later on. So please consider to introduce a Timer block. Thanks

[WeisLeDocto]

Hi !

Because it's very often that an experiment needs to be operated according to the time counting

The Generator Block can already put devices on hold following a specific pattern. For example, if it's driving a motor, it can specify times when the motor should stop for a given delay. Using a combination of Generator and Modifier objects, it is possible to achieve quite complex command patterns already.

Also sometimes, an experiment may need to be paused

That is a known current limitation of Crappy, there is no specific mechanism for putting an entire experiment on hold at once. That is because Crappy is mostly designed to achieve reproducibility between several runs: the same code should always drive a given setup in a reproducible way (modulo variability in external factors, e.g. sample stiffness). It would technically be possible to write a Block that drives equipment in an interactive way (through a graphical interface for example). However, combining the interactive approach of such a Block with the deterministic one of the Generator seems a bit contradictory and would likely prove quite difficult.

Another approach could be to act at the Blocks level to temporarily stop their event loop, for example using a global flag. However, a basic, naive implementation wouldn't do: a motor already running won't stop until it is told to do so. Therefore this kind of approach would also require quite some engineering.

Does that answer your question ? Maybe you had something else in mind for a hypothetical Timer Block ?

[occoder]

It would technically be possible to write a Block that drives equipment in an interactive way (through a graphical interface for example). However, combining the interactive approach of such a Block with the deterministic one of the Generator seems a bit contradictory and would likely prove quite difficult.

Please don't use any interactive element. Leave the UI part to the end user. The user should be reponsible for the conversion of UI event to trigger signal that can be digested by Crappy. (separation of concerns) Another approach could be to act at the Blocks level to temporarily stop their event loop, for example using a global flag. However, a basic, naive implementation wouldn't do: a motor already running won't stop until it is told to do so. Therefore this kind of approach would also require quite some engineering.

A pause is not a stop. The pause just freezes everything except for timestamp. If a motor is running in a constant speed, it should keep the speed. If it is accelerating, then pause just does not apply at the accelerating moment. After all, the acceleration won't last forever. Once the motor gets to the stable state, then it can be paused. Does that answer your question ? Maybe you had something else in mind for a hypothetical Timer Block ?

The hypothetical Timer Block could be a pure data processing block. It acts like a gate inbetween a upstream block and a downstream block, i.e. either pass-through or cut-off. The key factor is the accurate control (at sub-second level at least) over the gating timing. The time counting task could be a specific thread and its operation is under the control of cmd_label.

[WeisLeDocto]

Please don't use any interactive element. Leave the UI part to the end user.

I 100% agree, we're avoiding UI in the general case. Only for the configuration of cameras does it make sense to have a graphical interface. We once developed a UI interacting with Crappy for a specific application (https://github.com/LaboratoireMecaniqueLille/Braking_tribometer_command), and so did (at least) one user (#14).

If a motor is running in a constant speed, it should keep the speed.

Well, the original use case of Crappy is for tensile tests, and in these we definitely don't want the motor to keep running when no data is being acquired. Maybe you have examples in mind when disabling acquisition and control while still letting the actuators running is acceptable/desirable ?

It acts like a gate inbetween a upstream block and a downstream block, i.e. either pass-through or cut-off.

So, basically, you want to be able to enable or disable a flow of data based on some condition, right ? This is feasible, with variable complexity depending on what the condition is and how it is computed (acquired signal, predefined timestamps, etc.). Now, this Block (more likely Modifier) will only drive the flow of one stream of data. How is it related to your request for putting the entire experiment on hold ?

[occoder]

Well, the original use case of Crappy is for tensile tests, and in these we definitely don't want the motor to keep running when no data is being acquired. Maybe you have examples in mind when disabling acquisition and control while still letting the actuators running is acceptable/desirable ?

In this specific case of keeping motor speed, it just follows the pause definition, i.e. freezing the current state. Of course, you can leave the defintion to the end user. E.g. the pause can only be applied to mechanical static states.

So, basically, you want to be able to enable or disable a flow of data based on some condition, right ?

No. Disabling of data flow is just one aspect of the pause. BEFORE the time couting, a trigger command could be sent to related blocks to keep all related devices' states still. Likewise, AFTER the time couting, another trigger command could be sent to resume the states transition as well. And the key point here is always the time counting and actions are just attachements to the time counting events.

This is feasible, with variable complexity depending on what the condition is and how it is computed (acquired signal, predefined timestamps, etc.). Now, this Block (more likely Modifier) will only drive the flow of one stream of data.

Could you eleberate on how the time counting is done using the "predefined timestamps"? If this approach is generic enough, can it justify such a custom block being designed as a standard Block?

[WeisLeDocto]

When translating the feature you have in mind to an actual design, pausing Crappy simply means putting all the Blocks' event loops on hold, except for the the Timer Block. This would de facto disable data streams, and prevent any interaction of Crappy with hardware.

This would be quite unsafe IMO, as it would mean leaving hardware energized but unsupervised, even possibly still moving, heating up, etc. I can also think of cases when it would disrupt the flow of the experiment, e.g. for motors that require a continuous update of their command. But technically, it's just a matter of adding some shared flag driven by the Timer Blocks, and add a condition based on that flag there: https://github.com/LaboratoireMecaniqueLille/crappy/blob/e069fd4c6cf7209c56833290a24f3f4235c9684c/src/crappy/blocks/meta_block/block.py#L939-L949

May I ask you what use case you have in mind for such a feature ?

Could you eleberate on how the time counting is done using the "predefined timestamps"?

My comment was about pausing a single stream of data, but the following still stands true for a hypothetical Timer Block. In Crappy all the Blocks share a common time reference, so that they're always synchronized. I can think of two options for planning an event at a known time. Either use a fast but lightweight loop checking the current time and comparing it the the predefined timepoints, or use the sched standard library module.

[occoder]

May I ask you what use case you have in mind for such a feature ?

In fact, I'd like to run a kind of burn-in test that involves some conditioning steps. Each conditioning step lasts for certain period. But at the end of the period, a condition needs to be met. If current step fails to meet the condition, the step will be retried. There is max. number of retry that if run out the whole test will be terminated and deemed as a failed test. The test moves on to the next step if the condition check succeeds. A standard Timer block may help on time counting for each step and triggering condition check once current period of time runs out.

[WeisLeDocto]

I don't see the need for a Block putting the entire experiment on hold here, except maybe if you need to manipulate anything on your setup between conditioning steps. Your use case is clearly quite complex to implement. Still, the following pseudo-code can be a good start:

import crappy

# Generator that outputs the index of the condition to apply
step_path = [{'type': 'Constant', 'value': i, 'condition': f'next_step_id>{i}'} for i in range(...)]
step_id_gen = crappy.blocks.Generator(step_path, cmd_label='step_id'...)

# Perform action on the setup based on the received command
block_1 = SomeBlock(cmd_labels=('value_1', 'value_2', ...), ...)  # Just an example, might be a different syntax
# Acquire data from the setup
block_2 = SomeBlock(...)
...
# These Blocks might actually be a single IOBlock + InOut, or anything else

# User-defined table converting step index to command values
def index_to_values(data);
    if data['step_id'] == 0:
        data['value_1'] = ...
        data['value_2'] = ...
        ...
    elif data['step_id'] == 1:
        data['value_1'] = ...
        data['value_2'] = ...
        ...
    ...
    return data

# Convert step index to actual command values, and pass them to the driver Block
crappy.link(step_id_gen, block_1, modifier=index_to_values)

# Helper class checking whether a step was successful
class DataToIndex(crappy.modifier.Modifier):
    def __init__(self):
        super().__init__()
        self.index = 0
    def __call__(self, data):
        if <condition checking that step was successful>:
            self.index += 1
        data['next_step_id'] = self.index
        return data

# Parse data from setup to check if condition is met, increase next index counter if so 
# to switch to next step, and send to the Generator
crappy.link(block_2, step_id_gen, modifier=DataToIndex())

crappy.start()

It might not be sufficient if:

• Each conditioning step follows some different pattern that would be difficult to summarize with a small number of parameters
• There is no clear way for DataToIndex to know when a conditioning step is over, and thus to know when to check for the success condition and increment the index counter

However, with the information I have, this is the simplest code I can propose.

[occoder]

Thank you @WeisLeDocto , the example is very helpful.

I don't see the need for a Block putting the entire experiment on hold here, except maybe if you need to manipulate anything on your setup between conditioning steps.

Sorry, I forget to mention that during the day time, a worker has to pause the test before he enters into the testing area which is shared among multiple test setups. There is no clear way for DataToIndex to know when a conditioning step is over, and thus to know when to check for the success condition and increment the index counter

That's just where the proposed Timer block comes into play, because each step has a fixed lasting time. Therefore, the condition should not be checked all the time. Even if the condition is met early, the step should still continue until its time runs out. So only at the moment that the step's time runs out, the condition check should execute ONCE. Plus the above extra pause/resume functional need during each time counting period.

[WeisLeDocto]

So only at the moment that the step's time runs out, the condition check should execute ONCE.

Can the moment when a step ends be inferred from the acquired data ? If so, something like that would work:

class DataToIndex(crappy.modifier.Modifier):
    def __init__(self, min_delay_between_checks = 100):
        super().__init__()
        self.index = 0
        self.last_t = time.time()
        self.min_delay = min_t_between_checks
    def __call__(self, data):
        if <condition checking that step is over> and time.time() - self.last_t > self.min_delay:
            self.last_t = time.time()
            if <condition checking that step was successful>:
                self.index += 1
        data['next_step_id'] = self.index
        return data

Another question, is the intervention of the worker independent of the completion of burn-in steps ? Also, does it happen at times predictable enough to be entered in Crappy before starting a test ? Or should the pauses rather be interactively triggered ?

[occoder]

Another question, is the intervention of the worker independent of the completion of burn-in steps ? Also, does it happen at times predictable enough to be entered in Crappy before starting a test ? Or should the pauses rather be interactively triggered ?

Becuase the testing area is shared, the worker is not able to coordinate all the different tests beforehand. Such intervention is not predictable, or as you called "interactively triggered" or in more percise words "asynchronous to Crappy".

[WeisLeDocto]

In that case, if I understand correctly, you'll need a way to start a pause at any moment. That's what I mean with "interactive trigger", my English is not always on point. This could be achieved using a Button Block, that displays a basic UI with a button and increments a counter when clicked. The Timer Block would then pause the experiment for a given delay each time the button is clicked.

---

To summarize, you have a feature request for a basic "Timer" Block that pauses the Blocks' event loops so that the worker can enter the test area, and in addition to that, it is still a bit unclear how the rest of your script would look like for managing the burn-in steps, right ?

[occoder]

In that case, if I understand correctly, you'll need a way to start a pause at any moment. That's what I mean with "interactive trigger", my English is not always on point. This could be achieved using a Button Block, that displays a basic UI with a button and increments a counter when clicked. The Timer Block would then pause the experiment for a given delay each time the button is clicked.

This does not have to be implemented as a paused button, I prefer to leave the UI part to the end user. What about adding a pause and a resume methods to the Blocks? This does even more make sense, considering the Blocks already are able to response to a global stop notification. To summarize, you have a feature request for a basic "Timer" Block that pauses the Blocks' event loops so that the worker can enter the test area,

Correct. And it needs to remember the pause moment so that it can resume from where it was paused. and in addition to that, it is still a bit unclear how the rest of your script would look like for managing the burn-in steps, right ?

Each test step has a fixed time of period and relevant condition check at the end of the period. This also needs the Timer block 's help. As for the workflow that manages the test steps, it seems that Crappy's Generator is not the perfect fit. I'd like to make a custom output only IOBlock object which uses get_data to feed externally generated command (e.g. through a message queue) to the downstream blocks.

[WeisLeDocto]

This does not have to be implemented as a paused button

What I mean is that a button is already implemented in Crappy in a minimal UI, and could be used for driving a Pause Block.

What about adding a pause and a resume methods to the Blocks? This does even more make sense, considering the Blocks already are able to response to a global stop notification.

Precisely, they could be able to respond to such a notification given small adjustments in the code. I do see the interest of only pausing actuation Blocks, so that there's no gap in the data acquisition. My only concern is that end users would have to put extra care in the design of their logic/decision Blocks and layout, if these are not paused as well. Pausing Blocks individually would require some reserved label transmitting only stop and wake up messages. Or, with lower granularity, a per-Block flag indicating whether the Block should respond to a global pause. I have a preference for the latter implementation.

And it needs to remember the pause moment so that it can resume from where it was paused.

I'm not sure to understand this requirement. If a Block is on hold, its state is frozen and cannot change. It will always resume from where it was paused.

This also needs the Timer block 's help

To keep the code clean and generic, the "Timer", or rather "Pause" Block, should only be responsible for stopping and resuming other Blocks. It's rather the logic driving the Pause Block that should be shared to whatever object manages the steps. If this object is put on hold when a pause command is issued, it will anyway only operate once the pause ends.

I'm also still failing to understand how pauses in the experiment and step management are related, and why input from the Pause Block would be required to make decision on the next step.

As for the workflow that manages the test steps, it seems that Crappy's Generator is not the perfect fit.

Yeah I assumed so, given the information you already gave. Can you elaborate on what your requirements are, and how the current implementation fails to meet them ? I'm not fully satisfied with the current Generator implementation, so I'm curious about use cases where it's a limitation.

[occoder]

Precisely, they could be able to respond to such a notification given small adjustments in the code. I do see the interest of only pausing actuation Blocks, so that there's no gap in the data acquisition. My only concern is that end users would have to put extra care in the design of their logic/decision Blocks and layout, if these are not paused as well. Pausing Blocks individually would require some reserved label transmitting only stop and wake up messages. Or, with lower granularity, a per-Block flag indicating whether the Block should respond to a global pause. I have a preference for the latter implementation.

What about treating pause and resume at the same level of stop? This makes the whole concept straight and clear.

I'm not sure to understand this requirement. If a Block is on hold, its state is frozen and cannot change. It will always resume from where it was paused.

For example, current step needs to last for 5 minutes. At the end of 2nd minute, it is paused. Then the step needs to remember there still remains 3 minutes to go when it is resumed. Of course, there likely are other accompanied states, such as growing counters, that also need to be stored when paused and restored when resumed.

To keep the code clean and generic, the "Timer", or rather "Pause" Block, should only be responsible for stopping and resuming other Blocks. It's rather the logic driving the Pause Block that should be shared to whatever object manages the steps. If this object is put on hold when a pause command is issued, it will anyway only operate once the pause ends. I'm also still failing to understand how pauses in the experiment and step management are related, and why input from the Pause Block would be required to make decision on the next step.

I think the fixed time counting without any pause/resume is the basic function of a Timer block. It'd take the burden of counting time off the test step managers which only sends the parameters of the step including the lasting time to the Timer block. The Timer block also feeds back to the test manager to let it know when it can proceed with the next step. And pause/resume has nothing to do with making decision on the next step. It's just an advanced function of a Timer block. Yeah I assumed so, given the information you already gave. Can you elaborate on what your requirements are, and how the current implementation fails to meet them ? I'm not fully satisfied with the current Generator implementation, so I'm curious about use cases where it's a limitation.

The key drawback of the Generator block is that it only supports the predefined path. Currently, if the test manager needs to change the path during the runtime, it has to stop the Crappy first before loading the new one. And the cost of restarting Crappy is quite high in terms of latency. Imagine that what if each test step is dynamically generated (e.g. random test, test set switching upon user request, extra test steps conditionally appending, etc) rather than a predefine set of test steps? Modifier may mend some, but in nature the Generator block had better support more Input features.

[WeisLeDocto]

What about treating pause and resume at the same level of stop?

Do you mean at the same level as Block.stop() ? I don't understand what you mean here, and how it translates to an actual design.

Then the step needs to remember there still remains 3 minutes to go when it is resumed.

I think the simplest option would be to override the Timer.run() method so that it counts the time spent paused. The same principle could be implemented in Generators. If the state of the hardware is preserved during a pause, then the actual time spent in that state will be impossible to predict anyway.

I think the fixed time counting without any pause/resume is the basic function of a Timer block.

I'm not sure that just counting time is worth the extra complexity of having a dedicated Block. It is really just a matter of calling time.time() and updating a counter. But this is really an implementation detail.

And pause/resume has nothing to do with making decision on the next step. It's just an advanced function of a Timer block.

I would still recommend implementing the pause and resume function in a separate Block, to enforce single responsibility principle.

in nature the Generator block had better support more Input features.

I see, as it is now it can only support a sequential Paths layout. A first improvement could be to support a decision tree instead, and to take decision on the next Path dynamically at runtime. The most challenging aspect would then be to find an elegant and simple way for users to input this decision tree in Crappy.

[occoder]

Do you mean at the same level as Block.stop() ? I don't understand what you mean here, and how it translates to an actual design.

Yes, because you can treat a pause as a special case of a stop and a resume as of a special case of a start. The concept behind it is natural and smooth, i.e. low mental burden to the end users. I think the simplest option would be to override the Timer.run() method so that it counts the time spent paused. The same principle could be implemented in Generators. If the state of the hardware is preserved during a pause, then the actual time spent in that state will be impossible to predict anyway.

This part confuses me that the time span of a pause should not be the focus here. Nobody cares how long the pause really takes. Instead, caring more about the pause not messing up the original time setting, simply 5 min = 2min + 3min. I'm not sure that just counting time is worth the extra complexity of having a dedicated Block. It is really just a matter of calling time.time() and updating a counter. But this is really an implementation detail.

IMHO, time should be treated as the first class citizen in Crappy or broadly speaking in any experiment setup. So a standard Timer block is worthwhile. I would still recommend implementing the pause and resume function in a separate Block, to enforce single responsibility principle.

Agree. Maybe it's not even necessary to create such a pause Block. Embedding the pause/resume capability to each appropriate block might be an option as well. I see, as it is now it can only support a sequential Paths layout. A first improvement could be to support a decision tree instead, and to take decision on the next Path dynamically at runtime. The most challenging aspect would then be to find an elegant and simple way for users to input this decision tree in Crappy.

The Generator is different from other Blocks as its input includes ones not only from the physical world (generated by the devices) but also from the logic world (generated by the program) and from the human world (generated by the UI). Therefore the interface of Generator's input is better designed to be easy to link to these three worlds. For enough versatility, I'd recommend to design orienting to a popular middleware, such as Redis. This would avoid the unnecessary complexity involved.

[WeisLeDocto]

you can treat a pause as a special case of a stop and a resume as of a special case of a start.

I strongly disagree here, initialization and de-initialization should only be performed once during an experiment. The state hardware was left in before the pause could be strongly affected by initialization or de-initalization.

Nobody cares how long the pause really takes.

That's highly dependent on the context. For instance, if an oven should stay 5min at 1000° but gets paused 5min, the sample inside will stay in total 10min at 1000° and that's what actually matters.

At the very least, a good pause implementation should allow users to decide what happens for hardware when a pause starts. And to decide whether a Timer Block should keep counting during a pausebor not.

So a standard Timer block is worthwhile.

How would it differ from a Generator using delay=... stop conditions ?

Therefore the interface of Generator's input is better designed to be easy to link to these three worlds

That's not so much of a concern, any type of input always end up as a key/value pair and is treated similarly by the Generator. The struggle rather is how to make the internal Generator's logic arbitrarily complex while still keeping things clear and manageable for users.

This would avoid the unnecessary complexity involved

Sounds to me a bit overkill to add external dependency just for solving a very specific and uncommon issue. A super-Generator relying on an external tool should clearly be a complete separate Block.

[occoder]

That's highly dependent on the context. For instance, if an oven should stay 5min at 1000° but gets paused 5min, the sample inside will stay in total 10min at 1000° and that's what actually matters.

It seems that we have very different understanding about what a pause is. In this specific case, if you can predefine a fixed time period of a "pause" state that you need dwell on, then that is NOT a pause. It should be deemed as a normal working state, although it appears to be a pause state as no working parameters changing within the current path or test step. It's fair to say that ONLY those frozen state that asynchronously happens and its lasting time is not predictable should be called a pause. How would it differ from a Generator using delay=... stop conditions ?

IMHO, the delay function should have been decoupled from Generator block in the first place. The delay=time condition looks alien compared to other static conditions. Time data is a generic type of data applicable to nearly every experiment, therefore it deserves a specific standard processing block. Sounds to me a bit overkill to add external dependency just for solving a very specific and uncommon issue. A super-Generator relying on an external tool should clearly be a complete separate Block.

A super-Generator is much needed to enable Crappy fitting into more scenarios, especially those experiments with runtime generated parameters.

[WeisLeDocto]

ONLY those frozen state that asynchronously happens and its lasting time is not predictable should be called a pause

We're on the same line here, I picked 5min as an example, and thought it was obvious that in practice this duration was a priori unknown. My point really was that measuring the duration of a pause can in some cases be relevant, especially if the state of the hardware remains frozen during the pause.

IMHO, the delay function should have been decoupled from Generator block in the first place.

Interesting, I had never thought about it this way. I agree that decoupling time counting, logic, and signal generation is in theory a good idea. Now, in practice, and in the current state of Crappy, a Timer Block would be extremely similar to a Generator with delay conditions. Its arguments would be the same as for a constant Path, and it would also output the same kind of signal. That's why I don't see a need to reinvent the wheel on this specific aspect, the functionality already exists in Crappy.

That being said, I'm actually +1 on a clear, separate management of time / logic / signals in Crappy. I think that's something that should be carefully thought over and will include much larger changes than just removing time counting from the Generator Block. A Timer Block would likely be part of this new architecture.

---

To summarize a bit:

• What I can do on the short-term regarding your specific issue is to implement a pause mechanism at the Block level.
• A Timer Block, although easy to implement, would duplicate a functionality the Generator currently offers. It would be relevant to implement in the context of a complete refactoring of Crappy's signal management, which is unlikely to happen on the short-term.

[occoder]

To summarize a bit:

• What I can do on the short-term regarding your specific issue is to implement a pause mechanism at the Block level.
• A Timer Block, although easy to implement, would duplicate a functionality the Generator currently offers. It would be relevant to implement in the context of a complete refactoring of Crappy's signal management, which is unlikely to happen on the short-term.

It's great to see that we've landed on a common ground. Implementing a full-fledged Timer block might be a challenge, especially in a non-realtime OS, like Linux. More difficult, if not impossible, in a Win environment. So best wishes and may this project prosper.