Add a way to quantify plan progress

[stan-dot]

As a user I would like to see the progress of the plan I defined, be it in points scanned out of total number, or some other metric.

[callumforrester]

ProgressEvents provide a path towards this, but they are currently per Status object i.e. per motor-move, detector trigger etc.

Obviously there is no universal metric for the progress of a plan, it depends on what the plan is trying to accomplish. Some plans cannot quantify their progress very well at all. For example an adaptive scan may have no idea of its total number of points upfront.

For this reason I think it is best to have an optional way to include a self-reporting mechanism in a plan itself, maybe something similar to TQDM. For example:

def my_scan(points: Spec[Movable]) -> MsgGenerator:
    motors = points.axes()
    yield from bps.mv(motors, starting_point)
    for point in plan_progress(points.frames()):
        yield from bps.mv(motors)

plan_progress would insert a side effect updating a status that the RunEngine knows about.

It is then the responsibility of a plan author to define how to report progress. How does that sound to you @stan-dot?

Also, would this then become an issue on bluesky itself rather than blueapi?

[stan-dot]

more than half of plans might have some metric that goes up, and for those that have an uncertain length the denominator of the progress bar could be just the previous run - and it would go beyond 100%.

and TQDM seems most suited to a CLI use case.

I don't think leaving this up to the plan author is the best idea.

and still in the context of the RunEngine and a web UI the issue for blueapi remains to report it.

unless it's again like with the data streaming case before, the question whether blueapi is one way street for data or not.

[callumforrester]

I don't think leaving this up to the plan author is the best idea.

How would you propose we do it then?

[stan-dot]

to have a default, overridable way which for all scans in the form start stop step can be just the number of points. Given that that is the most run functionality currently, adding a plan progress bar just for this simple use case will make a big difference to the end user

[stan-dot]

the length of the plan would be estimated from the params.

[callumforrester]

So you want to inspect the params for start, stop and num, say? But then how/where does the default code that detects what num is and how much progress the scan has made towards stop actually go? Inside the RunEngine?

[stan-dot]

that's an implementation thing, the issue is about a user need.

One implementation could be handling the start an stop in the client side and listen for 'step' events, and increment count

[callumforrester]

This discussion is focused on implementation, as the requirements have already been agreed upon. I believe this calculation should be handled server-side, without relying on a default that may or may not suit the specific plan. Implementing it as suggested could lead to the following issues:

• It must be reimplemented many times, once per client
• It has no way to know if it misses a step event and gets out of sync
• It will behave in a very confusing way for adaptive scans, while I agree we should cover the 80% case, let's leave the 20% case in a neutral state rather than make it worse
• It thickens the client, we want a thinner client than GDA with as much business logic on the server side as possible.

[stan-dot]

of course the adaptive scans won't have this

re: missing a step - I thought that the RMQ message bus has a delivery guarantee.

But then how/where does the default code that detects what num is and how much progress the scan has made towards stop actually go? Inside the RunEngine?

I guess the server-side implementation would likely use some callback logic, but I might not be the best person to dream up the details here

[callumforrester]

Fair enough, if we agree on server-side I think the decision I want out of this discussion is whether the implementation lives in blueapi or actual core bluesky (in which case I will close this issue and raise one there).

I think @coretl will advocate for the latter since he is keen on the workflow of developing plans outside of blueapi and running them for users inside it.

[coretl]

Depends on implementation.

One idea is to use the existing document streams to work out progress.

• Run start metadata tells you the scan shape, and we could add something to say which stream it refers to
• Each Event document or StreamDatum tells you the last emitted seq_num for a given stream We would then get an opt-in mechanism for plan progress without needing to modify plans apart from run start metadata.

If we go with this then I don't mind where the implementation goes, as I don't care too much about overall scan progress in iPython, only the progress of the current complete operation, which we already have implemented.

[stan-dot]

@coretl 's description is a more verbose version of what I had in mind when writing

One implementation could be handling the start an stop in the client side and listen for 'step' events, and increment count

@callumforrester

This aligns with the websockets stuff, and the flow of information the other way, or with the bluesky-stomp library as used in blueapi. we could add an endpoint

def get_plan_progress(job_id: string): Tuple[number, number]:

[DiamondJoseph]

StartDocument does not guarantee scan shape or length. I think offering "progress reports if you provide scan shape information" is a reasonable give-and-take, and it's useful for analysis and reshaping down the line anyway.

But if scan shape has any unknown axes, we can't report on the progress: [-1, 5, 4] (20 point inner scan an unknown number of times) is unknowable and so is [20, -1] (20 repetitions of an unknown number of points, as we can't tie seq_num to outer scan progress).

If we're relying on the scan shape to figure out scan progress something something data reshaping service something something: maybe the progress in the UI should come from the data API, rather than adding more to the scan engine?

[graeme-winter]

Rather than trying to work out a general way to measure progress: the author of the plan should be able to publish the relative progress even if it is not linear, so could we provide a method (like tqdm) which allows the plan author to provide updates from time to time, and a general API which can be used to transmit these to the end user? Similar idea to logging.

There are other progress bar implementations which allow periodic updates, which could also be used as a model. This does not need to be perfect, and for adaptive scans it could also reasonably go retrograde if the plan suddenly decides that it wants to re-scan a region at a higher resolution.

I assume that this is really a courtesy to the end user to see how long it will probably be until this scan is complete right?