Improve reporting of progress during a NAME run

[agstephens]

First stage

• [ ] Initially, NAME should update as follows:
  • 10% done (after init file);
  • 90% done after NAME has run (before plotting)
  • 100% when complete

More sophisticated progress reporting (later)

• [ ] We can use this file for progress monitoring:

$ cat Numbers_C1.txt
                        ,        % of run completed,        Count of particles,
    01/01/2022 01:00 UTC,                  8.333333,                         0,
    01/01/2022 02:00 UTC,                  16.66667,                         0,

    01/01/2022 06:00 UTC,                        50,                         0,
    01/01/2022 07:00 UTC,                  58.33333,                         0,

We can parse out the "% of run completed" field and provide that to the user - noting that we assume the actual running of NAME will be 80% of the total time taken (with 10% for initialisation of files etc and 10% for plotting).

[agstephens]

How to update the status in a WPS process? See here:

https://github.com/cedadev/goldfinch/blob/main/goldfinch/processes/wps_extract_uk_station_data.py#L116

[alaniwi]

Done this including the more sophisticated parsing of the model output file. (Not yet merged to main.)

See: https://github.com/cedadev/swallow/commit/2236325e77bc60e7447868fda8b36b32c2ce3295

https://github.com/cedadev/swallow/commit/717e1a9e2e7653b4ffbe5e575c4c7ff371cc0e33

Adding a statement to write to stderr for debug purposes (see line 313 of _name_base_process.py in the above commit), I see the following in a 24-hour test run:

UPDATE STATUS: Job is now running 0
UPDATE STATUS: Input file for Model model created 10
UPDATE STATUS: NAME model running 10
UPDATE STATUS: NAME model running 20
UPDATE STATUS: NAME model running 30
UPDATE STATUS: NAME model running 40
UPDATE STATUS: NAME model running 50
UPDATE STATUS: NAME model running 60
UPDATE STATUS: NAME model running 63
UPDATE STATUS: NAME model running 70
UPDATE STATUS: NAME model running 80
UPDATE STATUS: NAME model running 87
UPDATE STATUS: NAME model running 90
UPDATE STATUS: NAME model finished 90
UPDATE STATUS: Model code completed - starting plotting 90
UPDATE STATUS: Plots completed 100

So it is getting the numbers correctly. However, in the display on the staging server web UI, the status update is not seen until the end, so something is wrong with the actual call to update_status, or a limitation with the staging server environment. Hoping that this will be straightforward to fix.

[alaniwi]

Closing this issue as the print statements clearly show that it is doing the correct calls, and it is agreed that it ought to work with the real UI.