2023 Cylc Meetings

[hjoliver]

Following on from https://github.com/cylc/cylc-admin/issues/143

10 January

Recent activity

Attended:

• TP
• MH
• RD
• DM
• HO
• OS

Discussion Topics

cylc-flow 8.1.0: 96 closed issues, 7 open

• release end of week?
• cylc vro nearly done

cylc-uiserver 1.1.1 and 1.2.0 both in progress

• go straight to 1.2.0 but make sure the 1.x bug fixes are merged to master (include J-Hub 3.0 bug workaround)

cylc-ui: 1.4.0: 62 closed, 3 open

• release soon with 8.1.0
• job log view (MH) very close to being done

Master branch protection

• HO to do: merge via PR only, no force-push or deletion
• (Met O has daily back-ups just in case)

OS to present a webinar to IS-ENES3 in mid March. Should end up on you tube.

UK team to start work on rose edit and rosie soon, less time on Cylc

And will start pushing all users on to Cylc 8 very soon; lots of support likely needed.

HO to report on DS and AC hours left on UMP contract etc.

[hjoliver]

16/17 March

All present.

Bye Mel! 💐 🎖️

Discussion Topics

• Prioritize bug fixes and current PRs.
  • HO: 13 scheduler PRs up, need to finish to allow full user intervention powers
  • (this includes what to do about expire triggers, cylc set proposal)
• Quick update on workflow analysis view progress.
  • Not quite ready for review yet, but we can look a the approach being taken
• Making it easier to re-run a workflow:
  • Automatically specify start-tasks when re-running workflows (cylc play --start-tasks='*'?).
  • Reduce number of commands required? cylc play <id>; cylc trigger --flow=<flow> <id>//...; cylc play <id>.
  • Also, should we be able to warm-restart a workflow?
  • agreed useful; should be easy to implement; must be a new flow so we don't lose provenance
• share/lib and share/bin directories?
  • agreed; avoids the nuke-on-reinstall problem
• Should the expired output be included in the compulsory/optional output system?
  • needs further thought; it should work within the optional outputs system, but needs further thought because: success may not be optional IF the task does not expire
• How best to handle large log files in the UI?
  • needs further thought. Ideas include read number of lines with wc -l on the back, first N lines, last N lines, paginate, "fetchmore?" (MH) ... is this it?:* https://www.apollographql.com/docs/react/pagination/core-api/

If possible, agree cylc set proposal: https://cylc.github.io/cylc-admin/proposal-cylc-set.html

• (Not found the time yet).
• (Possibly needs updating now to not use the term "expire" for forcibly telling the scheduler to forget incomplete tasks)

[hjoliver]

12 June

Updates

HO:

• flood-forecasting sub-sub-workflows
  • motivations include separately developed subsystems, as well as dynamic graph structure
  • we need Cylc 9 already! 😱
  • That's probably several years off, without additional resources, so let's not promise anything yet
• DR
  • workflow-driven (in-workflow, or via watcher-worfkow)
    • disk-failover vs full DR
    • full DR: next restart enhancements (we have start-tasks with pre-initial-ignore, but, also triggering, cylc set etc., and clock-triggered tasks)
    • gentle failure (when possible) via workflow migration?
    • DM and HO to meet and discuss how Met Office does it, at some point soon
• BoM priorities:
  • DR stuff (above)
  • cross-user (with or without hub admin)
  • UIS remote spawn
  • Hub with pseudo-spawner (plus remote spawn)
  • Platform "cross product" fix
  • Monitoring ~10 oper accounts
  • Note NIWA and MO not using J-Hub in anger yet, for Cylc; will do in next year
  • Cylc UI and UIS can handle multi-user subscriptions, but we need to decide how to request that and how gscan should display it

Discussion Topics

General:

• getting the current ~30 PRs finished off
• (some previous meeting topics still in the wild - above)
• Another Cylc 7 final release is needed [DONE]
• Cylc 8.2.0 release before next meeting ~17 July; need restart-finished-flow at the least
• HO to shift 8.1.x issue to 8.2.0; we'll need to do some triage and bump a bunch of issues back
• TP offered to help HO with tests and reviews for current PRs

[wxtim]

17 Jul

---

Updates

HO

TP

• 🐛 🐛 🐛 👞 (Squashing bugs).
• Add cylc play -z=f,g,h option.
• Reviewing
• Failing to add tests for HO's PR's. Sorry. :(

OS

RD

• Mainly UI stuff

Discussion Topics

• Proposals for interventions https://github.com/cylc/cylc-admin/issues/172 ?
• Releases?
• Discussion Topics From Last time
• Upcoming Leave in UK Summer?

---

Notes

Present HO, DM, RD, DS, TP

Priorities

8.2

Release blockers?

• Can HO hand over incomplete PRs?
• OS to highlight PRs in question for HO.
• HO to bump bumpable items.

UIS JupyterHub 2.0

Need to have a release shortly after JupyterHub 2.0 upgrade branch merged. OS says it's nearly ready, but that careful review required. Documentation done.

8.3

• Intervention proposals
• Collapsing GScan

Discussions

5085 - Don't strip whitespace in Parsec

OS - will break MO workflows. HO - Agreed to close.

5237 - Workflow state command Cylc 7 ←→ Cylc 8 compat

HO Blocker at NIWA, why not reviewed OS Still marked draft

Clock Trigger Dependencies

Issue number @dwsutherland ?

OS If it's just clock triggers depending on earlier clock triggers it's ok.

Data Workflows

OS - Further work on the analysis view is ongoing.

Rose Stem

OS - Is starting to become a priority and may involve adding some requirements to Cylc which seem "odd" without reference to rose stem.

[wxtim]

11 September

---

Updates

[!NOTE] UI Server 1.3.1 Released last Friday, with UI 2.1.0.

HO

• cylc set command (almost done)
• cylc hold flow-specific (done, needs one more review)
• optional outputs (trying to resolve what I see as problem with the extended functionality)

TP

• Skip Mode Precursors
  • Testing Sim Mode (found some 🐛 )
  • Making Sim Mode changeable by broadcast.
  • Holiday.

OS

RD

MD

DS

• Graph no-rewalk and reloaded flag PRs ready to go/be-reviewed
• Moving onto the named clock/trigger sole prerequisite sequential spawning.

Discussion Topics

Super-high priorities for 8.3.0:

• implement intervention proposals, document
• avoid graph rewalk (done, needs review?)

BOM priorities:

• "workflow-specific" run-dir symlinking (agreed on the rose-like approach?)
• (needs custom environment variables passed to run-hosts)
• UIS log location!

After that, triage the rest and get what we can in.

UI:

• BUG completed tasks never removed if graph if graph - or table? - is default view
• new Gantt view? (HO tried to test...)

7.8 release

• need cylc suite-state forward compat (the sibling PR is merged) - testing Cylc 7??

Notes

Present

Priorities

• https://github.com/cylc/cylc-flow/milestones/cylc-8.2.2
• https://github.com/cylc/cylc-flow/milestones/cylc-8.3.0

[hjoliver]

30/31 October 🎃

Updates

• 8.2.2 released 🎉
• FYI: Status page now lists issue counts (auto-generated)
• Workflow auto-collapsing in the workflows view now implemented.

HO

• cylc set command (almost done - again 😬 )
• (Singapore workshop)
• (BOM training and demo session)

TP

• Working out what simulation mode is actually doing, and testing it: Notably the evaluation of simulation mode needs to happen at runtime, not on config parsing, else we cannot broadcast to it or skip mode. There are still some outstanding discussions about what skip mode should do with optional outputs.
• Helped a colleague (Joe Hickson) deliver a bout of Cylc training. (So did Mark, Oliver and Ronnie)
• Stopped PYTHONPATH affecting Cylc. If you want it use CYLC_PYTHONPATH
• Fixed a mod large pile of bugs.

OS

• Clearing review backlog.
• Looking into making "rose stem" a Cylc plugin (removing rose stem wrapper, unlocking other Cylc commands).

RD

• Lots of small things + lots of reviewing
• Will start work on cylc remove soon - https://github.com/cylc/cylc-flow/issues/5643

MD

DS

Discussion Topics

High priorities for 8.3.0:

• implement intervention proposals, document
• avoid graph rewalk (done, needs review?) - Note this is 8.2.x
• Tui 1.0

BOM priorities:

• Hub cross-user demo?
• "workflow-specific" run-dir symlinking (agreed on the rose-like approach?)
• (needs custom environment variables passed to run-hosts) - MERGED
• UIS log location?

cylc rewind?

(if there's time)

https://github.com/cylc/cylc-flow/issues/5416

# it works but it ain't snappy, ideas?
cylc set <cycle> --pre=all [--flow=new]

After that, triage the rest and get what we can in.

7.8 release

• need cylc suite-state forward compat (the sibling PR is merged) - testing Cylc 7??

Notes

Present

Priorities

• https://github.com/cylc/cylc-flow/milestones/cylc-8.2.3
• https://github.com/cylc/cylc-flow/milestones/cylc-8.3.0

[oliver-sanders]

This is now closed, the conversation continues in https://github.com/cylc/cylc-admin/issues/188