proposed modifications to flagversion logic

[o-smirnov]

At the moment, #712 and #609 are not properly fixed, #711 is unclear, and my all-conquering madmax-based strategy can't be implemented easily. I propose a single solution for all these niggles:

1. Any worker that changes the state of flags should use flagmanager to save a before_workername flagversion as the first thing it does, and an after_workername flagversion as the last thing it does.

2. The -swstart_worker option should do a flag rollback operation by default. The purpose of this is to restore the flags to the state they were in upon entry to start_worker. The pipeline needs to look at available flagversions to determine what this is: it's either (a) before_start_worker, if this exists, or otherwise (b) the nearest after_worker flagversion in the worker chain preceding the specified start_worker.

3. (In case of (2a), there's no need to re-write before_workername, since we've just loaded it.)

4. There should be an optional load_flags section in a worker config stanza, which would explicitly load a specified flagversion before commencing with the rest of the worker:

  load_flags:
    version: str             # name of flag version to restore
    merge: true|false    # restore version entirely, or merge into current flags
    optional: true|false  # if specified version does not exist and optional=false, exit with error

The point of (4) is twofold. Firstly, I can implement my madmax flagging strategy, by inserting this into the selfcal worker:

  load_flags:
    version: before_flagging__2           
    merge: false
    optional: false

Secondly, any fiddly manual flagging for a given MS (as discussed in #711) can be absorbed into the pipeline by (1) the user doing the flagging by hand, (2) the user saving it to a flagversion called manual_flagging, and (3) us inserting this into the flagging section:

  load_flags:
    version: manual_flagging
    merge: true
    optional: true

[o-smirnov]

@SpheMakh, how's this progressing? Need need need.

[SpheMakh]

I forgot about this.

On it, again.

[o-smirnov]

OK, so discussing startup flag options:

(1) "auto" (default):

• if a "legacy" flagversion exists, restore it and run
• if it doesn't exist, save current FLAGs to "legacy" and run

(2) "load [versionname]"

• load a given flagversion (legacy by default) and proceed. Error if no such version.

(3) "save":

• save FLAGs to "legacy" and run

[o-smirnov]

P.S. This is only when pipeline is run from the start. If -sw is in effect, refer to https://github.com/ska-sa/meerkathi/issues/760#issue-547500449.

[paoloserra]

As mentioned in the now-closed #609 , I would be in favour of adding a use_flag_manager option (default True) such that one can always experiment with the more stupid approach of piling flags on top of each other independent of how the existing ones got there.

[SpheMakh]

Hehe, as usual, I failed to account for the Maverick user! On it

[paoloserra]

closed by #962