Use ISO standard date-time formats

[cylc]

This is a completely general notation for specifying date/times and time ranges. It would presumably allow us to replace all the current cycling modules (daily, monthly, etc.) with a single one that could handle any cycle time sequence. And cycling for paleo-climate simulations?

MPI is developing an ISO time library for Python.

[cylc]

Also: http://pypi.python.org/pypi/isodate

[cylc]

http://en.wikipedia.org/wiki/ISO_8601

[matthewrmshin]

Would this eventual change be quite disruptive? Perhaps it would be desirable for cylc to write cycle time in the YYYYmmddTHH format sooner rather than later?

[hjoliver]

Matt, unfortunately I haven't had a chance to think about this yet, or even read the ISO standard references. Is the notation you've suggested sufficiently general to handle everything we could conceivably want to do with cylc - including cycling for paleo-climate simulations (how are pre-historic dates represented? And, we would need to include minutes too for future NWP suites.

Luis @m214089 - do you have any advice on this: i.e. can we quickly convert to a general cycle time notation that will make the future transition to ISO date-time representation easy on users?

[m214089]

Hi,

we currently change our time control in the models to an ISO based scheme. By the way a date looks like Matt wrote as one possible way. Recurrences/event are denoted RyyyyyYmmMddDT...... where everything not used could be missing.

I have a shorter than the standard description of the whole thing, but I can provide it to you only after the 15th, because I'm leaving for vacation later today ...

If that's fine with you just answer, and I'll provide you with the information by then.

Cheerio, Luis

[matthewrmshin]

I think the ISO 8601 format should be sufficient to handle every date time representation we'll need, as it is designed to do so. If it does not do so, we can put the blame on the standard format ;-) In particular, historical dates and 9999+ years can both be represented.

(A second thought. I guess this change does not have to be disruptive at all. When the ISO functionality is introduced in Cylc, we can provide a classic mode where YYYYmmddHH will be used, and an ISO mode where the (basic) ISO 8601 format will be used.)

[m214089]

Hi Matt, you're right. I do have some regex for parsing ISO 8601:2004, but can supply them only after the 15th. I hope that is still fine. Luis

[hjoliver]

Luis, that's fine - have a great vacation!

On 7/10/2012, at 7:51, m214089 notifications@github.com wrote:

Hi Matt, you're right. I do have some regex for parsing ISO 8601:2004, but can supply them only after the 15th. I hope that is still fine. Luis

— Reply to this email directly or view it on GitHub.

[hjoliver]

@matthewrmshin, I know the ISO standard is meant to cover all bases, I was just asking if the notation you suggested (YYYYmmddTHH) is general (i.e. can that particular notation express all that the standard covers?) It's not obvious that addition of a "T" between date and time helps any (how does that handle pre-historic dates?). Anyhow, that's just to explain better what I was asking above - I should read the standard myself, of course!

I agree this need not be a disruptive change, but at the same time if we could easily change to a new ISO-compatible cycle time notation before promised backward compatibility kicks in, maybe we could avoid having to support a distinct "classic mode" in the future.

[matthewrmshin]

@hjoliver The basic notation of the date would match this Pythonic regular expression (?P<year>-?\d*\d\d\d\d)(?P<month>\d\d)(?P<day?\d\d). All years will be at least 4 digits, left padded with zeros. E.g. 1BC is represented as +0000, 2BC is -0001, etc. The T helps because without it you are only dealing with a date. With it you are dealing with a date and time. (Hence, a digit-only string is always a date.)

[m214089]

The original standard can be found at:

http://dotat.at/tmp/ISO_8601-2004_E.pdf

Useful ISO 6801 features from my point of view:

Designators

P is time-interval designator
R is recurring time-interval designator
T indicates start of Time elements
Z is UTC designator

Duration designators

Y, M, D, H, M, S may be used
M can be minute, month, or both

Calendar date

year, month, day number within month

Maybe ordinal date

 year, day number within year

Representing dates

Date Basic Format Extended Format Calendar date YYYYMMDD YYYY-MM-DD 19850412 1985-04-12 Expanded ±YYYYYYMMDD ±YYYYYY-MM-DD ±0019850412 ±001985-04-12 Maybe ordinal date YYYYDDD YYYY-DDD 1985102 1985-102

Representing times

Time Basic Format Extended Format Local time hhmmss hh:mm:ss 232050 23:20:50 Milli seconds hhmmss.sss hh:mm:ss.sss 232050.500 23:20:50.500

For universal time (UT) a Z should be required at the end of the time string. Are timezones required? Combined date and time

T indicates start of time.

Calendar dates + UT time: YYYY-MM-DDThh:mmZ Maybe ordinal dates + UT time: YYYY-DDDThh:mmZ

Representing time-intervals

P indicates a duration (period). Intervals can come in four different varieties:

• start and end time: YYYY-MM-DDThh:mm:ss/YYYY-MM-DDThh:mm:ss
• duration: PnYnMnDTnHnMnS - NOTE: PnYnMnD and PTnHnMns are possible shortcuts. If n of a designator is 0,

it can be ommited, if it is trailing.

• start time and duration: YYYY-MM-DDThh:mm:ss/PnYnMnDTnHnMnS
• duration and end time: PnYnMnDTnHnMnS/YYYY-MM-DDThh:mm:ss

Recurring time-intervals

R is used to indicate recurrence. Recurring time intervals are expressed by:

• number of recurrences and start and end times: Rn/YYYY-MM-DDThh:mm:ss/YYYY-MM-DDThh:mm:ss
• number of recurrences and a duration: Rn/PnYnMnDTnHnMnS
• number of recurrences and start time and a duration: Rn/YYYY-MM-DDThh:mm:ss/PnYnMnDTnHnMnS
• number of recurrences and duration and end time: Rn/PnYnMnDTnHnMnS/YYYY-MM-DDThh:mm:ss

If number of recurrences is not provided, then the number is unbounded. For simplicity the implementation is only supporting the second expression for recurring time intervals.

Regex definitions/collection

Regex for date and time so far constructed:

^([\+-]?\d{4}(?!\d{2}\b))((-?)((0[1-9]|1[0-2])(\3([12]\d|0[1-9]|3[01]))?|W([0-4]\d|5[0-2])(-?[1-7])?|(00[1-9]|0[1-9]\d|[12]\d{2}|3([0-5]\d|6[1-6])))([T\s]((([01]\d|2[0-3])((:?)[0-5]\d)?|24\:?00)([\.,]\d+(?!:))?)?(\17[0-5]\d([\.,]\d+)?)?([zZ]|([\+-])([01]\d|2[0-3]):?([0-5]\d)?)?)?)?$

Regex for duration/time intervall so far constructed:

^P(?:\d+Y|Y)?(?:\d+M|M)?(?:\d+D|D)?(?:T(?:\d+H|H)?(?:\d+M|M)?(?:\d+(?:\.\d{1,2})?S|S)?)?$

or

^P(\d+Y)?(\d+M)?(\d+D)?(T(((\d+H)(\d+M)?(\d+(\.\d{1,2})?S)?)|((\d+M)(\d+(\.\d{1,2})?S)?)|((\d+(\.\d{1,2})?S))))?$

Not sure which one to use yet. The regex are for the full standard including business week, which we do not need. This needs to be cleaned up.

As well I have a set of routines in C for calculating the Julian date to proleptic Gregorian in a pure integer implementation including negative Julian days. The same for pure 360 day years and 365 day years. We would need to convert those to Python. Unfortunately none of the Python date utils is sufficient clean to be used.

I would be happy if you could have a look into the regex. They are not trivial ...

[benfitzpatrick]

We've riffed on this a bit here - please have a look:

https://github.com/cylc/cylc/wiki/ISO-8601

[hjoliver]

Nice, well done!! I'll print it out and read it on the plane tomorrow.

[hjoliver]

[This discussion temporarily went to #887]

[hjoliver]

Some progress:

[hjoliver]

10-minute cycling #68:

[benfitzpatrick]

Very cool

[benfitzpatrick]

Could we use the no-seconds, no-delimiters kind of dump format for the date/times?

[hjoliver]

Could we use the no-seconds, no-delimiters ...

Yes. What I've done so far is pretty rough, there will no doubt be some detail to attend to...

[hjoliver]

Update: integer and ISO cycling prototyped in hjoliver:119.iso8601-cycling. [EDIT: now in cylc/cylc - see below] Email sent to interested parties. Basic cycling - disregarding proper use of context points - is working, which shows the guts of cylc is handling all cycle calculations and comparisons properly. It may be quite easy to finish this off. Some things still to do:

• restore memoization of cycle time comparisons
• check use of isodatetime and cylc/time_parser.py with Ben - so far I've used it in an ad hoc way without spending the time to understand it properly - some usage examples would help - (the new cylc interface to isodatetime, lib/cylc/cycling/iso8601.py is quite minimal, however, and hopefully will not be hard to knock into shape).
• (related to the above point) use of context points, e.g. [[[T06]]] should be taken in the context of the suite start time, or else an explicit context used for tasks that start cycling later on.
• consider suite shutdown at final cycle time. Current cylc does not have the concept of a finite cycling sequence, instead we just compare each new point with the suite final time. In the isodatetime library, on the other hand, a recurrence will return None if the next point is out of bounds. Once cylc internals handles this properly it will probably make finite-range and shutdown tasks easy, but I need to consider the impact on the current system of putting tasks on hold once they go beyond the suite final cycle.
• (minor): use of T to represent the current cycle time in graph and message offsets is to be dropped, e.g. foo[T-6H] will become just foo[-6H]
• I need a better way to load point, interval, and sequence classes from either the integer or iso8601 modules depending on what's needed by the suite definition. At the moment the choice is hardwired in lib/cylc/cycling/loader.py because modules using these classes are imported before we find out what the suite needs.

[benfitzpatrick]

I think this would be good as a feature branch in cylc/cylc

[matthewrmshin]

I think this would be good as a feature branch in cylc/cylc

:+1:

[hjoliver]

Pushed to cylc/119-iso8601-cycling.

[hjoliver]

Somewhat remarkably, the advanced scheduling that I was worried about (shutdown and finite-range tasks, etc.) has turned out to require very little change to cylc internals.

Here's dev/suites/integer/two/suite.rc:

[scheduling]
    initial cycle time = 1 # T0
    final cycle time = 9
    [[dependencies]]
        # once at T0 (async tasks now shorthand for R1/c0/P1)
        graph = "start1 => foo"

        [[[ R1/c0/P? ]]]
            # once at T0
            graph = "start2 => foo"
        [[[ R/c0/P2 ]]]
            # repeat with step 2, from T0
            graph = "foo[T-2] => foo"
        [[[ R/c2/P2 ]]]
            # repeat with step 2 from T0 + 2
            graph = "foo => bar"
        [[[ R2/3/7 ]]] # or [[[ R2/3/P4 ]]]
            # repeat twice, at 3 and 7
            graph = "foo => qux"
        [[[ R1/7/7 ]]]
            # once at 7
            graph = "qux => egg"
        [[[ R1/9/P2 ]]]
            # once at 9
            graph = "bar => stop"

This has start-up, mid-run, and shutdown one-off tasks, and finite-range tasks. It graphs and runs correctly.

[hjoliver]

@benfitzpatrick - you could now look at implementing lib/cylc/cycling/iso8601.py by analogy with lib/cylc/cycling/integer.py.

[hjoliver]

We'll need to play with a lot of different suite graphs to make sure everything works. E.g. I haven't tested any conditional triggers yet.

Some other things to think about:

start-up and asynchronous one-off special tasks

I have removed these. They were a nasty special case in the code, and one-off tasks can be handled nicely and more generally in the new framework as simply single-repeat recurrence expressions. This:

    [[dependencies]]
        graph = "foo"

is currently just short for this:

    [[dependencies]]
        [[[ R1/c0/P? ]]] 
            graph = "foo"

(and we can do similarly for ISO8601 cycling).

These are just normal cycling tasks that happen to only run once, and as such they get removed from the suite when spent. Do we still need the original asynchronous tasks that are never removed? (does Met Office rely on this)? If so, maybe we can just mark tasks that should not be removed when spent.

notation for referencing the final cycle time

We might need this (similar to taking recurrence points in context with the initial cycle time) to allow tasks in the final cycle(s) without hardwiring absolute values.

backward compatibility

This will be a lot of effort (or maybe not ... need to think about it). Can we just make a clean break at cylc-6?

test battery

Upgrading this might take a bit of work.

user guide

ditto.

[benfitzpatrick]

notation for referencing the final cycle time

I think this is covered by our use of the fourth ISO recurrence definition (the following is from the ISO wiki page):

    initial cycle time = 20130325
    final cycle time   = 20130404T12
    [[dependencies]]

....

        [[[ R3/P1D/T06 ]]]        # = R3/P1D/20130405T06
            graph = "a12 => b12"  # Using the fourth ISO recurrence definition, repeat three times counting 
                                  # backwards 1 day from 06:00 immediately following (or including) the
                                  # final cycle time.
                                  # This means (post final cycle times do not run) run at:
                                  # 20130403T06
                                  # 20130404T06
                                  # This avoids hard-coding the cycle time.

        [[[ R1/P1W ]]]            # = R1/P???/20130404T12
            graph = "a13 => b13"  # Using the fourth ISO recurrence definition, repeat once at the
                                  # final cycle time (period is meaningless for R1 and could be anything).

[hjoliver]

I think this is covered by our use of the fourth

You're right, and in fact I had already put "the fourth recurrence" format in for integer cycling. I've amended the end of dev/suites/integer/two/suite.rc with this, and it works:

        [[[ R1/P?/c0 ]]]
            # once at final cycle
            graph = "bar => final"
        [[[ R2/P2/c0 ]]]
            # final two cycles
            graph = "bar => stopping"

[arjclark]

start-up and asynchronous one-off special tasks

I have removed these. They were a nasty special case in the code, and one-off tasks can be handled nicely and more generally in the new framework as simply single-repeat recurrence expressions. This:

[[dependencies]] graph = "foo"

is currently just short for this:

[[dependencies]] [[[ R1/c0/P? ]]] graph = "foo"

(and we can do similarly for ISO8601 cycling).

These are just normal cycling tasks that happen to only run once, and as such they get removed from the suite when spent. Do we still need the original asynchronous tasks that are never removed? (does Met Office rely on this)? If so, maybe we can just mark tasks that should not be removed when spent.

@hjoliver - we had an email conversation back in September ("Persistence of startup tasks") in which we discussed solutions to the following type of graphing problem:

[[special tasks]]
start-up = start
clock-triggered = foo, bar
...
[[0]]
    graph = start => foo
[[12]]
    graph = start => bar

where start is a startup task and the suite can be started up at any time.

The solution we arrived at was:

[dependencies]
graph = start
[[0]]
    graph = start => foo
[[12]]
    graph = start => bar

which allowed us to start the suite at any cycle time and dependencies on the start task without resorting to some automated graph generation. It also meant that, as the suite could be shutdown and restarted before something depending on start had run we didn't have any problems with the task clearing out the pool for dependencies purposes.

With your changes, I guess the required graphing following the wiki page on ISO8601 would have to be the final example case of:

        [[[ T04 ]]]
            graph = "a[] => b"              # b depends on a at the initial cycle time (special case)

So, for the suite above:

[[ T00 ]]
    graph = start[] => foo
[[ T04 ]]
    graph = start[] => bar

should have the same effect? I guess the start task will hang around forever as there are still things depending on it so it is deemed needed to be kept for future cycles? If it isn't kept then yes, we'd need a persist option for tasks that shouldn't be removed from the pool.

[arjclark]

Discussing with Dave, I think we're going to need to carefully check the logic for when things can leave/enter the task pool is behaving and note down any improvements that may need making down the line.

For example, if I specify a bunch of tasks to run in the final cycle (with some other dependencies on T-X) then they will sit in the task pool until that cycle is reached. For a large number of tasks this could have significant performance impacts. At the very least it'd be irritating having them knocking around the gui when running a 6-month trial. It's also not wonderfully clear to me how they'd interact with the runahead limit?

[hjoliver]

@arjclark - 2nd point 1st: shutdown tasks (and initial instances of other delayed-start tasks) knocking around in the pool for ever. This not a problem for the runahead limit, but it will be annoying and potentially could affect performance as you say. My idea to handle it is this:

1. at start-up, load all initial task instances into a temporary list instead of directly into the pool.
2. iterate through the list and determine the initial runahead limit.
3. now load tasks < runahead into the pool, and tasks > runahead limit into a holding list
4. in the main scheduling loop, release tasks from the holding list as the runahead limit catches up to them

Tasks in the holding list will not participate in dependency matching, and will not be visible in the GUI etc.

If no one sees a problem with this I'll implement it on the ISO8601 branch once Ben's changes are in. As I suggested earlier, I think we should proceed with pull requests to the feature branch for further changes - ok?

[hjoliver]

Thoughts on asynchronous tasks - we do need to be able to to trigger multiple cycling sequences off the same one-off initial (start-up or async) tasks. A problem with the status quo is that we put initial tasks in cycling sections rather than the other way around. This became clear to me toward the end of #434, and it makes even more sense in the context of ISO8601 recurrences. Consider (in the old notation):

[[special tasks]]
   start-up = start
[[dependencies]]
  [[[0]]]
      graph = start => foo

Here start has to be declared as special to prevent cylc inferring that it is a cycling task based on the cycling graph section it appears in. Instead, for the future we should do this:

[[dependencies]]
   [[[R1]]] # repeat once at initial cycle
      graph = start => foo
   [[[T0]]]
       graph = foo

Now if a task appears in a graph section, it belongs to that recurrence, so this says unambiguously without declaring start to be a special "start-up task" (as would be required if the trigger was in the cycling graph section), that start only runs once, at the initial cycle time, and also that the first instance of foo is at the initial cycle time. The initial trigger will work even if the initial cycle is not in the T0 recurrence, although I'm not sure why you'd want to do that (?). I think this should also work with offset sequences, which is currently ambiguous/broken and requires async tasks:

[[dependencies]]
   [[[R1]]] # repeat once at initial cycle
      graph = start => foo & bar
   [[[T0]]]
       graph = foo
   [[[T12]]]
       graph = bar

Although again, I'm not sure it makes much sense to have one or both of foo and bar start off-sequence... especially given that we will now have the ability to have multiple initial tasks with different cycle times ... so maybe this suggests that asynchronous tasks are not strictly needed anymore, but maybe we should still restore them because they are easier to understand in some cases, and because they make more sense for prep tasks that really don't have use for a cycle time (and for entirely non-cycling suites, e.g. for a sequence of build jobs - although that could be done with a one-off integer recurrence). Anyhow, I think the same argument goes for async tasks:

[[dependencies]]
       graph = start => foo & bar
   [[[T0]]]
       graph = foo
   [[[T12]]]
       graph = bar

In this case we can still handle the async graph section as a special kind of one-off recurrence, and if a task appears in both cycling and asynchronous graph sections cylc can infer that it's the first instance of the cycling task that's involved in the asynchronous trigger.

[arjclark]

@hjoliver - replacement of start-up tasks with R1 makes sense to me.

For:

[[dependencies]]
   [[[R1]]] # repeat once at initial cycle
      graph = start => foo & bar
   [[[T0]]]
       graph = foo
   [[[T12]]]
       graph = bar

having foo and bar run out of cycle isn't really an option for us (and just seems bad) particularly when you want them to be clock triggered.

re.

[[dependencies]]
       graph = start => foo & bar
   [[[T0]]]
       graph = foo
   [[[T12]]]
       graph = bar

I think having inference from graph = start => foo & bar combined with cycling is somewhat difficult to understand at a glance. Is there any reason you prefer that to this:

[[dependencies]]
[[[ T0 ]]]
    graph = start[] => foo
[[[ T04 ]]]
    graph = start[] => bar

as this explicitly says to trigger foo and bar off start at the initial cycle time?

@benfitzpatrick - any thoughts?

[benfitzpatrick]

I broadly agree with Andy's last example, but I would say:

[[dependencies]]
    [[[ R1 ]]]
        graph = start
    [[[ T00 ]]]
        graph = start[] => foo
    [[[ T04 ]]]
        graph = start[] => bar

as I think we should avoid inferred or implicit extra cyclers, which is after all the reason for removing all this special task behaviour.

Without the [[[ R1 ]]] part above, there should be a validation failure if the initial cycle time is not T00 or T04, and a warning even if it is.

I don't think there's any need for asynchronous tasks given this notation, which is exactly equivalent to the old way of doing it (remove [[[ R1 ]]] text, remove [] from start[]) but much clearer about where tasks belong.

I would modify:

Now if a task appears in a graph section, it belongs to that recurrence

to:

If a task with a relative offset (e.g. foo[-PT6H] or foo[T-6] in old money) appears in a graph section, it belongs to that recurrence

A task with an absolute cycle time defined in a graph section should not be the responsibility of that recurrence. This goes for e.g. foo[], bar[T06], baz[2014W451T0600].

[benfitzpatrick]

Hilary, where are you on this? Are you waiting for me to finish the tests and documentation?

[hjoliver]

Ben - no, but I am waiting for you to merge your work (other than tests and documentation) to the cylc/cylc iso8601-cycling branch so I can carry on from there. Sorry, I thought I'd made that clear-ish above:

... I'll implement it on the ISO8601 branch once Ben's changes are in...

(and the suggestion that we should proceed now by pull requests to the feature branch ... so I was waiting on your pull request - I was going to check with you today on what the delay was).

[hjoliver]

This is one discussion that might be easier around a table!

[[dependencies]]
    [[[ R1 ]]]
        graph = start
    [[[ T00 ]]]
        graph = start[] => foo
    [[[ T04 ]]]
        graph = start[] => bar

What I don't like about this is we still have special behaviour that is not obvious from the notation: the trigger start[] => foo is only valid for the first cycle of the recurrence, whereas one might reasonably assume it to apply throughout the recurrence. The fact that [] implies a specific single cycle does not help because the notation suggests (or at least it could do - it's ambiguous) "all foo trigger off the one start". In fact, we might sometimes want it to have that meaning: e.g. if foo does not depend on its own previous instance - so many instances of it can submit at once - but each instance does depend on setup done by start.

Why not drop the special behaviour and simply let the recurrence relations apply to all triggers defined under them:

[[dependencies]]
    [[[R1]]]
        graph = start
    [[[ R1/T0 ]]]
        graph = start[] => foo
    [[[R1/T04]]]
        graph = start[] => bar
    [[[ T00 ]]]
        graph = foo
    [[[ T04 ]]]
        graph =  bar

This is completely general and unambiguous isn't it? (and we can have multiple cycles trigger off a single cycle if we like)..

A task with an absolute cycle time defined in a graph section should not be the responsibility of that recurrence.

Hopefully it's obvious what I mean by now: if a task has an absolute cycle time attached, it's cycle time does not get replaced by those of the recurrence, but the trigger is valid throughout the recurrence: [[[T0]]] foo[abs] => bar says "bar triggers off foo[abs] for all [[[T0]]]" (the recurrence in question will mostly be an R1 in practice, but we shouldn't prevent more general use).

[hjoliver]

Andy's specific points:

having foo and bar run out of cycle isn't really an option for us

sorry, I was being lazy there - see my example immediately [edit: "immediately above"!] for the proper way to do this with foo and bar running on different cycles.

having inference from graph = start => foo & bar combined with cycling is somewhat difficult to understand at a glance.

again, I'm suggesting we think of recurrence relations as applying to triggers rather than tasks:

[[[R1]]]
    graph = start => foo & bar

Then it doesn't matter that foo and bar are "cycling tasks", we're just saying that at the time point of this R1, foo and bar trigger off start. Note also that the line between "start-up" and "cycling" tasks is now somewhat less clear: we can now have one-off tasks anywhere in the suite's evolution, or two-off, ... or n-off.

Finally, my approach is easier implement (not that that's the primary goal, but this does reflect the fact that things are simpler if recurrences apply to all triggers under them): all I need to do is trigger off the absolute cycle point, if present, instead of the new recurrence point. Whereas for the other approach, like current master, some unpleasant special casing is needed to record not just the recurrences for each task but also for which cycle points they can be used to define prerequisites.

[hjoliver]

Ben - no, but I am waiting for you to merge your work ...

@benfitzpatrick - I'm not trying to hurry you up, but I'm slightly concerned that you might now be waiting on me because you think you have updated the branch in cylc/cylc already, but haven't; or that you did and I inadvertently wiped out your commits with a forced push from my end (I don't think so, but stranger things have happened!)

[benfitzpatrick]

Sorry Hilary, it wasn't you - I've been off on this Friday and Monday, hence part of the delay. The rest of it is due to me spending most of last week implementing https://github.com/metomi/rose/pull/1202, https://github.com/metomi/isodatetime/pull/29, and https://github.com/metomi/isodatetime/pull/30. The last 2 are somewhat #119-related but not a lot. These were getting in the way of @arjclark's suite refactor work, and needed to be done soon. However they did slow you and me down on this, and I am sorry about that. I've also been fixing some bugs I found while testing the suites above (like this one: https://github.com/metomi/isodatetime/pull/32 which broke the weekly one I'd set up). Tomorrow (2014-04-08T08:00Z) I'll try to finish any outstanding #119-relevant bugs and push something up for a pull request so we can progress.

The documentation is taking a while to write (/phrase) and will have to wait!

[hjoliver]

No worries Ben, I just wanted to check. And yeah the documentation can definitely wait.

[benfitzpatrick]

Why not drop the special behaviour and simply let the recurrence relations apply to all triggers defined under them:

 [[dependencies]]
    [[[R1]]]
        graph = start
    [[[ R1/T0 ]]]
        graph = start[] => foo
    [[[R1/T04]]]
        graph = start[] => bar
    [[[ T00 ]]]
        graph = foo
    [[[ T04 ]]]
        graph =  bar

Agreed, this is the best one

[arjclark]

From @benfitzpatrick in #937:

Re: backwards compatibility, what would you think of auto-detecting output format ( e.g. 2013050401 old-format, 20130504T0100 new-format) based on the initial cycle time format - i.e.:

    if re.match(r"\d{10}$", initial_cycle_time):
        # use old format....
    else:
        # use new format...

That might remove the need for forwards and backwards switches, which would be nice, but it is a blunt instrument...

I don't think checking for a 10 digit number will be sufficient as cylc currently allows shortened initial and final cycle times such as 2013, implying 2013010100, and 201303, implying 2013030100 etc. See e.g. the tests for yearly and monthly in the periodicals tests.

It would probably be better to check for things such as "Yearly(XXXX,Y)", "Monthly(XXXX,Y)", "Daily(XXXX,Y)" and 1-2 digit integers between 0 and 23 to determine if auto backwards compatibility should be used.

[arjclark]

You could look at enforcing a particular cycletime type through a switch at the top of the suite.rc along the lines of:

[cylc]
    cycling mode = iso|classic

[benfitzpatrick]

You could look at enforcing a particular cycletime type through a switch at the top of the suite.rc along the lines of:

[cylc]
    cycling mode = iso|classic

I think it would be best to avoid this if we can...

I don't think checking for a 10 digit number will be sufficient as cylc currently allows shortened initial and final cycle times such as 2013

I had forgotten this! As you suggest, I'll put in some logic to look for previous-style dependencies section syntax as well - a use old dump format if any old stuff detected.

The Cylc periodicals suites are the only ones I could find (here) that knew about this initial cycle time syntax - everything else always used the full form.

[hjoliver]

I'll be posting a pull request shortly to address this loose end: https://github.com/cylc/cylc/issues/119#issuecomment-39160387

[hjoliver]

everything else always used the full form.

in retrospect, I should not have allowed the abbreviated form...

[benfitzpatrick]

It might be nice to allow writing final cycle time relative to the initial cycle time (e.g. +P6Y, +P2D) - thoughts? (ISO wiki diff)

[arjclark]

I like the idea, I think a number of our climate users work in terms of a run taking N Days or somesuch rather than "starts on X, cycles every Y days, I want it to happen N times... so stop on Z".

[arjclark]

@hjoliver - I'm hitting a slight sticking point with removal of start-up/cold-start tasks with looking at implementing a proof of concept suite. With the previous start-up/cold-start tasks we could warm start a suite and they would be excluded from the graphing automatically. While the R1 makes sense to me as a method of running once at the start of the suite - sold on this now - I'm not sure what to do about only having tasks run under cold-start conditions? I guess a "cold-start-only" entry under special tasks will be needed in order to mark tasks as run under cold-start only and automatically exclude under a warm-start?