Interpretation of negative years in the units attribute

peterkuma commented 4 years ago

I have encountered an issue with using a negative year in the units attribute of time variables in NetCDF files. The interpretation in cftime (Python) is that year zero does not exist, while in other software such as Panoply the interpretation is that year zero exists. This affects how the time variable is read and displayed, and effectively causes one year difference between the different implementations. The CF Conventions (Section 4.4. Time Coordinate) do not explicitly state how negative years should be treated, except for stating that year 0 has a special meaning. On the contrary, ISO 8601, seems to be more on the side of including year 0.

In particular, this issue comes up when using Julian date in NetCDF files, which has a reference time of 1 January 4713 BCE, 12:00 UTC. As of now it is impossible to use it in NetCDF files and get consistent results in Python (through the netCDF4 package) and Panoply.

I suppose there are multiple possible solutions to the problem. Either all implementations start using the same method of counting negative years (and it would be helpful if the CF Conventions make this unambiguous), or there would have to be information about the year numbering convention included in the NetCDF file, such as a new attribute or an indicator included in the units or calendar attributes.

Related issue in cftime: #200.

JonathanGregory commented 3 years ago

Thanks, well-spotted. That division doesn't seem useful to me either. I would be happy with deleting the 4.4.1 subsection break in both the standards document and the conformance document.

Dave-Allured commented 3 years ago

Hmmm. I can't agree with deleting the subsection break. There is too much history behind it, and it makes a lot of sense the way it is. This style is widely used elsewhere in the conformance document.

I think we should either move sentences, or leave this exactly the way it is now with my current additions. I mildly favor the latter. Your thoughts?

Dave-Allured commented 3 years ago

One more thing. I want to work this line in somewhere, but I can't figure out where to put it. Your thoughts?

Do not publish any serious works on April 1.

JonathanGregory commented 3 years ago

One more thing. I want to work this line in somewhere, but I can't figure out where to put it. Your thoughts?

Do not publish any serious works on April 1.

We could perhaps define a nofool calendar, in which 1st April is not a permitted date.

JonathanGregory commented 3 years ago

Dear @Dave-Allured

Hmmm. I can't agree with deleting the subsection break. There is too much history behind it, and it makes a lot of sense the way it is. This style is widely used elsewhere in the conformance document.

The conformance document follows the standard document in sections and subsections. When someone wonders which section to put something in, because the distinction doesn't seem clear, I think it's reasonable to infer that the distinction is actually unnecessary, and we can save thought and possible confusion by abolishing it. Introducing divisions is good for efficiency when the amount of stuff is too hard to navigate without them, but if they're not essential I'd rather not have them, because they make things harder. That's the reason for my response! I'm always uncomfortable when a section has a single subsection. I don't see the point in that. I know there are other instances in the standards document and I don't like them but we have more serious matters to attend to than this mild discomfort! :-)

I think we should either move sentences, or leave this exactly the way it is now with my current additions. I mildly favor the latter. Your thoughts?

If you prefer not to muddy the waters by removing the division, I'd leave it where it is and we will live with unclear rationale.

Best wishes

Jonathan

Dave-Allured commented 3 years ago

All right, leave the section break as is. Aside from trepidation about today's date, I think this PR #315 is ready to roll. Until the next brilliant insight comes along!

JonathanGregory commented 3 years ago

It would be helpful if others who commented earlier could please have a look at this version and give their opinion e.g. @larsbarring @martinjuckes @peterkuma. Please note that this issue has the limited intent of clarifying when negative years are allowed. It doesn't aim to deal with all outstanding issues concerning calendars. Thanks. Jonathan

taylor13 commented 3 years ago

I am in favor of the intent of the changes. With respect to the wording, I offer two general suggestions:

I think in reference to dates and calendars, we should avoid the terms "real-world", and, rather, say exactly what we mean by it.
I think the references to and description of the possible use of year 0 to represent climatological time are very confusing and need to be rewritten.

Specifically, I suggest: Replace the paragraph,

The year number in the reference date/time may be zero or negative, except where prohibited 
for certain calendars, as noted below.  A negative year number is indicated with a preceding 
minus sign.  When year zero is allowed, year numbering is algebraic; i.e., year zero is included 
in the counting.  The interpretation for historical year numbering is consistent with 
link:$$https://en.wikipedia.org/wiki/ISO_8601$$[ISO 8601] (modern revisions); 
by convention, year zero = 1 BC, year -1 = 2 BC, and so on.

with

The year number in the reference date/time may be zero or negative, except in the case of a ``julian``
 calendar or a mixed Gregorian/Julian calendar (denoted by either ``standard`` or ``gregorian``).  
When allowed, a negative year number is indicated with a preceding minus sign, and year numbering 
includes year 0 (in contrast with the Anno Domini (AD) system of numbering years commonly used by
 western historical scholars in which there is no year 0).  The inclusion of year zero in some CF 
calendars _is_ consistent with astronomical year numbering and with 
link:$$https://en.wikipedia.org/wiki/ISO_8601$$[ISO 8601] (modern revisions), which stipulate that 
year zero = 1 BC, year -1 = 2 BC, and so on.

Also replace the paragraph,

There is a special meaning for year zero in the real-world calendar, that is not recommended for 
current work.  See <<climatological-statistics>>.  This is an exception to the prohibition of year 
zero for certain calendars.

with

The one exception to the prohibition of year zero for certain calendars is in the now deprecated 
special use of year zero to indicate a climatology (see <<climatological-statistics>>).

In the conformance document, isn't the second sentence of the following wrong?

The use of a reference date/time in the year 0 to indicate climatological
time is deprecated. This restriction only applies to the real-world
calendar as used by the udunits package.

This might be interpreted as meaning that use of year 0 for climatology is o.k. for non-real-world calendars, but isn't such use absolutely restricted for any calendar that includes year 0 as a possible real year (like, for example, the proleptic_gregorian calendar)? If so, the above paragraph needs to be reworded.

This might not belong in this ticket, but in the climatological-statistics section, I would replace the paragraph

The COARDS standard offers limited support for climatological time. For compatibility with COARDS, 
time coordinates should also be recognised as climatological if they have a units attribute of 
time-units relative to midnight on 1 January in year 0 i.e. since 0-1-1 in udunits syntax, and provided 
they refer to the real-world calendar. We do not recommend this convention because (a) it does not 
provide any information about the intervals used to compute the climatology, and (b) there is no 
standard for how dates since year 1 will be encoded with units having a reference time in year 0, 
since this year does not exist; consequently there may be inconsistencies among software 
packages in the interpretation of the time coordinates. Year 0 may be a valid year in 
non-real-world calendars, and therefore cannot be used to signal climatological time in such cases.

with

For compatibility with the COARDS standard, a climatological time coordinate may (for certain 
calendars) be indicated by setting the time coordinate's units attribute to midnight on 1 January in 
year 0 (i.e., since 0-1-1), although this is no longer recommended.  Use of year 0 for this purpose 
is absolutely forbidden in the case of a calendar where year 0 is a valid year (i.e., for all calendars 
except a ``julian`` or a Gregorian/Julian calendar). The reasons for generally avoiding the special 
use of year 0 to indicate a climatology are: a) it does not provide any information about the intervals 
used to compute the climatology, and b) there may be inconsistencies among software packages 
in the interpretation of the time coordinates with a reference time of year 0.

larsbarring commented 3 years ago

I fully support this, it is a good step towards clarifying the meaning -- and use -- of different calendars, in particular together with what is discussed in #319 . And I support the changes suggested by Karl, with the addition that the use of "gregorian" should be coordinated with #319. If the last of the suggested changes (the one for the climatological-statistics section) is deemed out-of-scope for this issue, I think that it deserves its own focussed issue.

Edit: I forgot to mention that I do support Karl's suggestion to avoid the word "real-world [calendar]" because the CF Conventions may very well be used to store observational data (present and historic), and then there are many different views across the globe of what is (and has been) a real-world calendar.

peterkuma commented 3 years ago

Here are my (minor) comments on the PR #315, based on my limited understanding of the existing CF implementations:

Linking to Wikipedia (ISO_8601) is perhaps unwise, because we don't know how the article is going to change in the future, and the standard should be as standalone as possible (Wikipedia is not intended to be an authoritative resource on any topic). It would be better to link to either a specific version of the article, or link to the ISO 8601 standard instead.
"BC" should perhaps be "BCE", which as I understand is the preferred term in science. I have found one occurrence of the term "BCE" in the current CF Conventions in Section 4.4.1, as part of a quote from the udunits(3) man page, and no occurrence of "BC".
I think it would be good to define a "proleptic_julian" calendar in which date/times earlier than 1-1-1 0:0:0 years are allowed, in analogy with the "proleptic_gregorian" calendar.

martinjuckes commented 3 years ago

I support Karl's proposal. This is a great improvement, and resolves some serious issues of ambiguity.

However, I suggest expanding the calendar definitions slightly in section 4.1 and minimise the discussion of different calendars in the introduction of chapter 4. In Karl's text there are, I think, too many conditional statements which could cause confusion. Placing the rules for each calendar in the relevant part of section 4.1 could simply the text.

In section 4.1:

julian: -- Julian calendar. Date/times earlier than 1-1-1 0:0:0 are prohibited. Year 1 corresponds to AD 1 in the Julian calendar.

gregorian or standard -- Mixed Gregorian/Julian calendar as defined by Udunits. This is the default. Date/times earlier than 1-1-1 0:0:0 are prohibited. Year 1 corresponds to AD 1 in the Gregorian calendar.

proleptic_gregorian. -- A Gregorian calendar extended to dates before 1582-10-15. That is, a year is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is divisible by 400. Year 1 corresponds to AD 1 and year 0 corresponds to 1BC in the proleptic Gregorian calendar.

In the introduction, replace:

The year number in the reference date/time may be zero or negative, except in the case of a ``julian``
 calendar or a mixed Gregorian/Julian calendar (denoted by either ``standard`` or ``gregorian``).  
When allowed, a negative year number is indicated with a preceding minus sign, and year numbering 
includes year 0 (in contrast with the Anno Domini (AD) system of numbering years commonly used by
 western historical scholars in which there is no year 0).  The inclusion of year zero in some CF 
calendars _is_ consistent with astronomical year numbering and with 
link:$$https://en.wikipedia.org/wiki/ISO_8601$$[ISO 8601] (modern revisions), which stipulate that 
year zero = 1 BC, year -1 = 2 BC, and so on.

with:

The year number in the reference date/time may be zero or negative, except in the case of 
a `julian` or `standard`/`gregorian` calendars (see Section 4.1 below for more details).

Note:

inclusion of year zero is clearly stated in this first sentence, so it doesn't need to be repeated.
The connection to AD/BC is easier to discuss in the context of the calendars for which it is relevant.
The wikipedia article on ISO 8601 is useful, but using it as a link in the context of the AD/BC question is misleading: ISO 8601 does not mention AD/BC .. it just states that years should be numbered sequentially, with 0 preceding 1. If a reference for AD/BC is needed, there is a good discussion in Christian chronology in the Encyclopedia Britannica and elsewhere, but I'm not sure that we need to discuss it in the Conventions document. The AD/BC system appears to predate the introduction of zero into common usage by the Christian record keepers. Placing references to AD and BC in the relevant calendar definitions makes their meaning clear.

taylor13 commented 3 years ago

I like Martin's restructuring of the information. Two further suggestions:

In section 4.1 the no_leap or 365_day calendar is defined as a "Gregorian calendar without leap years, i.e., all years are 365 days long." I don't think we want to exclude 0 or negative years for this calendar, so we should drop the reference to "Gregorian calendar" and simply define it as "A calendar in which all years are 365 days long."

A similar edit of the all_leap or 366_day calendar is also needed.

If the length of months is relevant, we might need to mention that they are the same as in the Julian and Gregorian calendars, but with February a fixed length (28 or 29 days, respectively).

Finally, I support @peterkuma's suggestion to replace "BC" with "BCE" since this is a more neutral term and is consistent with usage elsewhere.

larsbarring commented 3 years ago

Yes, I agree, Martin's wording is good -- it is clear and succinct. A couple of minor comments:

The explanation of the gregorian or standard calendar should state that "Year 1 corresponds to AD 1 in the Julian calendar."
The last sentence of the explanation of the proleptic_gregorian calendar would become clearer if written "In the proleptic Gregorian calendar year 1 corresponds to AD 1 and year 0 corresponds to 1BC/BCE.

davidhassell commented 3 years ago

Hello,

A lot of suggestions have been made on this issue since the last update to the associated pull request (#315). I'm not sure if consensus has been reached, as the conversation has paused, but is it possible for someone to synthesise the suggestions made here during April 2021 so that the PR can be updated? (pinging @Dave-Allured for extra visibility as the PR owner).

Many thanks, David

JonathanGregory commented 3 years ago

Dear @davidhassell et al.

I will produce some text synthesising the comments made since the current pull request was written.

Since then, the preamble on calendars has been modified as a result of the agreement of issue 313 on leap seconds. As a result, I suggest that the new sentence from the pull request of this issue should go in a different place. Below I reproduce the new text from the working draft, because I think it's useful in spelling out what "calendar" means in CF. I have inserted the extra sentence in bold where I'd suggest putting it.

Cheers

Jonathan

4.4.1 Calendar

A date/time is the set of numbers which together identify an instant of time, namely its year, month, day, hour, minute and second, where the second may have a fraction but the others are all integer. A time coordinate value represents a date/time. In order to calculate a time coordinate value from a date/time, or the reverse, one must know the units attribute of the time coordinate variable (containing the time unit of the coordinate values and the reference date/time) and the calendar. The choice of calendar defines the set of dates (year-month-day combinations) which are permitted, and therefore it specifies the number of days between the times of 0:0:0 (midnight) on any two dates. Date/times which are not permitted in a given calendar are prohibited in both the encoded time coordinate values, and in the reference date/time string.

When a time coordinate value is calculated from a date/time, or the reverse, it is assumed that the coordinate value increases by exactly 60 seconds from the start of any minute (identified by year, month, day, hour, minute, all being integers) to the start of the next minute, with no leap seconds, in all CF calendars. This assumption has various consequences when real-world date/times from calendars which do contain leap seconds (such as UTC) are stored in time coordinate variables:

Any date/times between the end of the 60th second of the last minute of one hour and the start of the first second of the next hour cannot be represented by time coordinates e.g. 2016-12-31 23:59:60.5 cannot be represented.
A time coordinate value must not be interpreted as representing a date/time in the excluded range. For instance, 60 seconds after 23:59 means 00:00 on the next day.
A date/time in the excluded range must not be used as a reference date/time e.g. seconds since 2016-12-31 23:59:60 is not a permitted value for units.
It is important to realise that a time coordinate value does not necessarily exactly equal the actual length of the interval of time between the reference date/time and the date/time it represents.

It is recommended that the calendar be specified by the calendar attribute of the time coordinate variable. The values currently defined for calendar are:

[... to be continued]

JonathanGregory commented 3 years ago

Dear all

I have drafted a new version of the affected parts of the text of Sect 4.4, taking account of the comments made since the pull request was revised, mostly as suggested but not quite, as follows:

I included the deprecation of gregorian from issue 319.
I suggest we insert the paragraph with rules about years before 1 just before the list of calendars, rather than the earlier place where it appears in the PR. That follows Martin's point that it's easier to deal with the detail in one place. If we put it here, I don't think we need to insert the prohibitions into the definitions of individual calendars.
I don't think we need to talk about year 0 = 1 BCE etc. because we don't allow years before 1 in the standard and julian calendars (except for climatology). I don't believe it is correct to say that year 0 in proleptic_gregorian is 1 BCE, because that calendar doesn't have the right year lengths. It is not a "real-world" calendar.
As Karl suggested, I avoided using Gregorian to describe other calendars, because they're not.
I stated the Julian rule for leap years, and that we use Gregorian month lengths in all calendars except 360_day and none.
I included Karl's text for the cell methods section, with small changes (in bold) from his version. I say that it is impossible, rather than forbidden, for the year-0 convention to be used for climatological time in calendars where it's a valid year.

I think the reference to udunits.dat is out-of-date, isn't it? What should it be changed to?

Best wishes

Jonathan

4.4 Time coordinate

Variables representing time must always explicitly include the units attribute; there is no default value. The units attribute takes a string value formatted as per the recommendations in the Udunits package UDUNITS. The following excerpt from the Udunits documentation explains the time unit encoding by example:

The specification seconds since 1992-10-8 15:15:42.5 -6:00 indicates second since October 8th, 1992 at 3 hours, 15 minutes and 42.5 seconds in the afternoon in the time zone which is six hours to the west of Coordinated Universal Time (i.e. Mountain Daylight Time). The time zone specification can also be written without a colon using one or two digits (indicating hours) or three or four digits (indicating hours and minutes).

The acceptable units for time are listed in the udunits.dat file. The most commonly used of these strings (and their abbreviations) includes day (d), hour (hr, h), minute (min) and second (sec, s). Plural forms are also acceptable. The reference date/time string (appearing after the identifier since) may include date alone; date and time; or date, time, and time zone. The reference date/time string is required.

...

4.4.1 Calendar

A date/time is the set of numbers which together identify an instant of time, namely its year, month, day, hour, minute and second, where the second may have a fraction but the others are all integer. A time coordinate value represents a date/time. In order to calculate a time coordinate value from a date/time, or the reverse, one must know the units attribute of the time coordinate variable (containing the time unit of the coordinate values and the reference date/time) and the calendar. The choice of calendar defines the set of dates (year-month-day combinations) which are permitted, and therefore it specifies the number of days between the times of 0:0:0 (midnight) on any two dates. Date/times which are not permitted in a given calendar are prohibited in both the encoded time coordinate values, and in the reference date/time string. It is recommended that the calendar be specified by the calendar attribute of the time coordinate variable.

...

The values currently defined for calendar are listed below. In all calendars except 360_day and none, the lengths of the months are the same as in the Gregorian calendar for leap years and non-leap years. In the julian and the default standard mixed Gregorian/Julian calendar, dates in years before year 0 (i.e. before 0-1-1 0:0:0) are not allowed, and the year in the reference date/time of the units must not be negative. In these calendars, year zero has a special use to indicate a climatology (see [climatological-statistics]), but this use of year zero is deprecated. In other calendars, years before year 1 are allowed.

standard: Mixed Gregorian/Julian calendar as defined by Udunits. This is the default. A deprecated alternative name for this calendar is gregorian. Year 1 in this calendar is year 1 AD or CE of the Julian calendar.

proleptic_gregorian: A calendar with the Gregorian rules for leap-years extended to dates before 1582-10-15. That is, a year is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is divisible by 400.

julian: Julian calendar, in which a year is a leap year if it is divisible by 4, even if it is also divisible by 100.

noleap or 365_day: A calendar with no leap years, i.e. all years are 365 days long.

all_leap or 366_day: A calendar in which every year is a leap year, i.e. all years are 366 days long.

360_day: A calendar in which all years are 360 days and divided into 30-day months.

none: No calendar.

The calendar attribute may be set to none in climate experiments that simulate a fixed time of year. The time of year is indicated by the date in the reference date/time of the units attribute. The time coordinates that might apply in a perpetual July experiment are given in the following example.

...

Replace the paragraph in 7.4 beginning "The COARDS standard" with

For compatibility with the COARDS standard, a climatological time coordinate in the default standard and julian calendars may be indicated by setting the time coordinate's units attribute to midnight on 1 January in year 0 (i.e., since 0-1-1). This convention is deprecated because it does not provide any information about the intervals used to compute the climatology, and there may be inconsistencies among software packages in the interpretation of the time coordinates with a reference time of year 0. Use of year 0 for this purpose is impossible in all other calendars, because year 0 is a valid year.

Modify conformance document section 4.4 recommendations:

The use of time coordinates in year 0 and reference date/times in year 0 to indicate climatological time is deprecated.

Add to conformance document section 4.4.1 recommendations:

A time coordinate variable should have a calendar attribute.
The value standard should be used instead of gregorian in the calendar attribute.

davidhassell commented 3 years ago

Thank you, Jonathan. Your text is very clear to me.

All of the changes required for #319 are now here - is it OK for #319 to just refer to the PR for this issue, rather removing these changes from this issue and recreating in a bespoke PR for #319?

JonathanGregory commented 3 years ago

Dear @davidhassell

Yes, you're right, this does cover https://github.com/cf-convention/cf-conventions/issues/319. I hadn't thought of that, but it could be convenient if no-one objects to merging them. I didn't redefine the 365_day and 366_day calendars in terms of proleptic Gregorian because I had overlooked that point of yours. I did in in a different way, by stating that months are of Gregorian lengths.

Jonathan

davidhassell commented 3 years ago

it could be convenient if no-one objects to merging them

It is fine by me.

I did in in a different way, by stating that months are of Gregorian lengths.

Which I think works well!

larsbarring commented 3 years ago

@JonathanGregory This is all very good, and merging the two seems logical, especially as #319 is an offshoot from this issue. A couple of minor comments:

In the definition of the standard calendar the switch-over date could be mentioned. How about: "standard: Mixed Gregorian/Julian calendar as defined by Udunits. Dates onwards from 1582-10-15 follow the Gregorian system and earlier dates follow the Julian system. This is the default."
Could the deprecation of the gregorian calendar be a bit more prominent? After all, it was a calendar in its own right, even though being equivalent to the standard calendar (or vice versa). How about below the list of calendars writing a sentence (own paragraph) like:
"The gregorian calendar is deprecated, instead use the equivalent standard calendar"
Indeed, the udunits.dat file is outdated, see this comment from 2017. It has been replaced by xml files, which are a bit awkward [for humans] to read. @davidhassell might have a better overview.

davidhassell commented 3 years ago

Hello Lars,

I like your suggestions.

With regards Udunits, the time units are distributed across multiple XML files. For example, hour is defined in udunits2-accepted.xml and second is defined in udunits2-base.xml. Perhaps it might be better to say:

"The acceptable units for time are listed in the Udunits database [UDUNITS]."

where [UDUNITS] is the existing reference in the Bibliography section, which provides a version-independent link (http://www.unidata.ucar.edu/software/udunits/) to the Udunits home page. The online viewable database for a particular Udunits version is easily findable from the given link (the latest is https://www.unidata.ucar.edu/software/udunits/udunits-2.2.28/udunits2.html#Database).

Incidentally, what is the correct way to write Udunits? CF uses "Udunits", but Unidata documentation seems to favour "UDUNITS".

Mentioning @semmerson here in case I've missed something.

JonathanGregory commented 3 years ago

Dear @larsbarring and @davidhassell

Taking Lars's points in reverse.

I agree with what David proposes for referring to UDUNITS instead of udunits.dat.
It seems better to me not to have an independent entry for the deprecated gregorian, which would make it more prominent than it is now. If someone is reading the CF standard to find out what gregorian means, because they've found it in CF-netCDF data, and they don't immediately notice it in the entry for the standard calendar, they'd easily find it with a text search. If they're intending to write data, they're better off not knowing about gregorian!
I agree that the standard calendar of UDUNITS should be explained. Looking again at the CF 1.8, I see that it is explained, but this comes further down, following the text for arbitrary calendars about month_lengths etc. I have reproduced the current text below, for reference. I propose that we delete the existing text and instead insert into the entry for this calendar an abbreviated version rewritten to be more consistent with CF text and less UDUNITS-specific, as follows:

standard: Mixed Gregorian/Julian calendar as defined by [UDUNITS]. This is the default calendar. A deprecated alternative name for this calendar is gregorian. In this calendar, date/times after (and including) 1582-10-15 0:0:0 are in the Gregorian calendar, in which a year is a leap year if either (i) it is divisible by 4 but not by 100 or (ii) it is divisible by 400. Date/times before (and excluding) 1582-10-5 0:0:0 are in the Julian calendar. Year 1 AD or CE in the standard calendar is also year 1 of the julian calendar. In this calendar, 1582-10-15 0:0:0 is exactly 1 day later than 1582-10-4 0:0:0 and the intervening dates are undefined. Therefore it is recommended that date/times in the range from (and including) 1582-10-5 0:0:0 until (but excluding) 1582-10-15 0:0:0 should not be used as reference in units, and that a time coordinate variable should not include any date/times in this range, because their interpretation is unclear. It is also recommended that a reference date/time before the discontinuity should not be used for date/times after the discontinuity, and vice-versa.

proleptic_gregorian: A calendar with the Gregorian rules for leap-years extended to years before 1582. All dates consistent with these rules are allowed, both before and after 1582-10-15 0:0:0.

In this proposed text, I have been more explicit about the problems with the calendar and the recommendations to avoid them. I have moved the Gregorian leap-year rules from proleptic_gregorian to standard, and I have stated explicitly that the illegal range of dates in standard is OK in proleptic_gregorian, as I believe to be the case. I don't think any of this changes the convention, but I think it's useful to be clear. Is it all OK? It would imply some more things in the conformance document.

The existing text also says, "For time coordinates that do cross the discontinuity the proleptic_gregorian calendar should be used instead." I don't think this recommendation makes sense. You can't use the proleptic_gregorian calendar to represent dates in the Julian calendar. If they are real historical dates, with daily resolution, you have no choice but standard to do it properly. If you don't mind about being inexact, for instance if you only want the 1st of the month, you could use any calendar, including model calendars, to encode the dates; proleptic_gregorian wouldn't be particularly better. Hence I propose we should omit that recommendation.

Cheers

Jonathan

Current text:

The mixed Gregorian/Julian calendar used by Udunits is explained in the following excerpt from the udunits(3) man page:

The udunits(3) package uses a mixed Gregorian/Julian calendar system. Dates prior to 1582-10-15 are assumed to use the Julian calendar, which was introduced by Julius Caesar in 46 BCE and is based on a year that is exactly 365.25 days long. Dates on and after 1582-10-15 are assumed to use the Gregorian calendar, which was introduced on that date and is based on a year that is exactly 365.2425 days long. (A year is actually approximately 365.242198781 days long.) Seemingly strange behavior of the udunits(3) package can result if a user-given time interval includes the changeover date. For example, utCalendar() and utInvCalendar() can be used to show that 1582-10-15 preceded 1582-10-14 by 9 days.

Due to problems caused by the discontinuity in the default mixed Gregorian/Julian calendar, we strongly recommend that this calendar should only be used when the time coordinate does not cross the discontinuity. For time coordinates that do cross the discontinuity the proleptic_gregorian calendar should be used instead.

semmerson commented 3 years ago

Hi David,

Incidentally, what is the correct way to write Udunits? CF uses "Udunits",

but Unidata documentation seems to favour "UDUNITS".

UDUNITS

taylor13 commented 3 years ago

Thanks, Jonathan. Seems clear and close now. With these changes we'll be able to close both 298 and 319! Real progress.

larsbarring commented 3 years ago

Dear Jonathan, Many thanks, in particular I like the comprehensive explanation of the standard calendar. Many thanks also to @Dave-Allured for initiating this issue and steering the conversation into the good shape that allowed the final lap reaching all the way to this point. Just echoing Karl: real progress!

JonathanGregory commented 3 years ago

If everyone is content with the proposed text, I will make a new pull request for this issue. In the pull request, I will add @Dave-Allured to the CF authors in recognition of his raising the issue and the work he has done on it (unless he would prefer not). Jonathan

davidhassell commented 3 years ago

Thanks @JonathanGregory - I'm happy with your latest proposed text; and thanks @semmerson for putting us right :)

JonathanGregory commented 3 years ago

I have prepared https://github.com/cf-convention/cf-conventions/pull/331 for this issue and https://github.com/cf-convention/cf-conventions/issues/319. Since the changes are exactly as discussed and agreed above, I propose that the pull request should be merged on 23rd July, three weeks from today, if no concerns are raised.

davidhassell commented 3 years ago

Thanks, @JonathanGregory. Your PR looks good to me.

JonathanGregory commented 3 years ago

@zklaus has made the following comment on the PR

Perhaps this is a good opportunity to update the second udunits.dat link in the following line 217 as well.

I also wonder if we should be explicit in whether CF follows udunits in the definitions of year and month. The way these two paragraphs are written, the reader might come away with the impression that we are only warning of potential misinterpretation by some other software, whereas my understanding so far is that we adopt the udunits definition, albeit begrudgingly. But perhaps my understand is wrong or you would prefer to address this in a separate issue?

JonathanGregory commented 3 years ago

I realise that we agreed to delete the existing excerpt from the UDUNITS that describes the standard calendar, because we have put that information in the description of that calendar. I will modify the PR.

I agree with the point @zklaus makes about the deprecated units. Would the following be OK:

UDUNITS defines a year to be exactly 365.242198781 days (the interval between 2 successive passages of the sun through vernal equinox). It is not a calendar year. UDUNITS defines a month to be exactly year/12, which is not a calendar month. The CF standard follows UDUNITS in the definition of units, but we recommend that year and month should not be used, because of the potential for mistakes and confusion.

I propose to delete this text: "UDUNITS includes the following definitions for years: a common_year is 365 days, a leap_year is 366 days, a Julian_year is 365.25 days, and a Gregorian_year is 365.2425 days." It is correct, but I don't think it's necessary or helpful.

Although this PR can eliminate udunits.dat from the time coordinate section, the file is mentioned elsewhere in the standard, so I think a separate defect ticket is needed on that subject.

JonathanGregory commented 3 years ago

I have made the above changes in https://github.com/cf-convention/cf-conventions/pull/331

davidhassell commented 3 years ago

Thanks, Klaus and Jonathan - This alternation (https://github.com/cf-convention/cf-conventions/pull/331/commits/bd7498d3625a1ee464f375952bfd202f7fbc4371) looks good to me.

JonathanGregory commented 3 years ago

Thanks, David. If @zklaus and others are also content with the new version, I suppose it can be accepted three weeks from three days ago, which makes 23rd July

JonathanGregory commented 3 years ago

There have been no further comments for three weeks and sufficient support has been expressed, so this change is therefore accepted according to the rules. I have merged https://github.com/cf-convention/cf-conventions/pull/331. Thanks to all contributors to the discussion, especially @peterkuma, who raised the issue, and @Dave-Allured, who worked on the text and has been added to the list of authors of the CF convention.

cf-convention / cf-conventions