search

co-cddo / open-standards

Collaboration space for discussing and exploring technical and data standards
135 stars 18 forks source link

Date-Times and Time-stamps Challenge #37

Closed Lawrence-G closed 6 years ago

Lawrence-G commented 7 years ago

Date-Times and Time-stamps

The case for ISO8601

A short title which describes the challenge. Avoid acronyms or jargon.

Category

Challenge Owner

Terence Eden and Lawrence Greenwood are the challenge owners. The challenge was originally posted by Chris Little.

Short Description

Humans are adept at accommodating and understanding a variety of time and date notations, such as 7th July 2016, 7/7/16, July 7 2016, 2016-07-07 and 10:12am, 12 past 10 in the morning, 09:12GMT, 09:12UTC, 10:12BST, 12:12EEST.

There is a well established, global, international notation for dates and times: ISO8601. This standard, and subsets of it, are widely used in the ICT domain, and have the advantage of automatically being 'sortable' on most computer systems. That is, dates like 7 Jul, 7 Aug and 7 Sept would usually sort into the alphabetic order: Aug, Jul, Sept, rather than the expected temporal order Jul, Aug, Sept.

By adopting ISO8601 notation for dates and times in online documents, spreadsheets, databases and for filenames and online references such as URLs, greater interoperability will be achieved at less cost and with less confusion.

2016-07-07T09:23:00Z

User Need

Inconsistent recording of dates and times, causing confusion especially with the international exchange of computer documents, where there are different cultural practices. For example, confusion of UK and USA dates, such as 9/11/2001, will be reduced.

Expected Benefits

Users will be able to more easily order documents of interest into strict date-time order, and there will be greater interoperability for transferring date-time information between disparate computer systems. There will be a better validation of date-time information. Sorting algorithms can be simplified. Listings can be more readily understood. International exchange of information improved.

Lawrence-G commented 7 years ago

This challenge was originally suggested on standards.data.gov.uk

This subject has been discussed before in issue #7. That discussion was in answer to the question ISO8601 or RFC3339 ? I don't think that consensus was reached. This challenge was intended to make the case for ISO8601 but the discussion is still open.

galund commented 7 years ago

Unfortunately I think many real-world end users (i.e. citizens) would be confused by ISO8601.

HTML now provides a great way to get around this, i.e. the <time> element, with its optional datetime attribute. I think we could be strongly recommending / mandating this, instead of just saying "use ISO8601 everywhere".

In other words the recommendation should (I think) be:

1) dates and times embedded in data are purely for machine consumption (i.e. are not intended to be read directly by end users) should use ISO8601 format, where that is permitted by the format of the data in question.

2) documents intended to be read by end-users SHOULD contain a visible indication of their locale;

3) dates and times embedded in such documents MUST meet the expectations of the relevant locale and SHOULD use an unambiguous format e.g. by spelling out month names and mentioning the time zone of any time;

4) references to dates and times MUST include an unambiguous date/time reference where the format supports it, for example in HTML by using the <time> element with a datetime attribute.

edent commented 7 years ago

Thanks @galund. For the benefit of others, there's a good description of <time> on MDN. As you point out W3C mandates a subset of ISO8601

Unfortunately I think many real-world end users (i.e. citizens) would be confused by ISO8601.

This might be a good area for user research. Anecdotally, we know that people get confused between UK and US format dates (03/07/2017 is either 3rd July or 7th March). When documents are sent from foreign embassies, it can be confusing to determine whether the times are local or UTC.

I'm not sure that adding JUN or BST is necessarily less confusing than the 8601 representation. But we can see.

Thanks for the comments.

PeterParslow commented 7 years ago

There are bits of ISO 8601 that are confusing. I was going to say that it is generally most appropriate to define a specific subset for use in a particular application / exchange - as is done by W3C for HTML time & the IETF in RFC3339.

So as usual one needs more about the use case / user requirements. For example, 8601 includes a way to encode repeating time intervals & 'truncated times' (e.g. just which century you're talking about). If these aren't needed, they just cause confusion. Conversely, 8601 doesn't include things like 'the Jurassic' or 'before Henry VIII' - which some use cases require (ISO 19108 can help with these).

It could be that the W3C or IETF profiles are sufficient for most of the use cases in mind (although I don't like the -0 time zone in RFC3339 for several reasons!)

A few thoughts on galund's proposal: at 2 I wouldn't use the term 'locale'; it's specifically the time zone which is required (including daylight savings time or not). More generally, it's important to distinguish 'machine readable' use cases & 'purely human readable' ones (if there are any).

puntofisso commented 7 years ago

I don't have much useful to say if not to reinforce what Peter and Terence have already noted: a collection of user needs and use cases to discuss. I would say that ISO 8601 is generally a good idea as it's the most general and recognised standard; documenting how its use is intended in the specific use cases would be a good starting point:

online documents, spreadsheets, databases and for filenames

these might require different subsets for different use cases - most online documents will need to be human readable; spreadsheets and databases not so much, as they supply format conversion procedures; filenames should be intended to facilitate sorting.

haydavie commented 7 years ago

Under NHS Connecting for Health, back 2009, we needed to make specific statements for system/ vendor suppliers around time and dates, especially interchanges between systems and vulnerable changeovers to/from British Summer Time (BST) and the safer consistent display of dates.

Accurate recording of time is vital not only for clinical records but also scheduling and booking for example. It is needed to support:

To avoid confusion and clinical errors its was stated that the display of time must be clearly displayed in local time suitable for the consuming user’s perspective, which may include accounting for BST as well as translating UTC to local time

Essentially, in order to avoid the possibility of misinterpretation by receiving systems the sender must adhere to the following rules: • Business dates and times (e.g. effective time of clinical events/statements) must always use local time • Where precision of a time stamp includes hours, the time stamp must include the time zone offset • Whether time is recorded in UTC, GMT or BST the time zone offset must be stated.

A whole range of good practice was also developed (and available in the National Archives site) which included Date and Time

Date Time Input (and implementation guidance) http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig/datetime.pdf http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig/datetimeqig.pdf

Date & Time Display (and implementation guidance) http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig/datedisplay.pdf http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig/timedisplay.pdf http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig/datetimedispqig.pdf

Source: http://webarchive.nationalarchives.gov.uk/20160921135207/http://systems.digital.nhs.uk/data/cui/uig

edent commented 7 years ago

Thanks @haydavie. To clarify, an ISO8601 datetime of 5:30pm during BST would look like

2017-04-19T17:30:18+01:00

According to those NHS documents, the preferred way of displaying that time would be

19-Apr-2017 17:30 - with no timezone indicator.

I think we will have to carefully state that this standard is primarily for:

PeterParslow commented 7 years ago

That clarification effectively assumes a profile of 8601. The separators (hyphen, colon) are not required, although they are recommended when using 8601 for human readable text. To put it another way, 8601 itself recognises & supports distinct use cases.

edent commented 7 years ago

@PeterParslow you are quite right - I was using it as an example to show that it's a long way from "15th Sept 2017" for example.

Lawrence-G commented 7 years ago

I’ve had a couple of ‘off Hub’ discussions that raise some points of note on this challenge - This is a summary - How many staff and/or public sector bodies can access either ISO 8601: 2004, or BS ISO 8601 :2005, or indeed any ISO BSI standards? A department that doesn't have access to either version of this standard cannot comment on its suitability or implement it?

However you access the BS ISO 8601, please look at these two documents also: 1.    BSI 17/30339386 DC, dated 05th January 2017, “Draft BS ISO 8601-1 Data elements and interchange formats - Information interchange - Representation of dates and times Part 1: Basic rules”, and 2.    BSI 17/30339389 DC, same date, similar title but “Part 2: Extensions”. The window of opportunity to comment on both these Drafts for Comment has closed. Publication for the updated BSi parts 1 and 2 is scheduled for 30th January next year.

As far as I can tell, changes between the 2005 version and the DC’s are not red-lined, so this article [https://www.iso.org/news/2017/02/Ref2164.html ] on ISO’s website last month may save some time, as it explains some of the background to the changes. I’m not sure of the policy for standard’s under development, but my instinct would be to park the challenge until next February.

Re a profile for the standard in this challenge- I am concerned that a detailed profile with excerpts from the standard would cross the line between fair dealing and copyright violation. Below, is part of the most relevant clause from one of BSi’s licence agreements:   “End users may copy up to 10% of the content of a Standard and paste it to another document for internal use at the licensed site(s) only. The copied content must contain "Copyright BSI © Date (date is the date of copyrighted material”

There are options - a reference to the standard that includes enough detail in the body of the text to say that the profile will use paragraphs a/b/c or will exclude paragraphs x/y/z – whichever is shorter.

PeterParslow commented 7 years ago

This is an old criticism of ISO/BSI's business model. As agreed earlier, the 'open' in 'open standard' is more about the process of creating the standard (at least, in the definition which the British Government uses!). Anyway, I believe it is quite possible to comment on ISO 8601 based on experience of using subsets of it in existing systems. Or the subset which is public at W3C, IETF, https://www.iso.org/iso-8601-date-and-time-format.html or even on Wikipedia. And if you want/need to read more, many city/county libraries have a full set of BSI standards (although they seem to have disappeared from Hants & Somerset library searches - cuts, I suspect!) As a last technical thought, we may well be happy to settle on the W3C or IETF profiles, which would avoid the concern about 'fair use'. Now for an about face: I do think that this is one ISO standard so fundamental that they should consider making it publically available (like those at http://standards.iso.org/ittf/PubliclyAvailableStandards/index.html).

philarcher commented 7 years ago

I've only ever come across references, direct or indirect, to ISO 8601. Our own widely used xsd datatypes are built on it (https://www.w3.org/TR/xmlschema11-2/#dateTime for example) as well as the old profile mentioned in earlier messages (https://www.w3.org/TR/NOTE-datetime). HTML5 tells you how to parse dates/times in gory detail at https://www.w3.org/TR/html/infrastructure.html#dates-and-times.

RFC 3339 is itself a profile of 8601 so in terms of date and time representation, ISO 8601 really must be the bedrock - but profiles are useful.

One might be tempted to say that when recording data, always use 8601, but if it's being displayed, more human-readable forms are clearly better. But what about someone working on an Excel spreadsheet? I just checked - what you see is what ends up in the CSV so, sadly, we can't think in terms of Excel displaying one format but storing another. This is a bit of a killer since Excel is so widely used and liked.

Incidentally, 'GMT' is seen ambiguous internationally as many people, incorrectly, read that as 'UK time' and so, right now, they would say it's 17:57 GMT (it's 17:57 BST of course). And when in Portugal last year I asked what the Portuguese call what we call BST I got some funny looks but the answer seemed to be 'the time, dumbo' (I'm sure there's more to it than that).

RFC 3339 has a section on Rarely Used Options which is interesting. Something like xsd:gMonthDay is fully defined as an XML datatype but would anyone trust general software to understand it?

Which leads me to think in terms - as has been said by others - that a profile is needed. Current profiles generally don't help the readability problem. The NHS guidance about entering dates and times is useful in this regard. The HTML5

And, as Peter has pointed out, ISO 8601 has nothing to say about non-Gregorian dates. For things like geological epochs, relative times, Chinese calendars and more, you need the OWL Time Ontology https://www.w3.org/TR/owl-time/. I know TNA has long experience of working with Regnal years - they'd be the folks to ask about that.

@haydavie - forgive me for being lazy but does the NHS material cover all of:

If not, maybe GDS should create such a profile? Unless someone wants to set up and run a W3C Community Group on this (that might then feed into the likely to launch soon DXWG https://www.w3.org/2017/dxwg/charter)

galund commented 7 years ago

Worth mentioning at this point, perhaps, the existing GDS guidance on date entry in HTML forms. I didn't remember seeing it until I was in a presentation about it today, but I think it's pretty sound (despite not recommending the same thing as the NHS guidance - which is why I think we need to consider both here):

https://www.gov.uk/service-manual/design/dates

It's not directly relevant to the question of embedding data in other formats, of course.

Lawrence-G commented 7 years ago

Like galund discovering the GDS guidance on date entry, I recently became aware of the Cabinet Office information management guide's entry on dates (published on the CO intranet) -

"How do I name my documents? A document name should consist of a date, a description of what the document is, and its title."...

....Dates must be numeric in the format: yyyy-mm-dd (ISO 8601:2004, and BS EN 28601: 1992)"

This format strikes a balance between human and machine readability in naming files.

Lawrence-G commented 7 years ago

While researching adoption of ISO 8601 in Europe I found the following - For datasets in the 34 environmental themes that fall under the INSPIRE directive the default temporal reference system is the Gregorian calendar, with dates expressed in accordance with ISO 8601.

Some sources indicate adoption by CEN as EN 28601 e.g. https://en.wikipedia.org/wiki/Date_and_time_notation_in_Europe though I wasn't able to find any reference to the standards on the CEN website ??

Lawrence-G commented 7 years ago

ISO 8601:2004: proposal version 1

Challenge: Date-Times and Time-stamps

Introduction

This proposal is to use a standard for dates and times where machine readability is the main concern. This includes in APIs and other data exchanges such as metadata in documents and websites. In addition when the ability to date-sort is important - for example in filenames.

Excluded from this proposal is the representation of dates and time intended to be read or inputted exclusively by humans for example within the text of a document.

The challenge owner suggested ISO 8601:2004 An international standard for the exchange of date and time information in an unambiguous manner.

For example: 2017-05-16T10:30:56+01:00

User need approach

The user need identified by this proposal is to avoid confusion in the exchange of date and time information. By adopting ISO 8601 as the standard inconsistency in recording the date and time will be reduced.

ISO 8601 is currently adopted by the following:

ISO 8601 is cited in standards already accepted by the Open Standards Board - iCAL, vCard, ODF, OCDS, IATI, JobPosting, MAIT, HTML5.

Every major programming language has ISO 8601 support - both for generating and parsing

ISO 8601 and profiles of the standard allow flexibility in the precision of date and time recording to align with user needs.

Freely available guidance

This proposal notes that the ISO standard or its BSI equivalent BS ISO 8601:2004 are not available free of charge.

There are available profiles of the standard, including RFC 3339, which uses a subset of ISO 8601 and is intended for use in internet technologies for example in APIs.

W3C date and time format is also an ISO 8601 profile that is written to simplify its use in WWW standards.

These and other published guidance on the standard are freely available to implementers.

Achieving the expected benefits

Functional needs

For the functional needs identified in this proposal, ISO 8601 or a subset of the standard is to be used when a date or date and time is being recorded or exchanged in an IT system

For example: 2017-05-16T10:30:56+01:00

The date and time is represented by listing the elements in descending order of size. Starting with the largest (year) first and moving through months, days, hours, minutes, seconds, milliseconds, and microseconds. A UTC offset may be present.

Time if required, is recorded in 24-hour format with 2 digits per element. The extended format uses a ‘:’ separator. Reduced accuracy time formats are supported.

Time is assumed to be local time. UTC and UTC with local offset are supported.

Other steps to achieving interoperability

RFC 3339 is a date and time format for use in Internet protocols that is a profile of the ISO 8601 standard for representation of dates and times using the Gregorian calendar.

The main focus for RFC 3339 is timestamps for Internet protocol events whereas ISO 8601 has a wider scope for use. RFC 3339 aims for simplicity to achieve maximum interoperability and may be preferable for some applications. RFC 3339 may be suited to some particular use cases and should also be recommended for use in this proposal.

W3C date and time format is also an ISO 8601 profile that is written to simplify its use in WWW standards.

Proposal

For timestamps used in data exchange, and other machine readable uses:

Human readable timestamps remain unaffected, but best practice will be to include an unambiguous timestamp within the document.

oughnic commented 7 years ago

@Lawrence-G; @edent The user need is driven by cultural differences. Computers are not affected by cultural differences, so the proposal to address machine interoperability without addressing human interoperability fails to address the user need.

Inconsistent recording of dates and times, causing confusion especially with the international exchange of computer documents, where there are different cultural practices. For example, confusion of UK and USA dates, such as 9/11/2001, will be reduced.

I suggest the proposal is changed to

PeterParslow commented 7 years ago

I support the idea of a separate guide to providing address information for human readers. The NHS guidance suggest by haydavie may be appropriate. ISO 8601 is not ambiguous for this, but is not commonly understood (it's "a bit techie")

I agree with the proposal, as far as it goes - for machine to machine interchange.

psd commented 7 years ago

Note our standard for exchanging calendar events, RFC5545, cites ISO8601 for calendar appointments.

For precise timestamps made by machines such as recording entries in a register, logs and data sourced from telemetry, registers uses RFC3339 which is a profile of ISO8601.

For dates recorded by humans registers have a date type which is ISO8601, but may have less precision for recording dates which are only know to the nearest day "1998-09-29", month "1998-09" or year "1998". A consumer needing a precise date is then at liberty to round up, down or take a mean.

When presenting date to humans in web pages and other documents, government services should follow the GOV.UK service manual and style guide. I think asserting a standard here could be harmful, as it's the job of a service to present content in whatever way makes sense to its users, who may be in a different locale.

Note whilst GOV.UK pages include dates in friendly human format, they also include HTML5 datetime values in the time element, which is ISO8601 format with an optional time value. Try view-source on a GOV.UK publication page and look at the history section.

Lawrence-G commented 7 years ago

As part of our evaluation of ISO 8601:2004 these are the assessment questions and answers.

Applicability

Q1. Does the formal specification address and facilitate interoperability between public administrations? Yes The interoperability of (IT) systems will be improved employing a standard for date and time information and representation.

Q2. Does the formal specification address and facilitate the development of information or IT systems in government? Yes Where a system requires the unambiguous recording of date and time information.

Q3. Are the functional and non-functional requirements for the use and implementation of the formal specification appropriately detailed? Yes It’s a fairly limited set of requirements to represent time and dates

Q4. Is the formal specification applicable and extensible for implementations in different domains? Yes The date and time format is applicable to any domain

Q5. Is the formal specification largely independent from products of single providers (either open source or proprietary)? Yes Not tied to a particular product

Q6. Is the formal specification largely independent from specific platforms? Yes Platform independent

Q7. Has the standard been written such that its realisation can be/is demonstrated using more than one technology (e.g. XML and JSON)? Yes Date and time information given in ISO 8601 format can be used in various technologies including JSON, XML , HTML

Maturity

Q8. Has the formal specification been sufficiently developed and in existence for a period to overcome most of its initial problems? Yes Current version was published in 2004 the standard’s first version was published in 1988

Q9. Are there existing or planned mechanisms to assess conformity of the implementations of the formal specification (e.g. conformity tests, certifications, plugfests etc)? Yes There are online resources and applications for several programming languages to check conformity. Example

Q10. Has the formal specification sufficient detail, consistency and completeness for the use and development of products? Yes The specification has a number of options for the level of accuracy in representing date and time and gives enough information on the use of the standard in practice.

Q11. Does the formal specification provide available implementation guidelines and documentation for the implementation of products?

Not applicable Implementation is a simple code format used alongside other standards. The standard documentation contains examples of use.

Q12. Does the formal specification provide a reference (or open source) implementation?

Not applicable There are a number of tools available online to generate or validate iso 8601 date-time stamps. Example

Q13. Does the formal specification address backward compatibility with previous versions? Yes The basic representation of date and time in the standards remains compatible. Minor additions are made to the basic format.

Q14. Have the underlying technologies for implementing the formal specification been proven, stable and clearly defined? Yes

Openness

Q15. Is information on the terms and policies for the establishment and operation of the standardisation organisation publicly available? Yes https://www.iso.org/structure.html

Q16. Is participation in the creation process of the formal specification open to all relevant stakeholders (e.g. organisations, companies or individuals)? Yes http://www.iso.org/iso/home/standards_development/who-develops-iso-standards.htm Note: process is not open to individuals.

Q17. Is information on the standardisation process publicly available? Yes https://www.iso.org/developing-standards.html

Q19. Information on the decision making process for approving formal specifications is publicly available? Yes http://www.iso.org/iso/home/standards_development.htm

Q20. Are the formal specifications approved in a decision making process which aims at reaching consensus? Yes ISO standards are based on consensus

Q21. Are the formal specifications reviewed using a formal review process with all relevant external stakeholders (e.g. public consultation)? Yes the standard is reviewed by the ISO Maintenance Agency

Q22. All relevant stakeholders can formally appeal or raise objections to the development and approval of formal specifications? Yes During the commenting and balloting stage of the standards process

Q23. Relevant documentation of the development and approval process of formal specifications is publicly available (e.g. preliminary results, committee meeting notes)? No We have not found any published for this standard http://isotc.iso.org/livelink/livelink?func=ll&objId=16315607&objAction=browse&viewType=1

Q24. Is the documentation of the formal specification publicly available for implementation and use at zero or low cost? No The documentation costs CHF138 from ISO or £190.00 from BSI. There are however closely related standards ,profiles and guides to the standard that are freely available.

Intellectual property rights

Q25. Is the documentation of the IPR for formal specifications publicly available (is there a clear and complete set of licence terms)? Yes Patent declarations available here -https://www.iso.org/iso-standards-and-patents.html Customer license - https://www.iso.org/terms-conditions-licence-agreement.html

Q26. Is the formal specification licensed on a royalty-free basis? Yes There is no charge for implementation of the standard.

Market support

Q27. Has the formal specification been used for different implementations by different vendors/suppliers? Yes Widely used standard within other standards.

Q28. Has the formal specification been used in different industries, business sectors or functions? Yes

Q29. Has interoperability been demonstrated across different implementations by different vendors/suppliers? Yes No examples but there should not be a problem with this standard

Q30. Do the products that implement the formal specification have a significant market share of adoption? Not Applicable Not applicable to any particular products

Q31. Do the products that implement the formal specification target a broad spectrum of end-uses? Not Applicable The standard is found in a large number of applications but not specific products

Q32. Has the formal specification a strong support from different interest groups? Yes Many organizations have adopted the standard- W3C ,, meteorological, maritime https://www.w3.org/TR/html51/infrastructure.html#dates-and-times http://www.metoffice.gov.uk/datapoint/product/uk-hourly-site-specific-observations/detailed-documentation http://www.oceandocs.org/handle/1834/4467

Potential

Q33. Is there evidence that the adoption of the formal specification supports improving efficiency and effectiveness of organisational process? Yes The aim of the standard is to remove ambiguity in date-time stamps this is expected to improve effectiveness and efficiency in systems that rely on date and time information.

Q34. Is there evidence that the adoption of the formal specification makes it easier to migrate between different solutions from different providers? Yes There should be no problems moving providers

Q35. Is there evidence that the adoption of the formal specification positively impacts the environment? Not Applicable Not applicable. No impact to environment expected.

Q36. Is there evidence that the adoption of the formal specification positively impacts financial costs? No No evidence found of impact on costs

Q37. Is there evidence that the adoption of the formal specification positively impacts security? No No impact on security. Though RFC3339 has a note on systems preferring not to allow local time to reveal location information.

Q38. Is there evidence that the adoption of the formal specification can be implemented alongside enterprise security technologies? No No evidence found but this standard can be implemented alongside any technologies

Q39. Is there evidence that the adoption of the formal specification positively impacts privacy? No No impact on privacy

Q40. Is the formal specification largely compatible with related (not alternative) formal specifications in the same area of application? Yes RFC 3339 https://tools.ietf.org/html/rfc3339 and W3C date format https://www.w3.org/International/questions/qa-date-format

Q41. Is there evidence that the adoption of the formal specification positively impacts the accessibility and inclusion? No No impact on accessibility and inclusion

Q42. Does the formal specification have a defined maintenance organisation? Yes ISO TC 154 https://www.iso.org/committee/53186.html

Q43. Does the maintenance organisation for the formal specification have sufficient finances and resources to be sure of freedom from short- to medium-term threats? Yes

Q44. Does the maintenance organisation have a public statement on intention to transfer responsibility for maintenance of the formal specification if the organisation were no longer able to continue? No National standards orgs e.g BSI we assume would be able to maintain

Q45. Does the formal specification have a defined maintenance and support process? Does the formal specification have a defined policy for version management? Yes See ISO TC 154 https://www.iso.org/committee/53186.html

Coherence

Q46. Is this an existing European standard or an identified technical specification in Europe? (Note: CEN, CENELEC and ETSI are the European standards bodies. Technical specifications provided by organisations other than CEN, CENELEC or ETSI can be under consideration to become a European standard or an identified technical specification in Europe for example through the Multi Stakeholder Platform.)

ISO 8601 is the temporal standard in the [INSPIRE directive](https://data.gov.uk/inspire for spatial data

Q47. Does this specification or standard cover an area different from those already identified or currently under consideration as an identified European standard or specification?

Yes

Lawrence-G commented 7 years ago

ISO 8601:2004: Proposal version 2

Challenge: Date-Times and Time-stamps

Introduction

This proposal is to use a standard for dates and times where machine readability is the main concern. This includes in APIs and other data exchanges such as metadata in documents and websites. In addition when the ability to date-sort is important - for example in filenames.

Excluded from this proposal is the representation of dates and time intended to be read or inputted exclusively by humans for example within the text of a document.

The challenge owner suggested ISO 8601:2004 An international standard for the exchange of date and time information in an unambiguous manner.

For example: 2017-05-16T10:30:56+01:00

User need approach

The user need identified by this proposal is to avoid confusion in the exchange of date and time information. By adopting ISO 8601 as the standard inconsistency in recording the date and time will be reduced.

ISO 8601 is currently mandated by the following organisations:

ISO 8601 is cited in standards already accepted by the Open Standards Board - iCAL, vCard, ODF, OCDS, IATI, JobPosting, MAIT, HTML5.

Every major programming language has ISO 8601 support - both for generating and parsing.

ISO 8601 and profiles of the standard allow flexibility in the precision of date and time recording to align with user needs.

Free Alternatives

This proposal notes that the ISO standard or its BSI equivalent BS ISO 8601:2004 are not available free of charge.

There are available profiles of the standard, including RFC 3339, and the W3C date and time format note. These and other published guidance on the standard are freely available to implementers.

Achieving the expected benefits

Functional needs

For the functional needs identified in this proposal, ISO 8601 or a subset of the standard is to be used when a date or date and time is being recorded or exchanged in an IT system.

For example 2017-05-16T10:30:56+01:00 This shows the instant of the 16 May 2017 at thirty minutes and fifty-six seconds past ten o'clock, British summer time, one hour ahead of Coordinated Universal Time (UTC)

The date and time is represented by listing the elements in descending order of size. Starting with the largest (year) first and moving through months, days, hours, minutes, seconds, milliseconds, and microseconds. A Coordinated Universal Time (UTC) offset may be present.

Values have a fixed number of digits that must be padded with leading zeros.

Different levels of accuracy are supported by the standard, with the minimum level being four digits representing the year. With a reduced accuracy date, hyphen separators are used to avoid confusion e.g. YYYY-MM.

Time if required, is separated from the date by a ‘T’ character and is recorded in 24-hour format with 2 digits per element. The extended format uses a ‘:’ separator between hours and minutes. Reduced accuracy time formats are supported.

Time is assumed to be local time. UTC and UTC with local offset is supported.

Interoperability

RFC 3339

RFC 3339 simplifies ISO 8601 for use in Internet protocols by omitting the little-used options. RFC 3339 uses the ISO extended format with hyphens. Most fields are mandatory, The exception being fractions of a second the use of which is optional.

RFC 3339 uses a four digit representation of years. ISO 8601 allows either four or two digit years.

RFC 3339 allows the separator between date and time to be a space character instead of a ‘T’ to aid readability ISO 8601 does not have this provision. In RFC 3339 if the time in UTC is known, but the offset to local time is unknown, this can be represented with an offset of "-00:00". This is not part of ISO 8601.

W3C Note on Date and Time Formats

W3C date and time format is an ISO 8601 profile that is written to simplify its use in WWW standards.

The W3C profile allows six levels of detail in the date and time format from the year only to A complete date plus hours, minutes, seconds and a decimal fraction of a second in ISO extended format. Years are expressed as four digits.

Time is expressed in UTC or in local time with a time zone offset in hours and minutes. Fractions of a second if required are supported. Fractions of hours and minutes permitted in ISO 8601 are not supported by this profile

Human readable formats

Human readable timestamps remain unaffected, but best practice will be to include an unambiguous time stamp within the document. The GOV.UK Service Manual has guidance on the use of dates in digital service to give a consistent user experience.

Proposal

For timestamps used in data exchange, and other machine readable uses: the standard will be ISO 8601. RFC 3339 and W3C Date Time are acceptable subsets which are free and open.

PaulDav commented 7 years ago

Can you clarify the scope? - the email that pointed me here says

"The suggestion is to use ISO 8601 for the time and date in online documents, databases, for filenames and online references."

... but the Challenge Descripton says

"By adopting ISO8601 notation for dates and times in online documents, spreadsheets, databases and for filenames and online references such as URLs, greater interoperability will be achieved at less cost and with less confusion."

'Spreadsheet' is missing from the 1st definition.

I am defining a standard for Local Authorities in England to publish data, as a csv file; and it contains a couple of date fields. Many councils are likely to prepare this information using Excel and then save-as csv. When I try that, I find that entering a yyyy-mm-dd type date into a cell, is immediately represented as dd/mm/yyyy. Saving as csv then saves the dd/mm/yyyy content.

I can format the row as 'text', which then retains the yyyy-mm-dd content, and saves OK as csv. But, if I re-open the csv by double-clicking the file, the column formatting is not retained, and the dates are presented as dd/mm/yyyy, and then saved that way.

Many councils are likely to prepare their data using Excel and then save as csv. Even where data is exported and transformed using another tool, they are likely to check their work using Excel. As Excel is the dominant tool for preparing this type of data, there will be practical difficulties for implementation in spreadsheets which are meant for both human-readable and machine-readable consumption.

Can we either provide some guidance on how to acheive the standard when preparing data using Excel, saved as csv? ... or allow dd/mm/yyyy as cells in a csv file - where that file is meant to be both human-readable and machine-readable.

galund commented 7 years ago

Because of the need to interoperate with systems that assume a US date format, I would say that dd/mm/yyyy in CSV is a particularly bad idea, I'm afraid!

There is a suggestion on the interwebs that typing the date as a formula, i.e. ="2017-04-03" might allow it to survive the transition through CSV. But I'm afraid I don't have a copy of Excel here to test (I might try at home if someone else doesn't beat me to it.)

I think the general advice would have to be to treat Excel CSV purely as an output or reporting format - i.e. to always go back to some Excel version (or other source format) rather than editing a CSV. But obviously that's easy to say and possibly very difficult to integrate with some business processes.

Lawrence-G commented 7 years ago

In reply to PaulDav, some clarification, the text in the email (to open standards community google group) missed spreadsheets in the description of the proposal but they are a type of electronic document. Your comment raises the point that there may be a problem when identifying when ‘machine readability is the main concern’ I agree with galund's comment regarding spreadsheet software.

I don’t really want to start a discussion on a particular feature of Excel here, but I have had a chance to try (on Excel 2010) and it’s possible, in the format cell menu to set dates to yyyy-mm-dd and save as CSV. Checked with a text editor and the format is preserved. How the dates are displayed on opening the CSV depends on a programs regional and user preference setting and whether they are recognised as a date. This aids readability for the user.

edent commented 7 years ago

Some thoughts from the W3C about their use of a subset is 8601. https://github.com/w3c/html/issues/413

Lawrence-G commented 7 years ago

Based on the discussion here, we have written an evaluation for the Open Standards Board of the standard proposal.

Title: Evaluation questions for Date-Times and Time-stamps Draft Profile

Standard: ISO 8601:2004

Context

What needs does the standards profile meet? A notation for dates and times in online documents, filenames and online references such as URLs and API responses to achieve greater interoperability with less confusion.

Which organisations or functional areas should refer to this Standards Profile?

All government bodies described within the scope of the Open Standards Principles should apply this standards profile, subject to the implementation dependencies described below if agreed by the Open Standards Board.

Recommendation

What is the Standards Panel’s recommendation to the Open Standards Board? - Describe which stakeholder groups and outcomes the Standards Profile meets and how.

The Standards Panel recommends ISO 8601 for date-time stamps in APIs and other data exchanges such as metadata in documents and websites.

In addition when the ability to date-sort is important for example in filenames.

This proposal if agreed by the Open Standards Board, should be compulsory for use in government.

Why does the Panel consider this is the most effective course of action? - Describe the scope.

The Standards Panel considers that the case for ISO 8601 requested by the challenge has been made. It meets the needs described in the challenge and meets the criteria for open standards.

Assessment Overview

Summary of the assessment - What are the pros and cons relating to applicability, maturity, openness, intellectual property rights, market support, potential and coherence discovered during the assessment?

Applicability: ISO 8601 was deemed applicable where machine readability is the main concern. The standard supports interoperability, recording date and time stamps unambiguously in documents, data, filenames and references such as URLs and API responses. Maturity: This is a mature standard Openness and IPR: Some concerns were raised over the price of the standard documentation but this is not thought to be a barrier to use as free profiles and guidance available. There is no charge for the use of the standard. Market support and potential: ISO 8601 is referenced in many other standards and is not tied to particular products or services. The majority of modern programming languages have native support for ISO8601. Coherence: This is an international standard. Alternatives considered - What other proposals and standards were considered and why are these not being recommended? The other leading date-time stamp standards are profiles of ISO 8601 e.g IETF RTF 3339 these are suitable for use in their intended domains, whereas ISO 8601 offers the flexibility for use in many applications that exchange the date and time.

Implementation

What effect would implementation of this Standards Profile have on service delivery?

By adopting a standard for time-date stamps ambiguity will be reduced in the recording and exchange of data that require a date-time stamp Our services will be using an international standard reducing possible errors when exporting/importing data from other countries. Archival research regarding dates of publication will become unambiguous.

How does this approach deal with any backwards compatibility issues?

No issues with backwards compatibility were raised by the panel in conversations on GitHub.

What might be on the horizon (e.g. are there any threats relating to this approach or the associated standards)?

A new version of the standard will be published in 2018. This new version will be in two parts -Basic and Extended rules. Part one is largely the same as the 2004 version with the following changes.

Part two adds extensions such as uncertain and approximate dates. The current version will continue to be relevant and is complete enough to serve most purposes.

What are the benefits or opportunities relating to this approach or the associated standards (economic, social and environmental)?

By adopting ISO 8601 the international standard for date-time stamps several benefits will be released.

When will implementation begin and when is it likely that it will be completed?

Are there any non-technical barriers that remain which will need to be addressed before successful implementation could be achieved (e.g. legal, organisational)?

Implementation can be started immediately that the standard is mandated/recommended

Barriers were identified by the Panel members on GitHub these were:-

The standard profile should address these concerns:

What trials have been undertaken?

Describe any reference implementations that have been built or tested, and pilot projects or any plugfests etc.

A widely used standard that is used in other standards already mandated by the Board e.g iCAL, vCard, ODF, OCDS, IATI, JobPosting, MAIT, HTML5.

Adopted by the Cabinet Office as the standard for date stamps in filenames.

Process

The people involved in the development of the proposal and the assessment of the standards Challenge owners: Terence Eden, Lawrence Greenwood

Members of the Standards Board and Panel involved in the evaluation of this Standards Profile

Nicholas Oughtibridge Peter Parslow Giuseppe Sollazzo Paul Davidson Paul Downey

Other contributors on GitHub George Lund Davie Hay

What date should this Standards Profile be reviewed?

Taking into account any anticipated changes in technology or standards.

2019 to assess the impact of the new version once released.

Was it necessary to notify the European Commission?

If so, what stage has been reached; is the outcome known? No

torgo commented 6 years ago

This revised proposal looks good to me. I have always been a big fan of ISO-8601.

puntofisso commented 6 years ago

+1 from me too. I wonder (although I'm aware this is beyond the remits of the panel and board) if a set of examples could be released in the style of Gov,Uk's service manual, in order to advocate for and simplify adoption?

johnlsheridan commented 6 years ago

+1 from me too. The revised proposal is very clear and has my full support. As discussed above, whilst ISO-8601 is not open access, the W3C date and time, and IETF 3339 profiles are freely available, as are various other sources explaining the standard (Wikipedia).

Lawrence-G commented 6 years ago

The date-times and time-stamps guidance has been published - here