Luxon does not support U+2010 (HYPHEN) as a date separator in ISO8601 format

[ddolcimascolo]

Describe the bug Luxon does not support U+2010 (HYPHEN) character as a date separator in ISO8601 format. But to be honest, this is the first time I've seen dates formatted using this character. I'm not even sure luxon should support that but I wanted your feedback.

To Reproduce

> require('luxon').DateTime.fromISO('2015\u201010\u201026T23:50:07Z')
DateTime {
  ts: 1634291908319,
  _zone: SystemZone {},
  loc: Locale {
    locale: 'fr-FR',
    numberingSystem: null,
    outputCalendar: null,
    intl: 'fr-FR',
    weekdaysCache: { format: {}, standalone: {} },
    monthsCache: { format: {}, standalone: {} },
    meridiemCache: null,
    eraCache: {},
    specifiedLocale: null,
    fastNumbersCached: null
  },
  invalid: Invalid {
    reason: 'unparsable',
    explanation: `the input "2015‐10‐26T23:50:07Z" can't be parsed as ISO 8601`
  },
  weekData: null,
  c: null,
  o: null,
  isLuxonDateTime: true
}

Actual vs Expected behavior I'm expecting the date to be parsed correctly.

Desktop (please complete the following information):

• OS: [e.g. iOS] Ubuntu 20.04
• Luxon version [e.g. 1.25.0] 2.0.2
• Your timezone [e.g. "America/New_York"] Europe/Paris

Additional context These dates I'm getting come from a hardware device exposing data through a REST API. The API returns utf-8 JSON with these dates in a few fields. Sample below

[
  {
    "component_name": "hg1",
    "component_type": "purehgroup",
    "details": "‐‐remhostlist h1,h2 ",
    "event": "setattr",
    "id": 218791,
    "opened": "2015‐10‐26T23:50:07Z",
    "user": "os76",
    "current_severity": "warning",
    "code": 42
  },
  {
    "component_name": "hg2",
    "component_type": "purehgroup",
    "details": "‐‐remhostlist h1,h2 ",
    "event": "setattr",
    "id": 218792,
    "opened": "2015‐10‐26T23:50:07Z",
    "user": "os76",
    "current_severity": "critical",
    "code": 84
  }
]

[icambron]

I seem to have misplaced my copy of the ISO spec. But the basic answer is: if it's in the spec, I'd be happy to add support for it. If it's not, I won't. So if you're able to track that down, we can go from there

[ddolcimascolo]

That's not clear to me per https://datatracker.ietf.org/doc/html/rfc3339#section-5

The spec mentions "-" but doesn't specify it as dash, minus, hyphen or whatever

[EcksDy]

I'd like to resurface this as I've stumbled upon this strange behaviour where luxon can't parse its own ISO strings:

  const time = DateTime.fromSeconds(0).toUTC().toISO();
  console.log(time);
  const duration = Duration.fromISO(time);
  console.log(duration);

This is the output:

1970-01-01T00:00:00.000Z
Duration {
  values: undefined,
  loc: Locale {
    locale: 'en-US',
    numberingSystem: null,
    outputCalendar: null,
    intl: 'en-US',
    weekdaysCache: { format: {}, standalone: {} },
    monthsCache: { format: {}, standalone: {} },
    meridiemCache: null,
    eraCache: {},
    specifiedLocale: null,
    fastNumbersCached: null
  },
  conversionAccuracy: 'casual',
  invalid: Invalid {
    reason: 'unparsable',
    explanation: `the input "1970-01-01T00:00:00.000Z" can't be parsed as ISO 8601`
  },
  matrix: {
    years: {
      quarters: 4,
      months: 12,
      weeks: 52,
      days: 365,
      hours: 8760,
      minutes: 525600,
      seconds: 31536000,
      milliseconds: 31536000000
    },
    quarters: {
      months: 3,
      weeks: 13,
      days: 91,
      hours: 2184,
      minutes: 131040,
      seconds: 7862400,
      milliseconds: 7862400000
    },
    months: {
      weeks: 4,
      days: 30,
      hours: 720,
      minutes: 43200,
      seconds: 2592000,
      milliseconds: 2592000000
    },
    weeks: {
      days: 7,
      hours: 168,
      minutes: 10080,
      seconds: 604800,
      milliseconds: 604800000
    },
    days: {
      hours: 24,
      minutes: 1440,
      seconds: 86400,
      milliseconds: 86400000
    },
    hours: { minutes: 60, seconds: 3600, milliseconds: 3600000 },
    minutes: { seconds: 60, milliseconds: 60000 },
    seconds: { milliseconds: 1000 }
  },
  isLuxonDuration: true
}

[icambron]

@EcksDy You're parsing an ISO DateTime string as a duration, which has a different ISO 8601 format. You want DateTime.fromISO, not Duration.fromISO

[kohenkatz]

So if you're able to track that down

I stumbled upon this today, and decided to do a little digging. The Library of Congress has a PDF of ISO 8601-1:2016, and here is what it says:

Section 3.4 Characters used in the representations

The representations specified in this International Standard make use of graphic characters as specified in 3.4. Note that, except for "hyphen", "minus" and "plus-minus", these characters are part of the ISO/IEC 646 repertoire.

In an environment where use is made of a character repertoire based on ISO/IEC 646, "hyphen" and "minus" are both mapped onto "hyphen-minus". Representations with a "plus-minus" shall only be used in such environment if the interchange repertoire includes "plus-minus".

So I could see both ways of interpreting this:

1. On one hand, UTF-8 has a hyphen character, so the first paragraph says to use it.
2. On the other hand, UTF-8 is part of ISO/IEC 10646, which is based on ISO/IEC 646, so maybe "mapped onto hyphen-minus" in the second paragraph should mean that we should pretend that separate hyphen and minus characters don't exist.