Open icefoxen opened 6 years ago
I like this idea. However, instead of ISO 8601, I would say that dates should be converted to UTC and represented as strings in Internet Date/Time Format as defined by RFC 3339.
ISO 8601 is not a free standard, and it supports a very broad range of date and time formats. RFC 3339 defines an ISO 8601 profile that provides a high level of interoperability.
I added the following section to a new v1.1.0 branch.
It is recommended that JSON5 generators convert date and time values to Coordinated Universal Time (UTC) and represent them as strings in internet format as defined by RFC 3339 in the interest of interoperability.
Example 1 (Informative)
{
billenium: '2001-09-09T01:46:40Z',
}
Alternatively, JSON5 generators may represent dates and times as Unix time, which is the number of seconds that have elapsed since 00:00:00 Coordinated Universal Time (UTC), Thursday, 1 January 1970, minus the number of leap seconds that have taken place since then.
Example 2 (Informative)
{
json5Birthday: 1338150759534,
}
If a JSON5 parser expects a value to be a date or time, it may attempt to convert the value from one of the earlier defined formats to a date or time.
Most JSON encoders/stringify respect objects with a .toJSON()
method and use that for encoding... Date has such a method, and it encodes to ISO-8601 format. It may not be part of the JSON standard, but it has effectively become part of ECMA/W3C standard.
At this point, the best bet would be to support automatic hydration of full ISO strings that match a .toJSON()
output to dates. Extended formats should be possible, but wouldn't want to create a specification for this that effectively breaks the internet, so to speak.
I would propose that if new Date
functionality is really desired, particularly in encoding/decoding with JSON5, then a Date2
object/class be created in order to facilitate that functionality. Cleaning up timezone interactions and conversions should also probably be a priority. I'd suggest looking to moment and moment-tz for inspiration. There have been discussions with tc39 on this subject.
@tracker1 I'm hesitant to have parsers automatically convert date-like strings to dates. Strings would no longer be just strings, but they'd also be maybe-dates. And if the platform doesn't have support for fractions of seconds, then the string '2018-03-16T22:36:13.713Z'
would lose information if forced into a less precise date construct.
The nature of strings is that they can be very precise if need be. For example, the number 1e-1000
may get rounded to zero, but the string '1e-1000'
can't lose precision.
Developers are welcome to write parsers that convert strings to dates, and in fact I've tried the idea myself, but I don't think it should be the standard.
I'm fine with not doing anything with it... just didn't want to add something to the spec regarding dates, as date encoding already has a defacto implementation.
If you have a defacto implementation, other people are just going to implement it wrong. If you have a standard that people can test against, they know how to implement it right. This is what standards are for.
The only safe solution, imo, is to extend the spec to have some form of tagged literals, as was proposed on the original discussion: https://github.com/json5/json5/issues/3#issuecomment-342050877. Besides, this solution would allow supporting any custom types of literals.
Probably, the best bet would be to use the same syntax as the new ECMAScript tagged template literals: https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Template_literals.
Note that supported tags and their evaluation functions would always need to be registered with the parser, thus allowing full control of supported types of literals and simultaneously ensuring the safety of the solution.
My main problems with dates are inside API calls: Client code has to know that specific fields in JSON structure returned by the API are dates and need to be parsed manually over and over again for each API call that returns dates. That is pretty common in all web apps.
I was thinking about allowing something like this in JSON:
{
firstName: 'John',
lastName: 'Smith',
birthday: new Date('1986-08-04'),
createdAt: new Date('2019-07-24T13:37:34.752Z'),
}
If the spec is extended, I'd prefer bogdan's solution... though, would probably be somewhat better to remain compatible as a string.
// USE ASCII Characters for kind of what they are meant for.
const DLE = String.fromCharCode(16); // Data Link Escape
const SOH = String.fromCharCode(1); // start of heading
const STX = String.fromCharCode(2); // start ot text
const ETX = String.fromCharCode(3); // end of text
const EOT = String.fromCharCode(4); // end of transmission
const DLE = String.fromCharCode(16); // Data Link Escape
An encoded data record should start with:
// Encoded record ...
`${DLE}${SOH}TYPE${STX}DATA${ETX}${EOT}${DLE}`
If the string starts with DLE + SOH
and ends with ETX + EOT + DLE
then it can be checked for an encoding. Further split on STX
into header and body parts.
In this case TYPE
could be "DATE/RFC3339"
and the DATA
would be a properly encoded string.
In this way, we could use existing escape characters for encoded data, and if there's a supported decoder, it can be added and handle any encoded complex type, while still complying with original JSON.
For more complex data types, there's also...
const FS = String.fromCharCode(28); // file separator
const GS = String.fromCharCode(29); // group separator
const RS = String.fromCharCode(30); // record separator
const US = String.fromCharCode(31); // unit separator
https://gist.github.com/tracker1/84fd0c1a31b3d17b176495f37b3431ba
I don't think it's a good idea to start treating strings as maybe-dates, especially when it breaks backward compatibility.
On another note, I think putting non-printable characters in strings would make JSON5 less human readable than JSON.
In my opinion, platforms and languages have too many differences when it comes to date/time storage, time-zone conversions, parsing, formatting, precision, etc. - and also, these are often subject to change on various systems as they evolve over time, adding/removing time-zone, changing precision, etc.
I think that specifying anything about dates is probably unnecessary, as most systems already specify how they want dates/timestamps represented - and usually with good reason, as it has to match their back-end database or other systems, so it's most likely safer to stay away from this topic entirely.
I disagree strongly with @mindplay-dk . Anyone who is very particular about how their dates are interpretered will always have the option of encoding them as strings or objects.
But 99.99% of the use cases are already handled in rfc3339. People have already spent a lot of time and effort figuring out wire syntaxes for these things and we depend on those wire syntaxes literally every time we use the Web. These are solved problems.
With respect to syntax: I think that a string prefix is the most backwards-compatible and yet recognizable thing. Something like:
"jsondate:2009-01-02"
People could turn on date handling in their parsers if they want them auto-parsed or turn it off if they prefer to leave them as strings.
If parsing strings as maybe-dates is a parser option, then I don't think it belongs in the syntax.
Honestly, I don't see the problem in a generic solution, such as:
{
"customerId": "abcd",
"orderDate": {"_": "date", "value": "2019-11-25T16:27"}
}
var parseClassMap = {
"date": function parseISODate(spec) {
return new Date(Date.parse(spec.value));
}
};
var data = JSON.parse(jsonString, parseClassMap);
And something inverse for JSON.stringify
.
@jordanbtucker : Okay then, let's drop the idea of having it be a parser option. We can define a unambiguous and backwards-compatible syntax and just move forward.
I could live with either the string-embedded solution or dcleao's solution. The string-embedded version is a lot less "syntax" but dcleao's is more generic for future extensions.
@prescod Please see my comments at https://github.com/json5/json5-spec/issues/4#issuecomment-373868688
Once you start treating strings as maybe-dates, it is no longer deterministic and it breaks backward compatibility.
Hey Jordan: I did read your comment and I thought that my prefix addressed it. The chances that there exists a string prefixed with "jsondate:" in the wild, which is NOT a JSON date seems minuscule to me. The chances of your program being broken by a cosmic ray are much higher. And we could lower it even further by calling the prefix "json5date". Google can find only two bits of code that use that string of characters in any context. And they still wouldn't match because the case isn't the same.
So strings aren't maybe-dates. They are dates if and only if they start with the prefix. Otherwise they are just strings.
But if you want to avoid even that minuscule level of risk, then I would endorse @dcleao's syntax and would volunteer to create a pull request if that's what you suggest.
@prescod What if you actually want a string that starts with "jsondate:" instead of a date?
Read through this whole thread again, here are some thoughts.
Ideally, I would like to have support for a date/time literal - I just don't see how we can remain compatible with JSON and ES5 if we do so.
Following a simple recommendation (as OP proposed) would be easy and natural in JS though.
Dates do convert to RFC 3339 format by default:
new Date().toJSON() // "2019-11-28T17:22:21.490Z"
Likewise, the Date
constructor accurately parses that format without botching the timezone:
new Date("2019-11-28T17:22:21.490Z").toJSON() // "2019-11-28T17:22:21.490Z"
These formats are also natively supported by JSON schema, where they are also encoded as strings.
So on the JS side (and other languages) arguably there is already a popular standard - even if this doesn't automatically give you native Date
objects, it's a pretty good option, and it would be possible to write function that take a JSON schema and data and automatically convert to/from a JS native Date
or other platform-specific date/time representation.
So that would be my suggestion. If we have to do anything for dates/times, it doesn't need to be more than a recommendation, possibly even mentioning JSON schema just to be helpful and to show that this is something we did consider and discuss?
I think I'm with OP on this one.
@jordanbtucker : Thanks for your question. It's a good one. Escaping is going to get ugly.
The problem with:
If a JSON5 <emu-xref href="#parsers">parser</emu-xref> expects a value to be a date
Is that JSON parsers don't expect anything. JSON parsing is usually schema-less (one of the big benefits of JSON!). If I were a JSON5 parser author, I wouldn't know what to do with that line of the specification.
What do you think about explicit type-tagging as proposed above:
{"_": "date", "value": "2019-11-25T16:27"}
Yes, it might "open the can of worms" of every other type like binary or regular expressions. But on the other hand, that's a decision you can make right now as a spec author. You can either decide to "let 100 flowers bloom" in which case JSON5 will be extensible, or you can decide to say that the key "_" (or some other clever identifier) is reserved for this use only, unless another use becomes urgent in the future. It is quite possible that some years in the future, having such an extension mechanism pre-reserved could allow some important differentiator between JSON and JSON5.
But that's all a digression. My main point is that DATE is one of the basic types in every programming language, database and API and it remains a pretty big gap that JSON5 does not have an unambiguous way of handling it. Dragging in JSON Schema or something similarly complex to solve it seems like overkill.
The term "parser" is intentionally left generic. It means anything that consumes JSON5 text. It does not necessarily mean a JSON5 parsing library like https://github.com/json5/json5. That being said, if a JSON5 parser library uses some form of schema, then it would be able to expect a date. That is the intention of that clause.
JSON5 is in wide use, so making any breaking changes requires extra care, as it has the potential to disrupt all existing implementations. It also has the effect of causing fragmentation into versions. If first-class date handling is included in the JSON5 syntax, then we must define a new version of JSON5, and parsers and generators must know which version they are handling.
Adding dates to the syntax, something that almost no other programming language or data interchange format has done, adds more complexity than it's worth. Dates are handled just fine by strings and schema.
What do you think about explicit type-tagging as proposed above:
{"_": "date", "value": "2019-11-25T16:27"}
@prescod This was already commented on - same problem here, it's a breaking change: what if you actually want to encode that value?
The essence of this argument is you can't change the meaning (semantics) of anything that could already exist in JSON data structures - doing so would be a breaking change.
But that's all a digression. My main point is that DATE is one of the basic types in every programming language, database and API and it remains a pretty big gap that JSON5 does not have an unambiguous way of handling it. Dragging in JSON Schema or something similarly complex to solve it seems like overkill.
I understand the desire for a date type (and, ideally, I'd love to have that) but any date encoding or convention that reuses existing JSON types is probably more or less out of the question, since it will inevitably change the meaning of an existing valid JSON expression.
So I'm by no means disagreeing with what you said - there's definitely a gap here, and I would love it if we could think of a solution.
I will mention I-JSON here - if the chosen approach was merely a recommendation, we could point to this. It deals not only with date/time (in the manner we've been discussing) but also with permitted number ranges, recommendations regarding binary data payloads, and a few other details.
Again, those are only recommendations, so here's one idea for encoding dates and possibly other types of values in the future - template literal string syntax:
{
"name": "Rasmus",
"birthday": $date`1975-07-07`
}
This is JavaScript compatible in terms of syntax - you just need to have a template function $date
in scope when you evaluate this:
const $date = value => new Date(value);
console.log({
"name": "Rasmus",
"birthday": $date`1975-07-07`
});
Of course, this is ES6 and not ES5 syntax, so this would somewhat change the mission statement of json5, though I don't imagine the world would care much if you can't copy/paste such an expression into an old version of IE.
Also, the name would be a bit of a problem - we can't simply change it to json6, since there is already a json6 spec in the wild. (and arguably, this feature might be a better proposal to that standard - though, personally, I feel like that spec has already added far too many features and options; I would strongly prefer json5 with this addition over json6 with it's much higher complexity.)
If you like the idea, I'd suggest supporting both $date
, $time
and $datetime
as defined by JSON schema and the I-JSON RFC. (And this should be described in the context of this format as distinct syntax literals that also happen to be valid JS expressions - not as an open-ended way to define your own custom types, as this would completely clash with the philosophy of JSON as a stand-alone data format with a fixed set of literal types.)
Thoughts?
Instead of _
I would suggest, maybe _DATA_TYPE_
as the key. Possibly come up with with a deterministic function to pass to JSON.parse and a .jsonEncode
extension method returning an object to stringify that would be checked by JSON.strungify ahead of the toJSON method.
@mindplay-dk, @prescod, @tracker1 the proposed, generic inline type syntax is not a breaking change, imo, in the sense that taking advantage of it would require the explicit specification of a "type parser map" to the JSON.parse
function (and, likewise, the explicit use of some inverse mechanism for the stringify
function) — it's opt-in, and, thus, can be seen as backwards compatible. An empty type parser map could also taken as meaning that the behavior was enabled, so that the interpretation given to the special "_" property would be honored (and also, for supporting the below presented generic forms for standard JSON types). The two ends of the pipe must, of course, be in agreement with the desired interpretation (parser maps).
In what concerns the mentioned "can of worms" that this would open, I disagree with this view. One way to see the worms is to map them to types. Users choose the worms that are allowed. Each allowed worm is added value to the user.
That said, the "only" possible worm that I see with this approach is the management of type names. Standard type names should/could be reserved to match all of the existing JSON types and some future ones. The date
type, and any other variants, would be obvious extensions to the JSON types. A standard prefix could also be reserved, to allow unconstrained, backwards compatible future evolution. All other type names would be user defined.
For completeness and regularity, a generic form could be defined for each standard type which would be semantically equivalent to its literal JSON value. Of course, the use of literals would always be preferred. As an example, in case I wasn't clear:
{
"name": "Duarte"
}
would be semantically equivalent to:
{
"name": {"_": "string", "value": "Duarte"}
}
With this generic mechanism in place, JSON5 could be easily extended to support other useful types that are slowly being introduced in JavaScript.
With this generic mechanism in place, JSON5 could be easily used to serialize and de-serialize objects of any custom classes — without a supporting schema — armed only with the inline type annotations.
Example with custom and standard types:
{
"_": "customer",
"name": "Luciano Franzi",
"birthDate": {"_": "date", "value": "1976-06-26"},
"address": {
"_": "address",
"street": "St. John",
"door": 50,
"country": "UK"
}
}
I've seen this pattern in use, throughout the years, in enterprise software.
I'm closing this issue as it breaks the unofficially official "no new data types" rule.
How many years would it take humanity to fix dates in json?
This kind of reminds me of the Time Zones video by Tom Scott.
That's a pity.
JSON5 is and will be a relaxed syntax version of JSON.
It does not have or will have a standard mechanism by which an externally defined data types can be indicated — a convention like the simple, special _
property.
@dcleao There is already a standard mechanism for defining data types. It's called JSON Schema.
Reopening and renaming since the original issue proposed adding a recommendation for using an interoperable formatting for dates in strings.
Okay, so returning to the original discussion, RFC 3339 is definitely the right choice in terms of interoperability - it's supported everywhere, sanctioned by the W3C, and so forth.
The only thing I'm wondering is whether this covers real-world use-cases well enough for things like scheduling, planning, calendars, etc.?
Given that it does not support time-zones:
Because the daylight saving rules for local time zones are so convoluted and can change based on local law at unpredictable times, true interoperability is best achieved by using Coordinated Universal Time (UTC). This specification does not cater to local time zone rules.
DST is horribly complex, so it's hard to disagree with the conclusion here.
But it seems like JSON5 has to prioritize interoperability and simplicity against the ability to represent time-zones, a commonly-used piece of data in many applications.
Not sure, but maybe it's okay to forego support for time-zones as part of a recommendation? I mean, it is only a recommendation, so you are of course allowed to represent timestamps however you choose.
But it does perhaps beg the question of whether this recommendation is going to be truly helpful? Given that, for a lot of well-known use-cases, the recommendation simply doesn't work. APIs already generally document the format they use for timestamps, and would need to do so either way, given that this is only a recommendation.
Also, RFC 3339 timestamps are already a general W3C recommendation for anything web related, which implicitly includes JSON - it isn't limited to any particular protocol, encoding or format.
So, is a recommendation going to have real value, or might it be redundant and distracting?
@mindplay-dk Good points.
The only thing I'm wondering is whether this covers real-world use-cases well enough for things like scheduling, planning, calendars, etc.?
IMO, if you need real time zone information (not just the offset), just include it in its own property along with the date.
So, is a recommendation going to have real value, or might it be redundant and distracting?
I think an important thing to remember is that JSON5 is JSON for humans, which means human readable dates are important. Many applications that communicate via JSON use Unix timestamps, and while that's great for machines, it's practically useless to humans. So the recommendation would help facilitate human readable and writable dates.
IMO, if you need real time zone information (not just the offset), just include it in its own property along with the date.
Good answer, so let's put that in the spec? e.g. "date/time values should be represented as strings in RFC 3339 format", and "timezones may be included in a separate property from the date".
Many applications that communicate via JSON use Unix timestamps, and while that's great for machines, it's practically useless to humans.
Good point, let's remove that whole section from your proposed text?
I'm also not sure you should say anything about JSON 5 parsers or generators - keep it simple, stay on point: date/time values in RFC 3339 format is recommended in JSON5 files, done. We're already "above and beyond", since this is really not about the JSON5 format per se, but just a general recommendation on how to use it in practice, so I'd keep this as short and clear as possible.
@mindplay-dk Good points. Maybe the JSON5 spec should have an "Authoring JSON5 Documents" section since JSON5 is meant for human-machine data transfer.
I can see discussion is completed, but wanted to suggest an approach.
I opened https://github.com/json5/json5-spec/issues/34 asking support for the new ES Template’s strings. The new format add support for tagged strings. Assuming that issue 34 is added, than it can be extended to support the tagged string, e.g. date2008-08-05
. This approach is generic, and can be extended to other types of literals (duration, etc.).
Parser will have the decision on how to represent tagged strings in their api. While certain languages have standard date objects, other do not have one accepted format.
If we were going to add backtick-quoted strings, adding tagged strings would seem more reasonable.
One small problem is then we're not strictly JSON5 anymore, but JSON6, which already exists.
I'm not fond of JSON6, which adds far too many opinionated features and complexity - whereas I really appreciate the focus on simplicity with the JSON5 proposal, which doesn't go overboard on anything. (And JSON6 doesn't actually specify anything about dates in tagged strings, so...)
🤔
New standards for something like JSON don't come twice in a life-time, so I think it's worth entertaining the idea for a bit longer?
If JSON5 were to someday proliferate and succeed JSON, it would definitely suck to go another 20 years without support for something as universal as dates.
I would definitely still support the idea of just recommending a date format - but at the same time, it's impossible not to recognize the fact that it doesn't really solve the problem: a parser can't parse a JSON file and know whether a given string is a date or a string. It could guess, but at the risk of making a wrong guess, crashing some code that expects a string.
Let's say we were to add support for tagged strings in general. How would we go about dealing with dates specifically? As I suggested earlier, I would propose reserving a $
prefix for dates and other future types:
const $date = value => new Date(value);
console.log({
"name": "Rasmus",
"birthday": $date`1975-07-07T23:20:50.52Z`
});
If this feature gets generalized to allow other function-names, anything without a $
prefix is up for grabs and may be used for any userland types. (Or even for mappings to factory-functions for your domain model, if you think that's a good idea.)
In terms of integration with the JavaScript environment, JSON.parse
accepts a reviver
argument, and this could be polyfilled to support userland reviver functions:
const data = JSON.parse('{"birthday": mydate`1975-07-07T23:20:50.52Z`}', (key, value, reviver) =>
// key is "birthday"
// value is "1975-07-07T23:20:50.52Z"
// reviver is "mydate"
);
Similarly, JSON.stringify
accepts a replacer
, which could be polyfilled to support something like:
const json = JSON.stringify({ date: new Date() }, (key, value) => {
if (value instanceof Date) {
return { value, reviver: "mydate" }; // generates a tagged string
}
});
For the reserved $date
reviver, the value
would be automatically revived and stringified:
const data = JSON.parse('{"birthday": $date`1975-07-07T23:20:50.52Z`}', (key, value, reviver) =>
// key is "birthday"
// value is a `Date` instance
// reviver is "$date"
);
const json = JSON.stringify({ date: new Date() }, (key, value) => {
if (value instanceof Date) {
return value; // generates a tagged string with the reserved $date name
}
});
Interestingly, the change to JSON.stringify
is not a breaking change, and is actually completely compatible with existing code, which, thanks to .toJSON()
in modern JS environments, already emits RFC3339 dates:
console.log(JSON.stringify({ date: new Date() })); // => '{"date":"2021-10-10T11:18:25.693Z"}'
So on that side, it looks like a very natural fit.
As for automatically reviving $date
tagged strings in JSON.parse
, on the other hand, this is technically a "breaking change" - in the sense that existing code does not expect Date
return values, and existing revivers do not expect Date
values being passed to them.
So even if we were to add support for tagged strings, reserving the name $date
might need to be more a recommendation that a requirement.
That's if we care about backwards compatibility with existing APIs at all, of course - an alternative would be to just add new JSON5.parse
and JSON5.stringify
functions. It's nice to think about compatibility with existing code and migration paths though.
Just spitballing here - any thoughts?
Thanks for feedback. I was not aware that JSON6 exists. I think it’s worth posting the $date`` as a proposal there. I would suggest that it make sense to codify recommendation to use the $date, $time, and $datetime - non JS languages may have different types here.
From a practical point if view, if json5 parser will adopt the tagged proposal, I think it should translate it to an object { $tag: “$date”, raw: “whatever” } with an option to try to apply the conversion at the parser level (create the date object). Different application may take different approach toward error handling, etc.
This grammar can later be extended to include the ability fir parsed section, etc.
Hope it make sense.
I looked the json6 at the link. Can you highlight the complexities you refer to ?
@yairlenga There's also a pet project of mine called JSONext. It's similar to JSON6 in that it brings ES5 and ES6 features to JSON, but it's not intended to be tied to one version of ES. It can evolve over time.
@mindplay-dk Although I like your ideas about tagged templates for JSON-like formats that would allow for it (not JSON5), for the sake of keeping this thread on topic, I'll ask that you file an issue in a different repo if you want to continue discussion.
Can we agree on adding a recommendation then?
I would suggest:
And perhaps, to be helpful, something along the lines of:
Note that this recommendation is coherent with the
toJSON
andtoISOString
methods of the ECMA-262 JavaScript standardDate
type and its constructor function.
Let's get it done and let's close this discussion? No point in beating a dead horse. 😄
Please do not have the parser turn quoted text into anything other than a string!
The application will convert the string to a date when it stores the data into it's own data structures and it has to do the conversion. It will also know it is (or is not) a date, there is absolutely no need to indicate the type in the string.
@spitzak Thanks for your suggestions. I agree that converting date-like strings to dates would be a regression for JSON5. Keep in mind that this repo is for discussing the JSON5 specification, which deals the structure and semantics of the format, not the API.
If we were to add verbiage around this topic, it would be to recommend that generators store dates as strings in RFC 3339 format to increase interoperability. Based on your comments, I think it might be wise to recommend that parsers do not automatically convert RFC 3339 strings into dates without providing some data contract or opt-in/opt-out functionality.
See also tc39/proposal-temporal.
Pertaining to the proposal, I would like to ask for more comments on time duration. Could that be specified in an interoperable way? {"interval":{"days": 3, "seconds": -2}}
or {"timestampDelta":2}
@bogdan All of the classes in Temporal already have interoperable representations as strings, and those strings are returned by both toString()
and toJSON()
. The strings use ISO 8601 (same as RFC 3339) when possible.
The beauty of JSON is that it has remained intact after JSON 1.0.
These enhancements with Dates and templates etc look cool and definitely help developers write better formatted files, but they also break a strong foundation of what JSON5 is about.
I think enhancements should be closed, and the spec left as it is. That will make sure adoption for JSON5 will remain well around the tech industry. Follow the footsteps of JSON I say
All you need is "It is recommended (but not required) that dates and times be represented as strings formatted per ISO 8601".
Then anyone who needs to build a library to handle dates and times can look up the standard and say "Oh, that's reasonable, I'll do that".
This is a known and persistent problem with existing solutions, and not using an existing solution seems unnecessarily obtuse.