Json.NET interprets and modifies ISO dates when deserializing to JObject

[FrancoisBeaune]

We're using Json.NET 8.0.2 in a C# 4.5 project, and we love it.

However this behavior bit us pretty hard:

var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }");

Here, in our time zone, ob["x"].Value<string>() returns "2016-03-31T02:02:00+02:00".

The problem disappears if we specify how to parse dates:

var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }",
    new JsonSerializerSettings() { DateParseHandling = DateParseHandling.None });

But I find it dangerously surprising that by default Json.NET interprets strings as dates (let alone the performance issues this inevitably brings with it).

Is this the expected behavior? If so, is there a way to disable this behavior globally?

[FrancoisBeaune]

Seems like JsonConvert.DefaultSettings would allow to globally override this behavior: http://james.newtonking.com/archive/2013/05/08/json-net-5-0-release-5-defaultsettings-and-extension-data

I would still be interested in hearing your opinion on this!

[antonovicha]

I suppose this is for convenience. We in our project do use this behavior because that way dates are human readable. On other hand MS Dates ( /Date(1224043200000)/ ) are not.

[FrancoisBeaune]

Deserializing a string to a string should not arbitrarily change the string. That really sounds like a bug to me, possibly a design bug. I'm not expecting breaking changes to be introduced to fix this, of course; I'm just interested in hearing the author's opinion on this. We found a workaround and that's good enough for us right now.

[antonovicha]

Way to represent a Date in Json is quite debatable because there is none in Json standard. Here is SO question regarding this topic: http://stackoverflow.com/questions/10286204/the-right-json-date-format

Could be that this is why author choose 'use ISO8601 string as Date' behavior.

[FrancoisBeaune]

@antonovicha That's completely beyond the point. I'm just deserializing a string into a string, I do not expect any changes to the string.

[FrancoisBeaune]

I'm hitting this problem again, with a different scenario.

Given a string s with the value ['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00'], I need to check that it is indeed a valid JSON array, and I then need to retrieve the unmodified value of the individual items. Unfortunately, setting default settings does not appear to have any effect.

Here is the code:

JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
    DateParseHandling = DateParseHandling.None
};

var s = "['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00']";

var array = JArray.Parse(s);
foreach (var item in array)
{
    var itemValue = item.Value<string>();
    Console.WriteLine(itemValue);
}

This prints

05/10/2016 13:51:20
05/10/2016 13:51:20

instead of

2016-05-10T13:51:20Z
2016-05-10T13:51:20+00:00

[tangdf]

var s = "['2016-05-10T13:51:20Z', '2016-05-10T13:51:20+00:00']";
using (JsonReader jsonReader = new JsonTextReader(new StringReader(s))) {

        jsonReader.DateParseHandling = DateParseHandling.None;

       var array = JArray.Load(jsonReader);

        foreach (var item in array) {
            var itemValue = item.Value<string>();
               Console.WriteLine(itemValue);
          }
 }

This prints

2016-05-10T13:51:20Z
2016-05-10T13:51:20+00:00

[jetuser]

They've commented on this before and I still disagree with their assessment of "by design" for this scenario:

865

[JamesNK]

Is this the expected behavior?

Yes it is. Use DateParseHandling if you want to change it.

[FrancoisBeaune]

@JamesNK The point is that DateParseHandling has no effect on JToken.Parse(), only on JsonConvert.DeserializeObject().

[JamesNK]

Set it on JsonTextReader that you pass to JToken.Load

[FrancoisBeaune]

Alright, thanks.

[thomaslevesque]

Is this the expected behavior?

Yes it is. Use DateParseHandling if you want to change it.

@JamesNK could you explain the reasons behind this design? It seems pretty strange to me that a string value should arbitrarily be interpreted as a date just because it looks like a date...

[JamesNK]

Because JSON doesn't have a date format.

[thomaslevesque]

Because JSON doesn't have a date format.

Maybe I'm missing something, but I don't understand how this answers my question... or maybe my question was unclear. I'll try to clarify.

It doesn't matter which date format is used; the problem is that JSON.NET tries to parse the string as a date when we're just asking for the raw string. If I have something like this:

[
    {
        "id": 123,
        "name": "foo",
        "comment": "blah blah"
    },
    {
        "id": 456,
        "name": "bar",
        "comment": "2016-05-10T13:51:20Z"
    },
]

The comment field is just a string, and isn't supposed to be interpreted as a date, even though it happens to contain something that looks like a date in the second element. When I do this:

var array = JArray.Parse(json); // with the JSON sample above
string value = array[1]["comment"].Value<string>();

I expect value to be exactly "2016-05-10T13:51:20Z", not the result of parsing it as a date and reformatting it in whatever happens to be the current culture. Right now, this code gives me "05/10/2016 13:51:20" (on a machine with the French culture). The string shouldn't be parsed as a date unless we explicitly request a date (e.g. with .Value<DateTime>()).

So what I'm asking is, why is the default value for DateParseHandling DateTime, rather than None?

[thomaslevesque]

Whether a date is a string or not is determined when loading the JObject, not when converting it to a strongly typed object.

(I just saw this message in the other discussion; GitHub seems to be having trouble delivering notifications)

This might be the root cause of the problem. Since there is no standard date format in JSON, JSON.NET shouldn't even try to determine the type when loading the JObject... it should just assume it's a string, and leave the decision to treat it as a date to higher layers that actually know what it's supposed to be.

Anyway, I realize that's not something that can be changed easily, as it would likely affect many users. But is there a way to get the DateParseHandling.None behavior when using JToken.Parse? i.e. without reverting to JToken.ReadFrom and setting DateParseHandling on the reader?

[JamesNK]

i.e. without reverting to JToken.ReadFrom and setting DateParseHandling on the reader?

No. If you want to customize that setting then use ReadFrom.

So what I'm asking is, why is the default value for DateParseHandling DateTime, rather than None?

Because I judged most people want a date rather than a string. And for people who don't then there is a setting to change it.

[FrancoisBeaune]

While I really like Json.NET and thank you for the great work you're doing on that library, I deeply regret this design decision.

As I and other have been pointing out again and again, here and elsewhere, it really is surprising that what is supposed to be an opaque string gets mutated based on the culture.

Having to sprinkle

JsonConvert.DefaultSettings = () => new JsonSerializerSettings
{
    DateParseHandling = DateParseHandling.None
};

at the beginning of every project using Json.NET is annoying and extremely error-prone. Moreover there are cases that even this setting doesn't handle, requiring deep convolutions just to get a string out of a string...

[FrancoisBeaune]

Moreover, this got to have performance implications, because for every string property of a JSON object, you need to find out if it's parsable as a date, even when one just want to get strings back!

[JamesNK]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

Performance isn't an issue. A couple of simple checks on the string length, and certain chars exist at certain indexes eliminates 99% of non-ISO8601 strings instantly.

You can keep discussing this among yourselves but I like what it does, I have no plans to change it, and I would do it again if give the chance.

And if you don't like the default it is literally one line to change: DateParseHandling.None

[FrancoisBeaune]

Fair enough. Regardless of this particular issue, thanks for the hard work and the high quality library.

[mjamesTX]

But if JSON.NET chooses to parse ISO8601 strings as DateTimes (which is fair given JSON has no date time format), shouldn't it also ToString them into the same format? That could eliminate the issue, making the fact that it was technically parsed into a DateTime in between irrelevant. I have a hard time accepting that this test should not pass:

JObject obj = JObject.Parse("{'time':'2016-08-01T03:04:05Z'}");
Assert.AreEqual("2016-08-01T03:04:05Z", obj.Value<string>("time")); // This will fail, regardless of the value of JsonConvert.DefaultSettings

[sehrgut]

Because JSON doesn't have a date format.

That's the point. JSON doesn't have a date format. You claim this is a JSON parser, but when you force-enable incompatible extensions to the standard, you no longer have a JSON parser. You have a NewtonsoftObjectNotation parser.

[EliseAv]

but when you force-enable incompatible extensions to the standard, you no longer have a JSON parser. You have a NewtonsoftObjectNotation parser.

While I do agree with your point, it made me very happy to find out that it permits comments even though the standard format explicitly disallows them. An obvious-to-use strict mode would go a long way for people who need strict JSON conformity.

[Cobaltikus]

I have been experiencing the same problem: a string being changed just because it happens to look like a date. This could easily be resolved by adding a new method in Json.NET to get the raw string value of a given property. As far as performance goes, it may not have a noticeable impact on deserializing a single object, but put it in a loop and now every action matters. The larger your dataset, the larger the performance impact. I hope to see a GetRawStringValue method in the future. In the meantime thank you for pointing out the workaround.

[sehrgut]

@ekevoo An obvious-to-use non-strict mode would be the way to go, since many times this library is called from within compiled assemblies. I can't modify every third-party library I use just because I've upgraded the version of Newtonsoft.Json attached to a project, to make them all opt-in to strict mode.

The users who want non-standard behaviour are the ones who have the responsibility to opt-in.

[EliseAv]

@sehrgut That little horse has left the barn. This is a decision to make at library inception time, not to a years-old library that is literally THE most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct Perl (use strict;) or lively Javascript ('use strict'; in es5).

[sehrgut]

The date parsing is a new feature. This operated correctly for most of the time it was BECOMING such a widely-used library, and was introduced as a breaking change very recently. Search the closed issues.

On Friday, November 4, 2016, Ekevoo notifications@github.com wrote:

@sehrgut That little horse has left the barn. This is a decision to make at library inception time, not to a years-old library that is literally THE most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct Perl (use strict;) or lively Javascript ('use strict'; in es5).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JamesNK/Newtonsoft.Json/issues/862#issuecomment-258570809, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIFLIZPgTIibEQDWQwSRt0BP_CtGA1Oks5q672rgaJpZM4H7thy .

[sehrgut]

As well, Perl and JavaScript strict pragmas are subsets of the standard. This bad date parsing behaviour is not even a superset, it's a deviation from correct parsing.

On Friday, November 4, 2016, Ekevoo notifications@github.com wrote:

@sehrgut That little horse has left the barn. This is a decision to make at library inception time, not to a years-old library that is literally THE most popular library of the runtime (if NuGet is to be believed).

Besides, it's not like strict mode is ever default. Look at the defunct Perl (use strict;) or lively Javascript ('use strict'; in es5).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/JamesNK/Newtonsoft.Json/issues/862#issuecomment-258570809, or mute the thread https://github.com/notifications/unsubscribe-auth/ABIFLIZPgTIibEQDWQwSRt0BP_CtGA1Oks5q672rgaJpZM4H7thy .

[JamesNK]

Json.NET has parsed ISO formatted strings as dates for over 4 years, which is when it switched to writing dates as ISO by default, and it it has parsed MS date style strings as dates since day one.

[PeterDowdy]

Oh man I am very much not pleased. I just lost a day to this. I even specified the data as a string to try to pin down where it was going wrong. This library will parse a date time string, alter it to the local timezone, and then output it as a string if a valid DateTime string shows up in a string property. wat

[fmannhardt]

This is horrible. I lost a few hours on this, too. Too much magic.

[carlbm]

Have to say that I disagree with this design decision too. Many hours lost today.

[mwinslow]

I strongly feel that the current default behavior is wrong. The defult should be DateParseHandling = DateParseHandling.None. It is really unwelcome that the data being read is being "fixed" underneath the covers by default.

[antonovicha]

@mwinslow I agree with you that since json do not have DateTime format it is not a good idea to magically convert string to DateTime. However changing current behavior to not convert will introduce breaking change for existing code base that already relay on this fact (like for example bunch of our applications).

[sehrgut]

Then do a major version increment. Breaking code that depends on bad behaviour should not outweigh implementing correct behaviour.

[mwinslow]

have to agree with @sehrgut , but I do want to state overall I think this a great library. Thank you!

[jimmyheaddon]

This is downright painful, especially when you're writing libraries where you cannot second-guess what users will pass you, and equally, you cannot expect them to understand this behaviour! We're having to fudge public libraries to work around this - it just doesn't make sense to parse in one format and not permit ToString() to do the same by default

[StefanBertels]

Ok, this is an old issue and it is closed. But I again stumbled on it.

I already knew DateParseHandling option and its timezone and culture problems , but nevertheless I didn't expect that

"2017-01-02T03:04:05+01:00"

would be parsed to wrong date (2017-02-01 ...)

This is bad because ISO 8601 is a valid, unambiguous format in Germany and ISO 8601 should be correctly parsed worldwide / with arbitrary CurrentCulture.

DateTimeOffset.Parse and DateTime.Parse work correct out of the box for ISO format -- without changing or specifiying some culture.

[sammck]

It does seem that if you are going to interpret some strings in a special way and turn them into non-strings, then at the very least you should also provide a way to add escape codes to a string that would allow you to encode strings that would otherwise match your special pattern. As it stands it is not possible to serialize an object, then unserialize the result, and get back the same thing that you started with.

[loop-evgeny]

I just wasted half a day of debugging on this and will likely waste much more to figure out the impact of changing the DateParseHandling default throughout our code. Years later, this design bug is still wasting people's time.

Oh, and JObject.Parse doesn't provide any way to set DateParseHandling, so I had to disassemble it and copy its contents into my own method only to change DateParseHandling.

[wasabii]

I also hit this problem. Took my developers over a week to discover the issue. We are parsing JSON from external sources, modifying it algorithmically, and then doing other things with it.

One of the free text fields going through, randomly, had it's value changed. A very data dependent bug. It was of course fixed by turning off the magic date stuff, but holy hell it wasted a lot of time trying to pin down.

[louiskruger4c]

It seems like nobody wants this "feature", so it is surprising to me that this has still not been fixed. In my case it even causes loss of precision. My string value is "2018-04-23T09:16:41.991Z", but after JObject.Parse mangles the string it ends up being "23/04/2018 09:16:41".

Please, please fix!

[thomaslevesque]

Given the amount of feedback, I think this issue should be reopened. Not a single commenter thinks the current behavior is a good thing. There should at least be an easy way to turn it off globally.

[BowserKingKoopa]

I just smacked into this too. It's nasty and totally unexpected.

[TritonNet]

Vote for not to do the magical conversion if the target type is a string. Why someone would expect when your property is a string, the serializer elevate a json value that looks like a DateTime to a DateTime object and then converts it back to a string? it's strange.

PS: It makes sense to me the default behaviour of converting to a DateTime object when the target is object/JToken/JValue/JObject (something, not a string; Which allows holding a DateTime object)

[0xced]

I also got bit by this today. 🙁

You can keep discussing this among yourselves but I like what it does, I have no plans to change it, and I would do it again if give the chance.

@JamesNK If you still have no plans to change it, would you then consider @mjamesTX's proposal (ToString them into the same format) to at least mitigate this issue?

[mwinslow]

One of the excuses for not changing this behavior is because it will break existing code. Isn't this exactly what semantic versioning is for? Why not change the default behavior (to not manipulate data implicitly) for when a new major version is released (that would be 12 I think)? The existing code can be migrated to the new major version when it is fixed to handle the more "sane" new default behavior. I think this would make a lot of people happy.

[thomaslevesque]

One of the excuses for not changing this behavior is because it will break existing code.

Although it would technically be a breaking change, I doubt anyone actually relies on the current behavior. Most people are unaware of it, and those who are don't like it.

[antonovicha]

I doubt anyone actually relies on the current behavior.

Don't doubt. Numerous of our applications are relaying on that behavior.