Json.NET interprets and modifies ISO dates when deserializing to JObject

[FrancoisBeaune]

We're using Json.NET 8.0.2 in a C# 4.5 project, and we love it.

However this behavior bit us pretty hard:

var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }");

Here, in our time zone, ob["x"].Value<string>() returns "2016-03-31T02:02:00+02:00".

The problem disappears if we specify how to parse dates:

var obj = JsonConvert.DeserializeObject<JObject>("{ 'x': '2016-03-31T07:02:00+07:00' }",
    new JsonSerializerSettings() { DateParseHandling = DateParseHandling.None });

But I find it dangerously surprising that by default Json.NET interprets strings as dates (let alone the performance issues this inevitably brings with it).

Is this the expected behavior? If so, is there a way to disable this behavior globally?

[mwinslow]

@antonovicha I'm not doubting that, somehow further back in this discussion that was mentioned. What I am advocating is that semantic versioning is used to ensure the existing applications still work, while people that would love to have the new default behavior can adopt a new major version (i.e., 12.*). This would allow for safely migrating the applications.

[Cobaltikus]

There's no need to change the current behavior. Just add a new property or function to get the original raw text string for any given value. Best of both worlds. Everyone is happy. If you want the date object, you got it already. If you want the original raw text value, use the new property or function. It will become standard practice to use the new stuff and people will stop getting surprised.

[thomaslevesque]

Don't doubt. Numerous of our applications are relaying on that behavior.

I'd be curious to see the code. I can't think of any sensible way to take advantage of it.

[mwinslow]

@Cobaltikus Regarding your comment about "no need to change the current behavior" If you look at the beginning examples regarding this issue you will note examples that show that the original string has already been manipulated once read through the library. With the current default behavior there is no way to get the original string value once you have loaded the data through the Newtonsoft.Json library. You don't have access to the raw string. Automatically parsing what is a string as a datetime just because it appears to be a date is a flawed design decision.

People are still commenting on this sneaky behavior two years after the issue was closed because the chosen default is not a sensible default. It really isn't. I guess since it works for the maintainers of this library this isn't going to change, but I think there will continue to be a steady stream of people that comment on this issue because when you get sideswiped by it you are tearing your hair out trying to understand why the data is being mangled unexpectedly. It is really brutal when it happens.

[Cobaltikus]

@mwinslow, Sorry for the confusion. I understood all of that already. What I am talking about is adding a brand new feature. There absolutely is a way. During the initial load, save the original raw text string. Leave everything else the same. When you want the string value, access it through this new property that does not exist today. If I was a maintainer of this library I could do it myself. Does that make more sense? The suggestion to add the new property or function was intended for the maintainers of this code, but I failed to make that clear.

[Cobaltikus]

Here's a better explanation. JObject.Value() returns the unexpected that we all dislike. JObject.RawString, or JProperty.RawString, or JToken.RawString, etc could be the new property that returns the expected value. But this RawString property doesn't exist. So I'm suggesting that a library maintainer should add it. Once this is in place users of this library could use it, and it would become standard practice to use it, and this problem would go away without having to change existing code for those who like the current behavior. A new property with the expected result would eliminate the need to change the existing behavior for the existing functions.

[loop-evgeny]

... and it should then use that RawString property when serializing back to JSON, so that the string survives the roundtrip.

[TheDuQe]

(Update : See Workaround below) Well, spent years with this great library to face suddenly this issue. It could be a "by design" behavior if I wouldn't lose following information :

Before parse : value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

JsonConvert.DefaultSettings =  () => new JsonSerializerSettings{DateParseHandling = DateParseHandling.None};
message.payload = JObject.Parse(json);

After parse : value=2018-03-22T02:18:17.3725571+11:00 <-- My time zone

Result : Values are equal but I have definitely lost one information : my customer's timezone which is important to my boss.

# WORKAROUND : Parse(json) is not the right method, use DeserializeObject(json) instead

Before parse : value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

JObject obj = JsonConvert.DeserializeObject(json) as JObject;
message.payload = obj["Payload"] as JObject;

After parse : value=2018-03-21T16:18:17.3725571+01:00 <-- Customer time zone

[jhellemann]

Thanks, @RoiArthur! You saved my day :) In my case all DateTimes are in UTC and it's really important to stay in UTC and not local time. So I also had to add:

JObject obj = JsonConvert.DeserializeObject(line, new JsonSerializerSettings
{                        
      DateTimeZoneHandling = DateTimeZoneHandling.Utc
}) as JObject;

[jhbertra]

Yup, I just hit this "feature". Boo 👎

Thanks to the community for providing workarounds though! Too bad the default implementation is something that needs to be worked around in the first place... oh well.

[bdcoder2]

I have LOST days to this "feature" -- overall, I like the library, but really? !! To "just decide" to convert a string to a date on whim is not a "design feature", it has been nightmare -- for me at least !

[benbenwilde]

@JamesNK would you consider taking a poll and changing the default DateParseHandling setting based on the results in the next major version release? Pretty much everyone uses this library, so they should have a say??

[TheDuQe]

@JamesNK Parsing string as date using JObject.Parse(json) should result in a DateTimeOffset in order to keep the full message content. Parse the string as a DateTime lead to lose important data (Time zone).

[mavericken]

I only lost about an hour figuring out this was why my code was breaking. Maybe you guys aren't seeing the beauty of this feature. A minefield of little gotchas goes a long way for job security. We wouldn't want new developers coming in and showing us up, now would we?

[SailSouthwest]

Possibly the worst design decision I've come across -- ever -- in an otherwise great library.

[jhbertra]

Serialization should be pure - this is not pure. This is implicit environment reading. This feature means that, by default, a test case could pass or fail depending on the timezone of the host machine.

[mazjindeel]

This is not a feature its a bug

[hickford]

For comparison, ServiceStack.JSON.parse returns a string "2016-03-31T07:02:00+07:00"

(ServiceStack.JSON.parse("{ 'x': '2016-03-31T07:02:00+07:00' }") as Dictionary<string, object>)["x"]

[ckpearson]

Just as an additional hat in the ring, this has just bitten us and taken a day or two to find what was going on.

The behaviour of a string being interpreted without being asked is pretty counterintuitive, and any "spooky action at a distance" in programming really should be avoided...

That said, a few settings changes to tweak this and ensure outbound dates get serialised the way we need them to isn't too onerous.

[bdcoder2]

Again -- this COULD and SHOULD be resolved by introducing a simple global setting, ie:

NewtonSoft.Json.Do_not_screw_with_dates = True

[markzielinski]

I just came across this as well, lost a few hours of time due to it. I've never seen such arrogance.

[TylerBrinkley]

It seems the majority of cases here where this causes pain is when the object is deserialized and then re-serialized and the user is expecting the strings to be identical when the value is not modified. Is there any way the JValue that is created upon deserialization could store the original string value and if the date value is not changed then it would serialize the JValue with the original string as opposed to re-serializing the date value?

[thomaslevesque]

@TylerBrinkley actually roundtrip deserialization/reserialization works fine. It's only when you try to get the value as a string from the JObject that you get the issue.

https://gist.github.com/thomaslevesque/e832247047791fcfbda488ae3f2df60c

[jsolman]

I strongly believe this is a bug and not a feature. @thomaslevesque the roundtrip deserialization/reserialization does not work properly either when there are fracitonal seconds in the time. It loses the fractional seconds.

[bdcoder2]

THIS IS A BUG -- period. Add the necessary code to leave dates alone by adding a configuration parameter or property.

[thomaslevesque]

@thomaslevesque the roundtrip deserialization/reserialization does not work properly either when there are fracitonal seconds in the time. It loses the fractional seconds.

Are you sure? The gist I posted above works fine with fractional seconds as well

[jsolman]

are you sure? The gist I posted above works fine with fractional seconds as well

Your gist may work; however, if you set the JObject value for the key retrieved to the string it gives you, and then re-serialize it will have lost the fractional seconds.

jObject["TheDate"] = jObject["TheDate"].Value<string>();

I guess that isn’t what you are calling round trip though as I guess you mean round trip without modifying the JObject key with the value retrieved from the key.

In my opinion it is still a deserialize and reserialize problem — since after copying the JObject value into a new JObject’s value it will not have the same value as the original after reserialization.

[jsolman]

Why didn’t they at least make the default string serialization format when converting to a string be something like DateFormatHandling.IsoDateFormat?

[B4rT45]

This is a great library but unfortunately you can't call such behaviour a feature - it's clearly a bug, however you look at it.

You pass an int and get an int, you pass a bool and get a bool, you pass a string and you might get a string, or a date that matches the passed string or not, or has a different time or represents a completely different calendar day altogether ... Surely, this can't be right.

Please note, manipulating and/or changing the processed data and/or data types is clearly not a responsibility of a serialisation library, at least not by default.

It should be up for the consumer to opt-in for the under-the-hood-magic conversion, i.e. the DateParseHandling.None should be the default and user should explicitly specify a different one if they want to have any auto conversions.

I think one could get away with calling such behaviour a feature, instead of a bug, if it was on the opt-in basis, or at least if it was handling dates properly which is very, very complicated and is a VERY good a reason in itself that it should not be done by default, if at all (even Microsoft's .NET DateTime implementation is pretty much useless, see nodatime).

Found today only by chance by having some unit tests that started failing and because there were quite complicated (I know they should not be, but I inherited them) I wasted a few hours to find this little nugget as this was the last place I would look for a solution for the problem. It was also very strange that my log was telling me I'm having and passing a string to serialise it but on the receiving I would accept certain types only (including string that I passed), and DateTime was not one of them, but since logs said I passed, lets say a Dictionary of <string, string> why the other end can't deserialise it as Dictionary<string, string>? Now I know, because it once of the elements comes back as <string, DateTime>

More importantly now I'm also worried that because of loosing precision/changing the dates (we run stuff globally) we can have some issues somewhere in the system (which must be VERY precise with times and dates) and those issues are just lingering somewhere waiting to manifests themselves at some point in future when it's too late, we are just not aware of them yet. I/we will have to go through the whole codebase to make sure other devs didn't make an assumption that they will get whatever they put it (because why would they make such a strange assumption?)

If it was "by design", then like others here, I think that this was a poor decision as it's not intuitive and that's what it should be.

Although there are some nice suggestions above how this could be solved (temporally) in the current version (in preparation for the breaking change) I hope it will be changed with in the next major (breaking) version.

Thanks for your great work.

[dancrn]

i too have been bitten by this. honestly, if you want to parse strings as dates (which imo isn't a good move) then at least do it correctly;

JObject.Parse("{\"date\":\"0001-01-01T00:00:00+00:00\"}").ToString();
//returns "{\n  \"date\": \"0001-01-01T00:00:00-00:01\"\n}"

where's the extra minute coming from? :upside_down_face:

[smasherprog]

I just encountered this bug as well. Took me for ever to figure out that there was actually a bug in this library. This library should NOT DESTROY DATA! This default setting to destroy data is really confusing.. Why is destroying data a good design decision?

[dlumeida]

This behavior is very frustrating. After some time trying to figure it out I could solve my problem using

services.AddMvc().AddJsonOptions(opt =>
{
    opt.SerializerSettings.DateParseHandling = Newtonsoft.Json.DateParseHandling.None;
});

But I think it should be set as a default behavior since it causes lots of troubles when working with different timezones, nodatime, etc. Another pain is when working with <input type="datetime-local" step="1"> in Google Chrome.

Ex: If the field is set to "2018-11-05 10:20:59" Chrome serialize it as "2018-11-05T10:20:59" and Json.Net parse it as DateTime. If the field is set to "2018-11-05 10:20:00" Chrome serialize it as "2018-11-05T10:20" and Json.Net parse it as String.

[mikeeheler]

I wonder how much time collectively has been wasted on people running into exactly this issue, as I've just wasted some of mine this morning.

// User entered in a timestamp from somewhere for their own reasons
string input = "{\"user_comment\": \"2018-07-26T19:45:36-07:00\"}";
JObject parsed = JObject.Parse(input);

Console.WriteLIne(parsed["user_comment"].Value<string>());
// "07/26/2018 19:45:36"

Console.WriteLine(parsed["user_comment"].Value<DateTime>().Kind);
// "Local" ... if you say so

No timezone info in the result, no timezone transformation (the machine this ran on was at UTC-8), no indication that the field was meant to be a date. Arbitrary user input that means that if this were in an array of similar objects, the same field in different objects would be represented differently. This user just happened to put a ISO timestamp in the comment.

It just ain't right. The sad part is there is probably a ton of code out there right now which depends on this broken behavior.

To work around this and use JObject.Parse I need to set a global somewhere? How is that meant to be compatible with unit testing or concurrency? There's no variant of JObject.Parse that takes a settings object that can override the broken default.

[smasherprog]

Your preaching to the choir here mike. It was a bug that people continually run into which will not be fixed unfortunately.

[RoadTrain]

Funny thing is that this bug helped me find a bug in my own code which was incorrectly handling negative timezones (i.e. Json.NET modified the date representation in a json file, which triggered the bug in my code).

Still, I find this design choice very surprising, and the overall handling of the issue looks unprofessional.

[thomas-darling]

@JamesNK I've just wasted a few hours of my life because of this incredibly stupid default behavior. I don't care about the reasoning that went into this - it's wrong, and should be fixed.

@TylerBrinkley

It seems the majority of cases here where this causes pain is when the object is deserialized and then re-serialized and the user is expecting the strings to be identical when the value is not modified.

Exactly! The fact that this is not the default behavior is absolutely insane.

[dancrn]

look at all the times this bug has been mentioned in other project's issues! some significant mentions are Microsoft (twice! Azure, and SignalR!) and Stripe.

but hey, it's a feature, not a bug 🙃

[TheMightyPope]

I think enough is said here about this bug, and it seems there are no solution in sight, so I'll try the DateParseHandling = DateParseHandling.None global approach. Great to just notice this after building a big project and now having to redo a bunch of things, re-test, etc.. As a side note, what I found was that even the conversion was wrong, the library threw away half an hour for some reason (...T00:30:00) became 12:00:00 AM in the local representation. We only noticed because of the lost 30 mins.

[rotanov]

Bit me hard too.

[springy76]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

Boy, there is more than one here who isn't pleased.

IT IS A BUG!

I can see in the debugger that the JToken still is aware of the original value but when requested to deserialize to a [DataMember] attributed string property using ToObject<T> the string is rebuilt from some interpreted data. And if that alone is not buggy enough it also uses an undefined string format for this which is NOT the configured DefaultDateFormatString yyyy'-'MM'-'dd'T'HH':'mm':'ss.FFFFFFFK but something ugly different using "/" and totally unsortable ymd ordering.

[sehrgut]

Sometimes you can't please everyone and in this case you're the person that isn't pleased.

The thing is, you can please everyone! By making your parser perform according to spec. That's literally the definition of pleasing everyone in this case.

If you want to make a wrapper class/subclass that auto-coerces datatypes, then do that too, if you must.

[mehrbat]

I also believe it is a bug and there must be a way to stop this conversion. How could it possibly convert without knowing the exact datetime format? Day and Month could be confused.

I have date/time and datetime offset in my input json as text, so no top level settings can sort it out. I would prefer an attribute on the properties of the target [c#] class to convert the values the way I want per property, like these ones: https://www.newtonsoft.com/json/help/html/SerializationAttributes.htm

[mcintyre321]

It's the default that keeps on giving. I just spent an hour trying to figure out why my serialization code was mangling my datetimes, when in fact it was that the Visual Studio debugger Json Viewer is parsing string properties as DateTimes and ToStringing() them. Wonder what library it uses under the hood...

[codermrrob]

Just burnt me too. Great library with just one horrible default behavior setting. I really feel that this should be something you have to turn on rather than turn off. In my scenario we deal with a lot of different date formats and typically need to re-format them into something else. Having this awesome library change the format of the date to something unexpected is the only not so awesome feature in it. Its made worse even because I explicitly request the property value as string and still it gets read as a date in the incorrect format.

[bdcoder2]

Did I miss @JamesNK's announcement on April 1st?

"As of version 12.1, date/time values will NO LONGER be internally interpreted and instead the original value will be retained".

[Cobaltikus]

Where is that announcement? Was it possibly an April Fool's Day joke? If so, by JamesMK, or by bdcoder2?

[bartelink]

I think we need to get the real JamesMK on this issue!

[bdcoder2]

It was meant as a joke -- the same way this bug is being treated as !

[milosloub]

@JamesNK It's a trap when develper use external library based on: JToken.Parse(someString) and there is no way to get rid of this NON-STANDARD behaviour. Finally this leads to full database of corrupted data and you can't do anything about this.

This is not acceptable!!!

[PathToLife]

This burnt me as well, was trying to cast as (string)value

Who the hell checks serialization of different strings to make sure they aren't objects. Serialization of string should give string.

TLDR; to whoopdydo a server using Newtonsoft Json. Just save a DateTime Json string as a troll user.