Missing milliseconds, when parsing Java 8 date-time, if they are zeros

[rycler]

Issue explained here: https://stackoverflow.com/questions/47502158/force-milliseconds-when-serializing-instant-to-iso8601-using-jackson

Version: 2.9.5

How to reproduce: Project where I use Spring Boot 2.0.0.M6, Spring Framework 5.0.1.RELEASE and Jackson 2.9.5

Test 1: Serialize Instant with milliseconds set to 000:

• Initialize Instant field using Instant.parse("2017-09-14T04:28:48.000Z")
• Serialize it using Jackson
• Output will be "2017-09-14T04:28:48Z"

Test 2: Serialize Instant with milliseconds set to some non-000 value:

• Initialize Instant field using Instant.parse("2017-09-14T04:28:48.100Z")
• Serialize it using Jackson
• Output will be "2017-09-14T04:28:48.100Z"

Questions:

• Is that behavior by design?
• Is there anything I can do to force serialization of 000?

[kporter13]

I'm seeing the same behaviour with Jackson 9.5.0 going from instant to epoch millis - if the milliseconds are zeros, they're trimmed, which causes problems for anyone explicitly expecting millis, rather than seconds. This workaround with a custom json getter worked in my case at least. https://www.codesd.com/item/effective-way-to-have-jackson-serialize-java-8-instant-as-epoch-milliseconds.html

[rycler]

Thanks for the workaround, however I feel like this is a pretty major bug and should be fixed. Maybe anyone knows a dependency, that can can temporally fix this for a maven project, so I don't have to modify the existing code base?

[magict4]

I end up with writing a custom Instant serializer to get around this problem.

I guess Jackson use Instant.toString() method to serialize Instant objects by default. I also find some discussions about Instant.toString() method on StackOverflow.

[sixinli]

can use something like https://stackoverflow.com/questions/41037243/how-to-make-milliseconds-optional-in-jsonformat-for-timestamp-parsing-with-jack as a workaround for deserialization

[StephenOTT]

This is also a problem for optional sub second precision. Where the trailing zeros are cut off. This seems to be a issue with the deeper dateTimeFormatter.

The issue is if a Json string contains a date with sub second precision such as HH:ss.000000000Z. This is trimmed to just HH:ssZ when it's converted back into a string.

The omissions of trailing zeros should be optional. But this does not seem to be a Jackson issue.

Has anyone figured out how to make trailing zeros kept using DateTimeFormatterBuilder ? From the looks of the class, the stripping of trailing zeros is always performed.

The goal here should be for the parser date to keep the sub second precision information. Even if there are trailing zeros. Otherwise you can not compare Json string values as the original will have a different string date then what was printed by the parsed version

[Flomix]

I actually prefer the encoding with optional second fractions. I can't think of any reason to force fractions to 3 digits:

• forcefully adding .000 to full second values would add no information and make the value less readable.
• 0.5 for half a second just looks more logical to me than 0.500.
• The ISO 8601 also doesn't speak of 3 digits anywhere, it allows the addition of a decimal fraction of the smallest value if necessary, which also speaks for dropping the value if there is no fraction and to not add artifical zeroes at the end.
• forcefully trim a .0006 to .000 or .001 would change the value, which I would consider a major bug.

The only circumstance where I would expect to always see a fixed number of decimals is when a custom pattern is used, like hh:mm:ss.SSSS for 4 digits.

[Flomix]

By the way, what would your expectations be for decimal fractions in other values than seconds? Like for 4:30 would you rather expect 2017-09-14T04.5Z or 2017-09-14T04.50Z or 2017-09-14T04.500Z? All 3 are valid ISO 8601 timestamps for 04:30:00.

[StephenOTT]

It's not that the dates are "different". It's about knowing what the actual/original precision the date was collected at. You can have dates from many sensors from many different people that build and control these sensors; they generate the same data, but with various precision. You can have a requirement to know what that precision was.

[Flomix]

Actually 12:00:00,000 and 12:00:00,00004 are quite different.

That requirement might be the case for some use cases, but neither the Java Time API nor ISO 8601 do support information about stored precision, so it should not be relevant here. My point is that neither the Java time API nor ISO 8601 even support milliseconds as well. Java8 time has seconds and nanoseconds. The smallest possible unit in the ISO standard is seconds, with the addition of an optional decimal fraction of the smallest used unit with unlimited precision things like milliseconds / nanoseconds / femtoseconds can be expressed too. Considering all this I see no reason to fix the number of decimals to 3 (or any other value). Consequently I wouldn't consider the current behavior bugged or even wrong.

[EDIT]: I'm sorry, I did miss a detail in the original post. I mixed Instant with ZonedDateTime, probably because I mainly have to do with the latter and had to deal with similar questions.

I just realized that the behavior between Instant and ZonedDateTime differs as well; one groups the number of decimals in blocks of 3 (0, 3, 6, ... digits), the other uses exactly the number of decimals needed. That seems indeed unintended. "2019-04-29T16:04:12.0001Z" (ZonedDateTime) "2019-04-29T16:04:12.000100Z" (Instant) "2019-04-29T16:04:12.1Z" (ZonedDateTime) "2019-04-29T16:04:12.100Z" (Instant)

[StephenOTT]

@Flomix how are you producing your second example? 2019-04-29T16:04:12.000100Z" (Instant)

The issue as i understand it, and have had to write a wrapping class around Instant is:

something like (1)2019-04-29T16:04:12.100000000Z and (2)2019-04-29T16:04:12.1Z is equal.

BUT when you submit (1), you can have a requirement to preserve what the original "precision" the date was generated at / parsed at. So if you are parsing a JSON object with a property that represents a date, the two dates (1) and (2) are technically equal, but if you lose the precision that the date was collected at, when you re-generate the JSON string, the new date string would be (2) rather than the original (1) date, and thus the JSON strings would not be equal. Further you lose the defined precision in the date. If you are collection various precisions from many different "sensors", you want to know what to know what precision the date was actually calculated at, not just the parsed form.

[Flomix]

Uhm... I just convert my datetime to an instant. My expectation was that in both cases ZonedDateTime and Instant would render to an identical string, since they represent identical values.

public class JsonTimeTest {
    public static void main(String[] args) throws JsonProcessingException {

        ObjectMapper om = new ObjectMapper().registerModule(new JavaTimeModule())
                .disable(SerializationFeature.WRITE_DATES_AS_TIMESTAMPS)
                .enable(SerializationFeature.INDENT_OUTPUT);

        Dto dto = new Dto();
        dto.datetim1 = ZonedDateTime.of(2019, 4, 29, 16, 4, 12, 100000, ZoneId.of("UTC"));
        dto.instant1 = dto.datetim1.toInstant();
        dto.datetim2 = ZonedDateTime.of(2019, 4, 29, 16, 4, 12, 100000000, ZoneId.of("UTC"));
        dto.instant2 = dto.datetim2.toInstant();

        System.out.println(om.writeValueAsString(dto));
    }

    public static class Dto {
        public ZonedDateTime datetim1;
        public Instant instant1;
        public ZonedDateTime datetim2;
        public Instant instant2;
    }
}

But again, your argument about stored precision doesn't hold, because it is not supported by the data types in question. You suggest to not use Java8 Time API at all since it doesn't fit your requirement (Joda doesn't as well if i'm not mistaken), which is okay. But that is on a completely different page, and not related to this Java8 Time API issue.

[cowtowncoder]

Ok. So I am not sure I know everything that goes on here, but let's see.

So: as to preserving precision: I don't think this is possible in general, and I don't think it should be goal of Jackson to try to automatically retain it. If this is important, then system that cares should indicate it with other metadata and probably use custom (de)serializer and/or pre-/post-processor.

However: I am not against having an option to trim / not trim "extra" trailing zeroes, so that whatever JDK offers can be used as-is with predictable behavior ("always include full 9 digits for nanoseconds").

It's just a question of

1. How to configure (with general databind features, module-specific... ?)
2. Consider backwards-compatibility based on current behavior.

An additional problem, however, is that in some cases JDK also has limitations, and this module uses JDK formatters for most of its functionality.

[StephenOTT]

@cowtowncoder Thanks for the followup. IMO, based on going through a impl and needing to retain trailing zeros and the precision in general (basically having sub-seconds in timestamps be optional from 0 to 9 digits), I think the best case is to provide a (de)serializer to retain the precision. The other issue is Instant does not store precision. So you end up having to create a custom class to store the data.

Example:

1. https://github.com/StephenOTT/STIX-Java/blob/master/src/main/java/io/digitalstate/stix/common/StixInstant.java,
2. https://github.com/StephenOTT/STIX-Java/blob/master/src/main/java/io/digitalstate/stix/json/StixInstantSerializer.java,
3. https://github.com/StephenOTT/STIX-Java/blob/master/src/main/java/io/digitalstate/stix/json/StixInstantDeserializer.java

IMO adding the "trim" or "dont trim" is not much of a improvement, because it comes down to your precision defined in your Data Formatter such as: https://github.com/StephenOTT/STIX-Java/blob/master/src/main/java/io/digitalstate/stix/common/StixInstant.java#L90.

From a JS date perspective would be what was the actual precision provided, so it can be kept or converted. .000 and .0 may be technically the same from the perspective of date comparison, but when you are looking to determine what precision the date was collected at, those zeros make sense.

[kupci]

Not exactly what you are looking for maybe, but worth mentioning from one of the links above is the appendInstant(int fractionalDigits) in DateTimeFormatterBuilder.

[beamerblvd]

Finally circling back to some old issues. Here's my take on this: The ISO standard in question does not require any particular number of fractional-second digits, and ISO-compliant parsers should be able to handle both the presence and absence of fractional-second digits. Given this, it's logical to conclude that encoders should not need to export a fixed number of fractional-second digits, because parsers should behave. However, that's a naive position when you dig a little. Not all parsers are well-behaved. In fact, most aren't. How many Jackson bugs have we fixed here? A lot.

So ... I believe the current behavior is the correct default and should be left as the default in all future versions, but I also believe there should be an option to specify a fixed number of fractional-second digits, in order to accommodate those systems that have issues when there are no fractional-second digits.

[mwmahlberg]

@beamerblvd I disagree. The ObjectMapper, when explicitly asked to serialize with nanosecond precision, should not trim millis.

[marwin1991]

I have also encountered this problem. Funny fact and work around is that, using:

@JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSXXX")

Helps to keep zeros like: 2021-05-19T15:12:41.330+02:00 or 2021-05-19T15:12:41.300+02:00

[StephenOTT]

@marwin1991 what is your storage for that field? In your example how do you receive and store the difference between:

1. 2021-05-19T15:12:41.330+02:00
2. 2021-05-19T15:12:41.330000+02:00

[marwin1991]

@StephenOTT I am using something like this:

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSXXX")
    private OffsetDateTime sendDate;

to get 2021-05-19T15:12:41.330+02:00

And if you would like have more "zeros" like here 2021-05-19T15:12:41.330000+02:00 you can use:

    @JsonFormat(shape = JsonFormat.Shape.STRING, pattern = "yyyy-MM-dd'T'HH:mm:ss.SSSSSSXXX")
    private OffsetDateTime sendDate;

But always the last 3 digits will be 0 because OffsetDateTime does not store such a precision

[stonux]

For me, this breaks sorting and comparisons in Zulu time zone (UTC):

Example:

2021-05-19T15:12:41.330000Z   // this instant should be AFTER the one below
2021-05-19T15:12:41Z               // the Z breaks the sorting order

According to ISO, alphanumeric sort must be equal to chronological sort.

I would like to INSERT Instant.toString() into an SQLite data base, which has no semantic timestamp data type. But:

• comparisons are broken, BETWEEN is broken
• ORDER BY is broken

Workaround: use Instant.plusNanos(1).toString() Example:

2021-05-19T15:12:41.000000001Z
2021-05-19T15:12:41.330000001Z    // chronological order re-established

This workaround fails of course if the original Instant was 2021-05-19T15:12:40.999999999Z To solve this issue, I now use

pstmt.setString(1, Instant.now()
                    .truncatedTo(ChronoUnit.MILLIS)
                    .plusNanos(100_000)
                    .toString());

This will always set 100 microseconds 000 nanoseconds. But now, I get consistent results:

2021-05-19T15:12:41.000100Z
2021-05-19T15:12:41.330100Z