Nanoseconds precision in Json format

[crisvo2024]

Bug Report

Describe the bug While Using Fluent-bit with the HTTP output. We discovered that the Timestamp precision is set to microseconds whenever the output is set to use format = json/json_lines format with json_date_format = iso8601/java_sql_timestamp

To Reproduce

• Example log message if applicable:

  
  fluent-bit -i dummy -o http://localhost:8888 -m '*' -p format=json_lines -p json_date_format=iso8601  -o stdout -m '*'

[0] dummy.0: [[1724678026.592204311, {}], {"message"=>"dummy"}] [0] dummy.0: [[1724678027.592222041, {}], {"message"=>"dummy"}] [0] dummy.0: [[1724678028.592239349, {}], {"message"=>"dummy"}]

fluent-bit -i http -p port=8888 -o stdout

[0] http.0: [[1724678027.595000026, {}], {"date"=>"2024-08-26T13:13:46.592204Z", "message"=>"dummy"}] [0] http.0: [[1724678028.593061832, {}], {"date"=>"2024-08-26T13:13:47.592222Z", "message"=>"dummy"}] [1] http.0: [[1724678028.999138761, {}], {"date"=>"2024-08-26T13:13:48.592239Z", "message"=>"dummy"}]

- Steps to reproduce the problem:
1. Start a Fluent-bit instance with a stdout output and a HTTP output, format =  json_lines and  json_date_format = iso8601
2. Start another Fluent-bit instance with the HTTP input and stout as output
3. Check timestamp precision. 

**Expected behavior**
The output of  fluent-bit -i http -p port=8888 -o stdout should include nanoseconds i.e.

[0] http.0: [[1724678027.595000026, {}], {"date"=>"2024-08-26T13:13:46.592204311Z", "message"=>"dummy"}]
[0] http.0: [[1724678028.593061832, {}], {"date"=>"2024-08-26T13:13:47.592222041Z", "message"=>"dummy"}]
[1] http.0: [[1724678028.999138761, {}], {"date"=>"2024-08-26T13:13:48.592239349Z", "message"=>"dummy"}]

**Screenshots**
<!--- If applicable, add screenshots to help explain your problem. -->

**Your Environment**
<!--- Include as many relevant details about the environment you experienced the bug in -->
* Version used: 3.1.6
* Configuration: release
* Environment name and version (e.g. Kubernetes? What version?): 
* Server type and version:
* Operating System and version: Kubuntu 22.04
* Filters and plugins: dummy, http

**Additional context**
In our use case, we are using fluent-bit to collect logs from multiple inputs and send them to a custom application via HTTP. These logs are produced very fast, and it is very important for us to be able to relate the logs from different inputs. i.e. being able to follow the flow of events and communication of the applications. 

While inspecting the Fluent-Bit code we found that in the function msgpack_pack_formatted_datetime from src/flb_pack.c:881 the nanoseconds are intentionally truncated. Why is this?

[cosmo0920]

This is because we use micro second precision when formatting ISO8601 format here: https://github.com/fluent/fluent-bit/blob/c69a6dfe9bd912892eb22dfddaa9101434e3ef8e/src/flb_pack.c#L1019-L1020

[crisvo2024]

Thank you for your response. I have locally modified the .%06 specifier to .%09, (in both cases where the format is either FLB_PACK_JSON_DATE_ISO8601 or FLB_PACK_JSON_DATE_JAVA_SQL_TIMESTAMP) and altered the function msgpack_pack_formatted_datetime replacing (uint64_t) tms->tm.tv_nsec / 1000 for tms->tm.tv_nsec. These changes compiled successfully and seem to work perfectly.

However, I raised this issue because in the commit where this logic was introduced, it is explicitly specified to use microseconds instead of nanoseconds.

https://github.com/fluent/fluent-bit/blob/91c7bc89eef3ad6e1cfb54954dcb9f2be45fc02d/src/flb_pack.c#L755-L768

I am concerned about potential side effects that I might not have identified. Is there any rationality behind this decision? should I submit this change as a Pull Request?

[cosmo0920]

Submitting as a Pull Request is awesome. But just changing the behavior is not good. This could be handled with the newly introduced format which should be called as iso8601_nanosec and its associated FLB_PACK_JSON_DATE_ISO8601_NANO or something like that.