search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.24k stars 1.05k forks source link

GELF 64-bit numbers support #2578

Open onyxmaster opened 7 years ago

onyxmaster commented 7 years ago

It appears that GELF as a format, being based on JSON does not support 64-bit numbers, while Graylog itself does support them. Is there a proper way to get 64-bit field data via GELF into Graylog without either losing precision (when sent as numbers they are converted to 64-bit FP with 52/53-bit significand) or storing them as strings?

kroepke commented 7 years ago

It appears there is not right now. I have just sent a string "18446744073709551557" and then tried apply a copy extractor and then do a numeric conversion on it, but that uses Double conversion as well. Presumably because Java expects a signed long and Graylog does not have enough information to guess that it needs to use the unsigned variant from Java8.

Even if Graylog could do it, I wonder what elasticsearch would make of it since it should have the same limitations regarding wide 64bit values, but I need to check that.

Leaving this open to solicit input from @Graylog2/engineers

kroepke commented 7 years ago

According to https://www.elastic.co/guide/en/elasticsearch/reference/current/number.html elasticsearch also only supports signed longs out of the box.

onyxmaster commented 7 years ago

Oh, signed 64-bit integers would be OK. I thought the support is already in there, since http://docs.graylog.org/en/2.0/pages/extractors.html#using-grok-patterns-to-extract-data mentions that signed 64-bit integers are supported (see last table in the section).

joschi commented 7 years ago

@onyxmaster If you're using a string field to ship the 64-bit number, you can use a "numeric converter" (or the shiny processing pipelines) to convert it into a number.

GELF, being based on JSON, is currently not able to ship "native" 64-bit numbers.

onyxmaster commented 7 years ago

Unfortunately, using converters is very inconvenient when the traced data format is completely dynamic (i.e. there is no well-defined set of fields). This requires too much coordination between application developers and graylog admins.

onyxmaster commented 7 years ago

If you consider this and this, it appears that 64-bit value support is already there in the GELF codec code at least (and it assumes the line number to accept 64-bit values for some reason). I can do some testing myself, but I wrote Java about ten years ago and probably will do more harm than good.

mikkolehtisalo commented 7 years ago

RFC-7159 does have somewhat relaxed interpretation of the number type. At least float numbers with double precision should be supported. If the number doesn't seem to contain decimal points, it's the same as 64-bit (signed) integer then?