search

Graylog2 / graylog2-server

Free and open log management
https://www.graylog.org
Other
7.4k stars 1.07k forks source link

sort messages by more than one fields #5693

Open hellobinge opened 5 years ago

hellobinge commented 5 years ago

hi there,

I am just wondering if graylog 3 will provide sorting with multiple fields? ElasticSearch actually allow sorting with multiple keys.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-sort.html https://stackoverflow.com/questions/29802042/elasticsearch-sort-order-for-multiple-fields

Thanks.

BaptisteBdn commented 5 years ago

Hi, I haven't been able to sort logs with multiple fields in Graylog 3.1. Will Graylog provide this feature in the next versions ?

It could help to solve the sorting for logs with the same timestamps. We would be able to first sort with timestamps and then with the log_offset. I'm sure it would be useful for other cases too.

Thanks

hellobinge commented 5 years ago

personally, i would actually recommend to look into elasticsearch, logstash, filebeat, etc.

especially filebeat, when streaming log, it will add a "byte offset".

BaptisteBdn commented 5 years ago

personally, i would actually recommend to look into elasticsearch, logstash, filebeat, etc.

especially filebeat, when streaming log, it will add a "byte offset".

Hello, I'm using filebeat and I sort my logs using the offset, the problem is that the offset has a maximum length and returns to 1 when it reaches it. It's not problematic it you sort on a short period of time because there is not a lot of logs.

However having the possibility to sort with multiple fields would solve the problem. I could first sort by timestamps and then sort by offset, that way timestamps with the same milliseconds would be ordered with the offset.

hellobinge commented 5 years ago

elastic search does support multi-field sort. i am not sure if graylog support this feature now.

https://www.elastic.co/guide/en/elasticsearch/reference/current/search-request-body.html#request-body-search-sort

(actually, there is another bug in graylog. graylog might delete empty characters at the beginning of the line. e.g. i send a log "\thello world" to graylog, but in graylog storage, this line becomes "hello world". i am not sure at which step, graylog removes this "\t" character. this makes the logs look not that beautiful, when i tried to show logs on web page. and that is the reason why i turned into pure elasticsearch stack, though i like convenience that graylog provides.)

Aenima4six2 commented 4 years ago

I would even be thrilled if Graylog could, once again, sort by at least one field. We currently run the latest Graylog in our dev environment, but cannot upgrade prod past 3.1.4+1149fe1 due to the lack of field sorting.

Helicube commented 3 years ago

Any news about these issue in version 4.0 ? I would like to solve the problem of inconsistent message order with equal timestamp this way.

csm10495 commented 1 year ago

Any updates on this? I started to try and use graylog but was hitting issues where timestamps matched up. I added a log-index field to the emissions... but i can't sort by both timestamp and that index :/

boosty commented 1 year ago

@mpfz0r recently worked on a related change: https://github.com/Graylog2/graylog2-server/pull/6711

While the linked PR will not introduce a custom multi-field sorting feature, it will add and sort by an internal sequence number, which helps when dealing with messages having identical timestamps.

Please see the PR for more details.