JonathanSerafini / fluent-plugin-mutate_filter

Fluentd mutate filter plugin designed to replicate logstash-filter-mutate
Apache License 2.0
2 stars 4 forks source link

Split removes field from hash. #2

Open aensidhe opened 7 years ago

aensidhe commented 7 years ago

I'm trying to parse nginx access log with additional parameters about upstreams. Here is log record:

[29/Nov/2016:13:01:23 +0000] 172.18.0.1 "GET" "/rest/api/v1/private/explore" "skip=0&take=30" "/rest/api/v1/private/explore?skip=0&take=30" HTTP/1.1 401 983 604 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36" "6-1480424352.352-172.18.0.1-983" "10.0.75.1:8100, 10.0.75.1:8000" "502, 401" "127.275, 0.001" "0, 0" "127.275, 4.085"

And my fluend.conf (it is for development environment, I know it is bad for production):

<source>
    @type tail
    path /logs/nginx/access.log
    read_from_head true
    <parse>
        @type grok
        grok_pattern %{NGINXACCESS}
        custom_pattern_path /fluentd/etc
        grok_failure_key grokfailure
    </parse>
    tag nginx.access
</source>

<filter nginx.access>
    @type mutate
    replace {
        "asdsad": "%{upstream_addr}"
    }
    split {
        "upstream_addr": ","
    }
</filter>

<match nginx.*>
    @type stdout
</match>

patterns.txt

Output:

2016-11-29 15:54:14.103514500 +0000 nginx.access: {"timestamp":"29/Nov/2016:13:01:23 +0000","clientip":"172.18.0.1","request_method":"\"GET\"","uri":"\"/rest/api/v1/private/explore\"","query":"\"skip=0&take=30\"","request_uri":"\"/rest/api/v1/private/explore?skip=0&take=30\"","server_protocol":"HTTP/1.1","status":"401","request_length":983,"bytes_sent":604,"http_referer":"\"-\"","http_user_agent":"\"Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/54.0.2840.99 Safari/537.36\"","rid":"\"6-1480424352.352-172.18.0.1-983\"","upstream_status":"\"502, 401\"","upstream_connect_time":"\"127.275, 0.001\"","upstream_response_length":"\"0, 0\"","upstream_response_time":"\"127.275, 4.085\"","asdsad":"\"10.0.75.1:8100, 10.0.75.1:8000\""}
aensidhe commented 7 years ago

Changed output and test to out-stdout.

aensidhe commented 7 years ago

Ok, got it. prune method of MutateEvent thinks that array is empty. I don't know if it is bug or feature.

Disclaimer: I do not really know Ruby :)