I am getting "Provided Grok expressions do not match field value" in the error field when I am sending some AWS Cloudfront Real time logs through kinesis to Elasticsearch.
I’m not a grok expert, but that message generally means that it is failing to find the match pattern specified. So my guess is there is a configuration that doesn’t have a fall through option for “i didn’t find any match, so do this” and instead it errors because there is no match at all.
I am also using right es_datastream_name parameter for Cloudfront as logs-aws.cloudfront_logs-default as per document but seems like my managed ingest pipeline logs-aws.cloudfront_logs-2.11.3 is not able to parse the logs to correct fields. Not sure if the pipeline is correct for parsing the Amazon Cloudfront real time logs
abhipandey9963
commented
2 weeks ago
I am getting "Provided Grok expressions do not match field value" in the error field when I am sending some AWS Cloudfront Real time logs through kinesis to Elasticsearch.
I’m not a grok expert, but that message generally means that it is failing to find the match pattern specified. So my guess is there is a configuration that doesn’t have a fall through option for “i didn’t find any match, so do this” and instead it errors because there is no match at all.
Cloudfront has two kinds of Logs(Standard- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/AccessLogs.html and Real time- https://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/real-time-logs.html) with different field names
I am also using right es_datastream_name parameter for Cloudfront as logs-aws.cloudfront_logs-default as per document but seems like my managed ingest pipeline logs-aws.cloudfront_logs-2.11.3 is not able to parse the logs to correct fields. Not sure if the pipeline is correct for parsing the Amazon Cloudfront real time logs
this is my JSON error { "_index": ".ds-logs-aws.cloudfront_logs-default-2024.01.16-000001", "_id": "d85ecea250", "_version": 1, "_score": 0, "_ignored": [ "event.original" ], "_source": { "cloud.region": "us-east-1", "aws.firehose.arn": "arn:aws:firehose:us-east-1:461485115270:deliverystream/Elastic-test", "data_stream.namespace": "default", "aws.kinesis.type": "deliverystream", "error": { "message": "Provided Grok expressions do not match field value: [1707232747.851\t43.133.38.182\t0.455\t403\t642\tGET\thttp\twww.avyaanpandey.com\t/\t537\tSIN5-C1\tux5iSeDweA1B42E1fnHI8XqmogAL6hmvSL--OotSDSQ5nqliPGY4Lw==\td3jr12hh9w9vyl.cloudfront.net\t0.455\tHTTP/1.1\tIPv4\tMozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1\t-\t-\t-\tError\t-\t-\t-\tError\t-\t-\tapplication/xml\t-\t-\t-\t40168\tError\tSG\tgzip\ttext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7\t\tHost:www.avyaanpandey.com%0AUser-Agent:Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1%0AAccept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7%0AAccept-Encoding:gzip%0AAccept-Language:zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7%0ACache-Control:no-cache%0AConnection:keep-alive%0APragma:no-cache%0AUpgrade-Insecure-Requests:1%0AConnection:close%0A\tHost%0AUser-Agent%0AAccept%0AAccept-Encoding%0AAccept-Language%0ACache-Control%0AConnection%0APragma%0AUpgrade-Insecure-Requests%0AConnection%0A\t10\tEJC0FPBIT5SPQ\td3jr12hh9w9vyl.cloudfront.net\t-\t-\t132203\n]" }, "data_stream.type": "logs", "aws.firehose.request_id": "9b0ee118-8367-4064-a297-e6e8c6c6f0ca", "cloud": { "provider": "aws" }, "cloud.provider": "aws", "@timestamp": "2024-02-06T15:20:13Z", "ecs": { "version": "8.0.0" }, "cloud.account.id": "461485115270", "aws.firehose.parameters.es_datastream_name": "logs-aws.cloudfront_logs-default", "data_stream.dataset": "aws.cloudfront_logs", "aws.kinesis.name": "Elastic-test", "event": { "original": "1707232747.851\t43.133.38.182\t0.455\t403\t642\tGET\thttp[twww.avyaanpandey.com](http://twww.avyaanpandey.com/)\t/\t537\tSIN5-C1\tux5iSeDweA1B42E1fnHI8XqmogAL6hmvSL--OotSDSQ5nqliPGY4Lw==\[td3jr12hh9w9vyl.cloudfront.net](http://td3jr12hh9w9vyl.cloudfront.net/)\t0.455\tHTTP/1.1\tIPv4\tMozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1\t-\t-\t-\tError\t-\t-\t-\tError\t-\t-\tapplication/xml\t-\t-\t-\t40168\tError\tSG\tgzip\ttext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7\t\tHost:www.avyaanpandey.com%0AUser-Agent:Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1%0AAccept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7%0AAccept-Encoding:gzip%0AAccept-Language:zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7%0ACache-Control:no-cache%0AConnection:keep-alive%0APragma:no-cache%0AUpgrade-Insecure-Requests:1%0AConnection:close%0A\tHost%0AUser-Agent%0AAccept%0AAccept-Encoding%0AAccept-Language%0ACache-Control%0AConnection%0APragma%0AUpgrade-Insecure-Requests%0AConnection%0A\t10\tEJC0FPBIT5SPQ[td3jr12hh9w9vyl.cloudfront.net](http://td3jr12hh9w9vyl.cloudfront.net/)\t-\t-\t132203\n", "kind": "event", "category": "web", "type": [ "access" ] } }, "fields": { "cloud.region": [ "us-east-1" ], "event.category": [ "web" ], "aws.firehose.arn": [ "arn:aws:firehose:us-east-1:461485115270:deliverystream/Elastic-test" ], "data_stream.namespace": [ "default" ], "aws.kinesis.type": [ "deliverystream" ], "data_stream.type": [ "logs" ], "aws.firehose.request_id": [ "9b0ee118-8367-4064-a297-e6e8c6c6f0ca" ], "cloud.provider": [ "aws", "aws" ], "@timestamp": [ "2024-02-06T15:20:13.000Z" ], "event.module": [ "aws" ], "aws.firehose.parameters.es_datastream_name": [ "logs-aws.cloudfront_logs-default" ], "aws.kinesis.name.text": [ "Elastic-test" ], "cloud.account.id": [ "461485115270" ], "ecs.version": [ "8.0.0" ], "error.message": [ "Provided Grok expressions do not match field value: [1707232747.851\t43.133.38.182\t0.455\t403\t642\tGET\thttp\twww.avyaanpandey.com\t/\t537\tSIN5-C1\tux5iSeDweA1B42E1fnHI8XqmogAL6hmvSL--OotSDSQ5nqliPGY4Lw==\td3jr12hh9w9vyl.cloudfront.net\t0.455\tHTTP/1.1\tIPv4\tMozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1\t-\t-\t-\tError\t-\t-\t-\tError\t-\t-\tapplication/xml\t-\t-\t-\t40168\tError\tSG\tgzip\ttext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7\t\tHost:www.avyaanpandey.com%0AUser-Agent:Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1%0AAccept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7%0AAccept-Encoding:gzip%0AAccept-Language:zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7%0ACache-Control:no-cache%0AConnection:keep-alive%0APragma:no-cache%0AUpgrade-Insecure-Requests:1%0AConnection:close%0A\tHost%0AUser-Agent%0AAccept%0AAccept-Encoding%0AAccept-Language%0ACache-Control%0AConnection%0APragma%0AUpgrade-Insecure-Requests%0AConnection%0A\t10\tEJC0FPBIT5SPQ\td3jr12hh9w9vyl.cloudfront.net\t-\t-\t132203\n]" ], "data_stream.dataset": [ "aws.cloudfront_logs" ], "event.type": [ "access" ], "aws.kinesis.name": [ "Elastic-test" ], "event.kind": [ "event" ] }, "ignored_field_values": { "event.original": [ "1707232747.851\t43.133.38.182\t0.455\t403\t642\tGET\thttp[twww.avyaanpandey.com](http://twww.avyaanpandey.com/)\t/\t537\tSIN5-C1\tux5iSeDweA1B42E1fnHI8XqmogAL6hmvSL--OotSDSQ5nqliPGY4Lw==\[td3jr12hh9w9vyl.cloudfront.net](http://td3jr12hh9w9vyl.cloudfront.net/)\t0.455\tHTTP/1.1\tIPv4\tMozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1\t-\t-\t-\tError\t-\t-\t-\tError\t-\t-\tapplication/xml\t-\t-\t-\t40168\tError\tSG\tgzip\ttext/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7\t\tHost:www.avyaanpandey.com%0AUser-Agent:Mozilla/5.0%20(iPhone;%20CPU%20iPhone%20OS%2013_2_3%20like%20Mac%20OS%20X)%20AppleWebKit/605.1.15%20(KHTML,%20like%20Gecko)%20Version/13.0.3%20Mobile/15E148%20Safari/604.1%0AAccept:text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,/;q=0.8,application/signed-exchange;v=b3;q=0.7%0AAccept-Encoding:gzip%0AAccept-Language:zh-CN,zh;q=0.9,en-US;q=0.8,en;q=0.7%0ACache-Control:no-cache%0AConnection:keep-alive%0APragma:no-cache%0AUpgrade-Insecure-Requests:1%0AConnection:close%0A\tHost%0AUser-Agent%0AAccept%0AAccept-Encoding%0AAccept-Language%0ACache-Control%0AConnection%0APragma%0AUpgrade-Insecure-Requests%0AConnection%0A\t10\tEJC0FPBIT5SPQ[td3jr12hh9w9vyl.cloudfront.net](http://td3jr12hh9w9vyl.cloudfront.net/)\t-\t-\t132203\n" ] } }