System/Grok : Manage grok patterns

[Luubosi]

would it be possible to add new feature - System/Grok : Manage grok patterns ? i thing api is ok for this, :9000/api/api-browser/#!/System/Grok and application of this grok in extractor. Thanks for your work and reply.

[suzuki-shunsuke]

Thank you for your feedback. I don't know the structure of content_pack. Do you know it?

http://127.0.0.1:9000/api/api-browser/#!/System/Grok/listGrokPatterns_get_0

    {
      "name": "HTTPD_ERRORLOG",
      "pattern": "%{HTTPD20_ERRORLOG}|%{HTTPD24_ERRORLOG}",
      "content_pack": null,
      "id": "5d6a0af36df4af000de157e2"
    }

[suzuki-shunsuke]

I have created the PR #160 .

[suzuki-shunsuke]

I have released the new version v7.1.0 . Please check.

https://github.com/suzuki-shunsuke/go-graylog/blob/master/terraform/docs/grok_pattern.md

[Luubosi]

Hi there are errors, please check. Provider i have 7.1.0

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:58:06 Running command: terraform apply -input=false -auto-approve ?[31m ?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m ?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4": 5: pattern = "ERROR_COUNT=[{]Value=%{?[4mDATA?[0m:ERROR_COUNT}, LAST_REQUEST" ?[0m "DATA" is not a valid template control keyword. ?[0m?[0m

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:58:53 Running command: terraform apply -input=false -auto-approve ?[31m ?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m ?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4": 5: pattern = "ERROR_COUNT=[{]Value=%{?[4mGREEDYDATA?[0m:ERROR_COUNT}, LAST_REQUEST" ?[0m "GREEDYDATA" is not a valid template control keyword. ?[0m?[0m

[terragrunt] [C:\graylog-terraform] 2019/09/04 14:59:36 Running command: terraform apply -input=false -auto-approve ?[31m ?[1m?[31mError: ?[0m?[0m?[1mInvalid template control keyword?[0m ?[0m on 007-grok_pattern.tf line 5, in resource "graylog_grok_pattern" "amc_grok_type4": 5: pattern = "%{?[4mDATE?[0m}[- ]%{TIME}" ?[0m "DATE" is not a valid template control keyword. ?[0m?[0m

[suzuki-shunsuke]

Please share terraform configuration files. I think your pattern parameter is invalid.

You can test the pattern parameter with Web UI or API Browser.

• http://127.0.0.1:9000/system/grokpatterns
• http://127.0.0.1:9000/api/api-browser/#!/System/Grok/bulkUpdatePatternsFromTextFile_post_1

[suzuki-shunsuke]

To be honest I'm not familiar with the Grok pattern because I don't use this feature. So if you have any questions about the Grok pattern, please ask at https://community.graylog.org/ .

http://docs.graylog.org/en/3.0/pages/extractors.html#using-grok-patterns-to-extract-data

[suzuki-shunsuke]

I succeeded to create a Grok pattern with the following Terraform configuration.

resource "graylog_grok_pattern" "test" {
  name    = "test"
  pattern = "test"
}

[Luubosi]

Hi, for example. i use grok pattern. terraform configuration

resource "graylog_grok_pattern" "amc_grok_type4" { name = "AMCJSON" pattern = "test %{TIME}" }

in new grok pattern you can user grok patterns in list /system/grokpatterns for example

COMMONAPACHELOG | %{IPORHOST:clientip} %{HTTPDUSER:ident} %{USER:auth} [%{HTTPDATE:timestamp}] "(?:%{WORD:verb} %{NOTSPACE:request}(?: HTTP/%{NUMBER:httpversion})?|%{DATA:rawrequest})" %{NUMBER:response} (?:%{NUMBER:bytes}|-)

is using existing grok pattern DATA, NUMBER, USER.

[suzuki-shunsuke]

Oh, I got it. I can reproduce it.

$ cat grok_pattern.tf
resource "graylog_grok_pattern" "test2" {
  name    = "AMCJSON"
  pattern = "test %{TIME}"
}

$ terraform plan

Error: Invalid template control keyword

  on grok_pattern.tf line 3, in resource "graylog_grok_pattern" "test2":
   3:   pattern = "test %{TIME}"

"TIME" is not a valid template control keyword.

You should escape "%".

https://github.com/hashicorp/hcl2/blob/master/hcl/hclsyntax/spec.md#template-literals

resource "graylog_grok_pattern" "test2" {
  name    = "AMCJSON"
  pattern = "test %%{TIME}"
}

The interpolation and directive introductions are escaped by doubling their leading characters. The ${ sequence is escaped as $${ and the %{ sequence is escaped as %%{.

[suzuki-shunsuke]

I fix the document.

• https://github.com/suzuki-shunsuke/go-graylog/pull/165
• https://github.com/suzuki-shunsuke/go-graylog/blob/ee2955889b7c41b8621d90f0560ef8176ec3f35c/terraform/docs/grok_pattern.md

[Luubosi]

Hi, thanks for clarification. Now it works correct and grok patterns is ok. thanks for documentation, this is solution - pattern = "%%{DATE}[- ]%%{TIME}"