Apache logs support in Observability

[nidhisinghai]

Hello @abhaypersistent

The issue involves the below following implementations:

1. Analysis and creation on metadata
2. Index creation
3. Specs creation
4. Generating dummy data for development

Thank You, cc : @abasatwar @spattnaik

[kuldeepbhise]

Hi @anirudha As part of custom log, we have made some changes in config file. i.e apache2.conf Step 1 We have added below log format entry, which will help to get metadata from the server.

LogFormat " {"timestamp":"%{%d/%b/%Y %T}t.%{msec_frac}t %{%z}t", "process_time_taken_microseconds":"%D", "filename":"%f", "remoteIP":"%a", "host":"%V", "request_url":"%U", "query":"%q", "request_method":"%m", "redirected_request_status":"%>s", "userAgent":"%{User-agent}i", "referer":"%{Referer}i", "localIP":"%A", "cookie_varname":"%{VARNAME}C", "content_of_varname":"%{VARNAME}e", "remote_host_name":"%h", "request_protocol":"%H", "keepalive_request":"%k", "remote_logname":"%l", "canonical_port":"%p", "query_string":"%q", "first_line_request":"%r", "time_to_serve_Request":"%T", "authenticated_user":"%u", "canonical_servername":"%v", "connection_response_status":"%X", "byte_received":"%I", "byte_sent":"%O", "pid_child":"%P", "thread_id":"%{format}P", "status":"%s" }" custom

Step2 Updated the fluentd conf file with below code (i.e. /home/kbhise/fluent/fluent.conf)

<source>
@type tail
 <parse>
   @type json
   types bytes_sent:integer,byte_received:integer
   #time_key time
   #time_format %time
   #time_format %d/%b/%Y:%H:%M:%S %z
  </parse>
 path /var/log/apache2/custom.log
 pos_file /home/kbhise/apache_pos2
 tag  apache2
</source>

<match apache2>
 @type opensearch
 ssl_verify false
 host localhost
 port 9200
 index_name apache_log_final
 verify_os_version_at_startup false
 suppress_type_name true
 #include_timestamp true
</match>

Step 3 Once we start the fluentd and apache server, opensearch will read the logs and create the index. Index created in opensearch-dashboard is shown in below screen shot.

CC: @abasatwar @spattnaik

[kuldeepbhise]

Hi @anirudha, We have created the Apache2 dashboard which is having few visualization examples. Please find the supporting video given below.

https://user-images.githubusercontent.com/105786991/172599882-448384be-adb4-4e01-b381-8c0ca3e8080a.mp4

CC: @abasatwar @spattnaik

[kuldeepbhise]

Configuring Fluent-bit to read apache2 logs.

[kuldeepbhise]

Hi @anirudha, We have configured the fluent-bit to ingest apache2 logs. (access.log and custom.log) 1. To read access.log (default), We have used default regex format for reading and writing the logs. Step 1 : In this step we need to update the fluent configuration file (fluent-bit.conf)

    # Parsers File
    # ============
    # specify an optional 'Parsers' configuration file
    parsers_file parsers.conf

 [INPUT]
    Name tail
    Path /var/log/apache2/access.log
    Tag  apache_demo

[FILTER]
    Name parser
    Match apache_demo
    Parser apache2
    Key_Name log
    Reserve_Data True

[OUTPUT]
    Name  opensearch
    Match *
    Host  0.0.0.0
    Port  9200
    Index fluent-bit_apache2_accesslog
    Suppress_Type_Name on

Step 2 : In this step we need to update the parser file (parsers.conf)

[PARSER]
Name   apache2
Format regex
Regex  ^(?<host>[^ ]*) [^ ]* (?<user>[^ ]*) \[(?<time>[^\]]*)\] "(?<method>\S+)(?: +(?<path>[^ ]*) +\S*)?" (?<code>[^ ]*) (?<size>[^ ]*)(?: "(?<referer>[^\"]*)" "(?<agent>.*)")?$
Time_Key time
Time_Format %d/%b/%Y:%H:%M:%S %z

Result: Index created in the opensearch: fluent-bit_apache2_accesslog

        "_index" : "fluent-bit_apache2_accesslog",
        "_id" : "MUtOTYEB2h1zzpv51GQ_",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-06-10T11:10:16.000Z",
          "host" : "172.24.176.1",
          "user" : "-",
          "method" : "GET",
          "path" : "/",
          "code" : "200",
          "size" : "10168",
          "referer" : "-",
          "agent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36"

2.To read custom log, We have explicitly mentioned the json format for reading and writing the logs. Step 1 : In this step we need to update the fluent configuration file (fluent-bit.conf)

 [INPUT]
    Name tail
    Path /var/log/apache2/custom.log
    Tag  apache_demo
    parser apache2

[OUTPUT]
    Name  opensearch
    Match *
    Host  0.0.0.0
    Port  9200
    Index fluent-bit_apache2_custom_log
    Suppress_Type_Name on

Step 2 : In this step we need to update the parser file (parsers.conf)

[PARSER]
    Name apache2
    Format json
    Time_Key time

Result: Index created in the opensearch: fluent-bit_apache2_custom_log

        "_index" : "fluent-bit_apache2_custom_log",
        "_id" : "XKt4XIEBBfyf-hDPA1oj",
        "_score" : 1.0,
        "_source" : {
      {
        "_index" : "fluent-bit_apache2_custom_log",
        "_id" : "X6t-XIEBBfyf-hDPJVp0",
        "_score" : 1.0,
        "_source" : {
          "@timestamp" : "2022-06-13T09:56:19.609Z",
          "timestamp" : "13/Jun/2022 15:26:18.481 +0530",
          "process_time_taken_microseconds" : "1128427",
          "filename" : "/var/www/wordpress/index.php",
          "remoteIP" : "172.24.176.1",
          "host" : "172.24.191.180",
          "request_url" : "/index.php",
          "query" : "",
          "request_method" : "GET",
          "redirected_request_status" : "200",
          "userAgent" : "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/102.0.0.0 Safari/537.36",
          "referer" : "-",
          "localIP" : "172.24.191.180",
          "cookie_varname" : "-",
          "content_of_varname" : "-",
          "remote_host_name" : "172.24.176.1",
          "request_protocol" : "HTTP/1.1",
          "keepalive_request" : "0",
          "remote_logname" : "-",
          "canonical_port" : "80",
          "query_string" : "",
          "first_line_request" : "GET / HTTP/1.1",
          "time_to_serve_Request" : "1",
          "authenticated_user" : "-",
          "canonical_servername" : "test-wordpress.com",
          "connection_response_status" : "+",
          "byte_received" : "455",
          "byte_sent" : "10168",
          "pid_child" : "2850",
          "thread_id" : "format",
          "status" : "200",
          "Peer_IP" : "172.24.176.1",
          "response_size" : "9839",
          "header_line" : "-",
          "log_error_request_id" : "-",
          "varname_content" : "-",
          "client_port" : "format",
          "note_varname_content" : "-",
          "handler_resonse" : "application/x-httpd-php",
          "time" : "format",
          "time_taken_to_serve_request" : "UNIT",
          "servername" : "172.24.191.180",
          "content_varname_request" : "-",
          "content_varname_response" : "-"
        }
}

[nidhisinghai]

Completed