Issues reading nginx logs

[marijusGood]

Hi team,

I seem to have the same problem as described before where none of my logs get parsed. I am deploying everything within a k8s cluster. The output of my cscli metrics

╭──────────────────────────────────────────────────────────────┬────────────┬──────────────┬────────────────┬────────────────────────┬───────────────────╮
│                            Source                            │ Lines read │ Lines parsed │ Lines unparsed │ Lines poured to bucket │ Lines whitelisted │
├──────────────────────────────────────────────────────────────┼────────────┼──────────────┼────────────────┼────────────────────────┼───────────────────┤
│ file:/var/log/containers/nginx-ingress-microk8s-controller-l │ 1          │ -            │ 1              │ -                      │ -                 │
│ dlp5_ingress_nginx-ingress-microk8s-ace27de928e788d77337c3c2 │            │              │                │                        │                   │
│ a8a60698f9b3a42109a834fb0d88a891eb3f04aa.log                 │            │              │                │                        │                   │
╰──────────────────────────────────────────────────────────────┴────────────┴──────────────┴────────────────┴────────────────────────┴───────────────────╯

Parser Metrics:
╭────────────────────────────────┬──────┬────────┬──────────╮
│             Parsers            │ Hits │ Parsed │ Unparsed │
├────────────────────────────────┼──────┼────────┼──────────┤
│ child-crowdsecurity/nginx-logs │ 3.00k    │ -      │ 3.00k        │
│ crowdsecurity/nginx-logs       │ 3.00k    │ -      │ 3.00k        │
│ crowdsecurity/non-syslog       │ 3.00k    │ 3.00k      │ -        │
╰────────────────────────────────┴──────┴────────┴──────────╯

when my nginx logs look like this:

83.150.2.35 - - [28/Jun/2024:14:15:08 +0000] "GET /selfTestPci586.d HTTP/2.0" 200 53 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 32 0.204 [es-dev-es-dev-443] [] 10.1.187.164:8443 53 0.203 200 d77633989c1e405f9321cfa84f545197
83.150.2.35 - - [28/Jun/2024:14:15:08 +0000] "GET /selfTestPci586.d HTTP/2.0" 200 53 "-" "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/126.0.0.0 Safari/537.36" 32 0.183 [es-dev-es-dev-443] [] 10.1.187.164:8443 53 0.182 200 733fed93485463aa28918026e93ced5b

and my values.yaml is:

agent:
  acquisition:
    - namespace: ingress
      # The pod name
      podName: nginx-ingress-microk8s-controller-*
      program: nginx

  # Those are ENV variables
  env:
  - name: DISABLE_ONLINE_API
    value: "true"
  - name: COLLECTIONS
    value: "crowdsecurity/nginx"
lapi:
  env:
    - name: DISABLE_ONLINE_API
      value: "true"

I tried setting container_runtime to cri, docker, container and ngnix with no avail. I also tried to add:

- name: PARSERS
    value: "crowdsecurity/cri-logs"

but it did not help. Help regarding this topic would be very appreciated, thanks!

[github-actions[bot]]

@marijusGood: Thanks for opening an issue, it is currently awaiting triage.

If you haven't already, please provide the following information:

• kind : bug, enhancementor documentation
• area : agent, appsec, configuration, cscli, local-api

In the meantime, you can:

1. Check Crowdsec Documentation to see if your issue can be self resolved.
2. You can also join our Discord.
3. Check Releases to make sure your agent is on the latest version.

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/helm-charts/blob/main/.github/governance.yaml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the forked project [rr404/oss-governance-bot](https://github.com/rr404/oss-governance-bot) repository.

[github-actions[bot]]

@marijusGood: There are no 'kind' label on this issue. You need a 'kind' label to start the triage process.

• /kind bug
• /kind documentation
• /kind enhancement

Details

I am a bot created to help the [crowdsecurity](https://github.com/crowdsecurity) developers manage community feedback and contributions. You can check out my [manifest file](https://github.com/crowdsecurity/helm-charts/blob/main/.github/governance.yaml) to understand my behavior and what I can do. If you want to use this for your project, you can check out the forked project [rr404/oss-governance-bot](https://github.com/rr404/oss-governance-bot) repository.

[LaurenceJJones]

Hmmm so I tested the logs within a debugger and its parses

Can you get the log lines directly from the file from what CrowdSec see's

[marijusGood]

Sure thing!

So the file that it tries to parse is /var/log/containers/nginx-in gress-microk8s-controller-ldlp5_ingress_nginx-ingress-microk8s-ace27de928e788d77337c3c2a8a60698f9b3a42109a834fb0d88a891eb3f04aa.log

I have attached a file that contains a part of the lines that are in the mentioned file ngnix.log

let me know if you need more information and thanks for the fast reply!

[LaurenceJJones]

Sure thing!

So the file that it tries to parse is /var/log/containers/nginx-in gress-microk8s-controller-ldlp5_ingress_nginx-ingress-microk8s-ace27de928e788d77337c3c2a8a60698f9b3a42109a834fb0d88a891eb3f04aa.log

I have attached a file that contains a part of the lines that are in the mentioned file ngnix.log

let me know if you need more information and thanks for the fast reply!

So looking at the logs that is cri format, so the container_runtime should be containerd which sets the correct s00 and then the program should still be nginx

Please ensure the value is set as containerd as if you miss any letters it will not work as intended

[marijusGood]

Thank you for the help, it worked!

I have a small question regarding email notifications. In my helm values I have added:

config:
  notifications:
    email.yaml: |
      type: email
      name: email_default
      log_level: info
      smtp_host: XXXXXXXX
      smtp_username: XXXXXXXX
      smtp_password: XXXXXXXX
      smtp_port: 587
      auth_type: login
      sender_name: "CrowdSec"
      sender_email: XXXXXXXX
      email_subject: "CrowdSec Notification"
      receiver_emails:
        - XXXXXXXX
        - XXXXXXXX
    slack.yaml: ""
    http.yaml: ""
    splunk.yaml: ""

and running kubectl get configmap -n crowdsec crowdsec-notifications -o yaml I see:

apiVersion: v1
data:
  email.yaml: |
    type: email
    name: email_default
    log_level: info
    smtp_host: XXXXXXXX
    smtp_username: XXXXXXXX
    smtp_password: XXXXXXXX
    smtp_port: 587
    auth_type: login
    sender_name: "CrowdSec"
    sender_email: XXXXXXXX
    email_subject: "CrowdSec Notification"
    receiver_emails:
      - XXXXXXXX
      - XXXXXXXX
kind: ConfigMap
metadata:
  annotations:
    meta.helm.sh/release-name: crowdsec
    meta.helm.sh/release-namespace: crowdsec
  creationTimestamp: "2024-07-05T14:03:50Z"
  labels:
    app.kubernetes.io/managed-by: Helm
  name: crowdsec-notifications
  namespace: crowdsec
  resourceVersion: "23367747"
  uid: 936867d2-3036-4d59-addf-97d9a29f0a37

but if I exec into the pod and do cat /etc/crowdsec/notifications/email.yaml the values are not there and the email is not sent and in /etc/crowdsec/profiles.yaml the information also has not changed

Thanks for the help

[LaurenceJJones]

Did you run a helm chart update using the new values.yaml? I dont know how the profiles are updated, as you would need to modify that to enable to email.yaml

[marijusGood]

yes, I have run helm update comand: helm upgrade --install crowdsec crowdsec/crowdsec -f crowdsec-values-dev.yaml --create-namespace -n crowdsec

This can be seen as crowdsec-notifications configmap has been updated with the values that I have specified