[BUG] Graylog is not able to locate IP (geolocalisation error).

pcuq-ads commented 2 years ago

Environment:

Platform: RS OPS Orange Cloud (F4)
Configuration: System 1.0.1

Current Behavior: Graylog generates error linked to IP Geolocalisation.

Expected Behavior: Graylog does not not generate errors.

Steps To Reproduce: Example: steps to reproduce the behavior:

Get Graylog error logs.
See error about the impossibility to locate IP.
See error about GeoLite2-City.mmdb missing file.

Test execution artefacts (i.e. logs, screenshots…) N/A

Whenever possible, first analysis of the root cause N/A

Bug Generic Definition of Ready (DoR)

[X] The affect version in which the bug has been found is mentioned
[X] The context and environment of the bug is detailed
[X] The description of the bug is clear and unambiguous
[X] The procedure (steps) to reproduce the bug is clearly detailed
[ ] The tested User Story / features is linked to the bug if available
[ ] Logs are attached if available
[ ] A data set attached if available

Bug Generic Definition of Done (DoD)

[ ] the modification implemented (the solution to fix the bug) is described in the bug.
[ ] Unit tests & Continuous integration performed - Test results available - Structural Test coverage reported by SONAR
[ ] Code committed in GIT with right tag or Analysis/Trade Off documentation up-to-date in reference-system-documentation repository
[ ] Code is compliant with coding rules (SONAR Report as evidence)
[ ] Acceptance criteria of the related User story are checked and Passed

pcuq-ads commented 2 years ago

Here is the last comment from @suberti-ads (get from story #379)

Dear @wruffine-csgroup , If i well understand , i should copy GeoLite2-City.mmdb on graylog pod server

I downloaded GeoLite2-City.mmdb file on following link : https://www.maxmind.com/en/accounts/717043/geoip/downloads

I have somes questions:

Should i copy manually it in graylog server pod ? I can copy it in same directory find in content pack in server pods but this update will be lost after each graylog restart ? How should we save this update ?

Enable Geo-Location Processor ? I see on our configuration lookup-table, Caches and data adaptater already created. So i think it is already installed and activate but it only missing GeoLite2-City.mmdb file?

wruffine-csgroup commented 2 years ago

@suberti-ads

You were right, this was not the most efficient method.

The following solution was tested and approved on the IVV cluster, and added to the infrastructure repository in the pull request COPRS/infrastructure#92:

in the graylog values, add the following paragraph after the metrics port description:

{% if graylog.geolite2db_uri %}
geoip:
enabled: true
mmdbUri: {{ graylog.geolite2db_uri }}
{% endif %}

in your generated_values.yaml file, in the graylog section, add the variable:

geolite2db_uri: https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOURLICENSEKEY&suffix=tar.gz

Deploy graylog once again
In the IHM, go to Settings -> Configuration, at the bottom of the page, update the values for the plugin Geo-Location Processor
- enable the plugin
- change the MaxMind path to: /usr/share/graylog/geoip/GeoLite2-City.mmdb

This configuration is the one given in the pull request COPRS/infrastructure#92.

LAQU156 commented 2 years ago

IVV_CCB_2022_w23 : Accepted CS, no more action on CS side, only OPS

suberti-ads commented 2 years ago

Dear @wruffine-csgroup ,

i edit REPO/app/graylog/values.yaml with wanted value:

suberti@AUSY-DELL:~/Documents/GitRepository/RefSys/rs-config/infrastructure$ cat ../apps/graylog/values.yaml | grep -A 5 -B 4 "geoip:"
        port: 9833
        protocol: TCP

  {% if graylog.geolite2db_uri %}
  geoip:
    enabled: true
    mmdbUri: {{ graylog.geolite2db_uri }}
  {% endif %}

  env:

also, I change my /inventory/sample/group_vars/all/generated_inventory_vars.yaml with

graylog:
  oidc_client_secret: XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX
  operator_password: YYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYYY
  geolite2db_uri: https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOURLICENSEKEY&suffix=tar.gz

but i failed to deploy graylog:

    "stderr": "The StatefulSet \"graylog\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden",
    "stderr_lines": [
        "The StatefulSet \"graylog\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"
    ],

On ops platform, i see a new pod which seems try to connect to graylog-master :


safescale  gw-ops-cluster  ~  kubectl get po -n security -o wide | grep graylog 
graylog-0                                                         2/2     Running     0              49d     10.244.78.36     ops-cluster-node-3     <none>           <none>
graylog-1                                                         2/2     Running     0              49d     10.244.110.32    ops-cluster-node-12    <none>           <none>
graylog-provisioner-job--1-kw7qf                                  0/1     Completed   0              11m     10.244.244.130   ops-cluster-node-13    <none>           <none>
 safescale  gw-ops-cluster  ~  kubectl logs -n security graylog-provisioner-job--1-kw7qf | tail -10
* Mark bundle as not supporting multiuse
< HTTP/1.1 401 Unauthorized
* Authentication problem. Ignoring this.
< www-authenticate: Basic realm="Graylog Server"
< x-graylog-node-id: 13b8688a-f773-45b1-a0dc-5b9bfd00e6d0
< content-length: 0
< date: Wed, 08 Jun 2022 16:39:11 GMT
< 
100   210    0     0  100   210      0  11186 --:--:-- --:--:-- --:--:-- 11666
* Connection #0 to host graylog-master.security.svc.cluster.local left intact

wruffine-csgroup commented 2 years ago

Dear @suberti-ads, According to this error:

"The StatefulSet \"graylog\" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'template', 'updateStrategy' and 'minReadySeconds' are forbidden"`

You need to delete the statefulset graylog before re-applying the new values. Deleting the graylog statefulset will not lead to data loss since graylog data are stored in the mongodb database.

suberti-ads commented 2 years ago

Dear @wruffine-csgroup , thanks! GeoCOnfiguration I follow procedure, Graylog pod have been redeployed but i don't find /usr/share/graylog/geoip/GeoLite2-City.mmdb file in them: (there is no directory geoip)

 safescale  gw-ops-cluster  ~  kubectl exec -ti -n security graylog-0 -c graylog-server -- bash
graylog@graylog-0:~$ ls /usr/share/graylog/geoip/GeoLite2-City.mmdb
ls: cannot access '/usr/share/graylog/geoip/GeoLite2-City.mmdb': No such file or directory
graylog@graylog-0:~$ cd /usr/share/graylog/geoip/
bash: cd: /usr/share/graylog/geoip/: No such file or directory
graylog@graylog-0:~$ cd /usr/share/graylog/
graylog@graylog-0:~$ ls
LICENSE  README.markdown  UPGRADING.rst  bin  config  data  graylog.conf.example  graylog.conf.subst  graylog.jar  log  plugin  plugins-default  plugins-merged

Moreover there is always missing error on Graylog logs:

 [...]
2022-06-09 12:47:34,954 INFO    [GeoIpProcessor] - Updating GeoIP resolver engine - GeoIpResolverConfig{enabled=false, dbType=MAXMIND_CITY, dbPath=/usr/share/graylog/geoip/GeoLite2-City.mmdb} - {}
2022-06-09 12:47:34,954 WARN    [GeoIpResolverEngine] - GeoIP database file does not exist: /usr/share/graylog/geoip/GeoLite2-City.mmdb - {}
2022-06-09 12:47:34,954 WARN    [GeoIpResolverEngine] - GeoIP database file does not exist: /usr/share/graylog/geoip/GeoLite2-City.mmdb - {}
[...]
2022-06-09 12:48:46,153 WARN    [MaxmindDataAdapter] - Unable to look up city data for IP address /XXX.XXX.XXX.XXX, returning empty result. - {}
java.lang.NullPointerException: null
[...]

wruffine-csgroup commented 2 years ago

Dear @suberti-ads ,

In the default uri link I gave for the geolite2 db, there is a value YOURLICENSEKEY:

https://download.maxmind.com/app/geoip_download?edition_id=GeoLite2-City&license_key=YOURLICENSEKEY&suffix=tar.gz

It corresponds to the licence key you get from the maxmind website. Have you changed the value? If not, it's a fault on my end. I should have explicited it.

suberti-ads commented 2 years ago

Dear @wruffine-csgroup ,

OK sorry for my missunderstanding, on https://www.maxmind.com/en/home i create a new licence ( i choose Option "No" for GeoIP Update)

I update also my /inventory/sample/group_vars/all/generated_inventory_vars.yaml with this new license.

But after redeployment result was the same: no file and error in logs.

Should i use update option during license creation ? if yes which version has been used by graylog <3.11 or >3.1.1 ?

Sorry if my question are foolish but i am not familiar with this.

wruffine-csgroup commented 2 years ago

Dear @suberti-ads

There are no foolish questions, only insufficient or bad informations. In this case, it was the latter. After further tests, I noticed the indentation I gave was not the right one, thus the geoip directive was ignored.

The correct indentation is the following:

{% if graylog.geolite2db_uri %}
  geoip:
    enabled: true
    mmdbUri: {{ graylog.geolite2db_uri }}
{% endif %}

Let me know if this works for you.

suberti-ads commented 2 years ago

Dear @wruffine-csgroup , There is still not working :( Hereafter my new indentation :

        protocol: TCP

{% if graylog.geolite2db_uri %}
  geoip:
    enabled: true
    mmdbUri: {{ graylog.geolite2db_uri }}
{% endif %}

  env:

wruffine-csgroup commented 2 years ago

Dear @suberti-ads,

Could you redeploy graylog by adding the -e debug option in the ansible command? This way, the kustomization files are not deleted and the name of the kustomization folder (/tmp/ansible_XXXXXXX) should be given to you at the end of the procedure. In this kustomization folder, can you then checkout the values.yaml file and check if the geoip values are present?

suberti-ads commented 2 years ago

My previous deployment command was:

 ansible-playbook apps.yaml -i inventory/sample/hosts.ini -e app=graylog

So i add -e debug option:

ansible-playbook apps.yaml -i inventory/sample/hosts.ini -e app=graylog -e debug

But it remove tmp dir

TASK [app-installer : apps - graylog | Remove resources tmp dir] ******************************************************************************************************************************************
task path: /home/suberti/Documents/GitRepository/RefSys/rs-config/infrastructure/roles/app-installer/tasks/install_app.yaml:88
[...]
changed: [gw-ops-cluster] => {
    "changed": true,
    "diff": {
        "after": {
            "path": "/tmp/ansible.hofi21hak8s_resources_graylog",
            "state": "absent"
        },
[....]

wruffine-csgroup commented 2 years ago

I'm sorry, again, mistake on my end: It's -e debug=true and not just -e debug

suberti-ads commented 2 years ago

Still sorry, this time tmp file seems not delete but i don't find it:

TASK [app-installer : apps - graylog | Remove resources tmp dir] ******************************************************************************************************************************************
task path: /home/suberti/Documents/GitRepository/RefSys/rs-config/infrastructure/roles/app-installer/tasks/install_app.yaml:88
skipping: [gw-ops-cluster] => {
    "changed": false,
    "skip_reason": "Conditional result was False"
}
Read vars_file 'collections/kubespray/roles/kubespray-defaults/defaults/main.yaml'

TASK [app-installer : apps - graylog | Warn about not deleted tmp folder] *********************************************************************************************************************************
task path: /home/suberti/Documents/GitRepository/RefSys/rs-config/infrastructure/roles/app-installer/tasks/install_app.yaml:96
ok: [gw-ops-cluster] => {
    "msg": [
        "DEBUG: the temporary folder containing the k8s resources has been kept tfor debugging purpose",
        "/tmp/ansible.36iz96klk8s_resources_graylog",
        "DO NOT FORGET TO DELETE IT WHEN YOU ARE DONE"
    ]
}
Read vars_file 'collections/kubespray/roles/kubespray-defaults/defaults/main.yaml'
META: ran handlers
Read vars_file 'collections/kubespray/roles/kubespray-defaults/defaults/main.yaml'
META: ran handlers

suberti@AUSY-DELL:~/Documents/GitRepository/RefSys/rs-config/infrastructure$ sudo find /tmp -iname *36iz96klk8*
suberti@AUSY-DELL:~/Documents/GitRepository/RefSys/rs-config/infrastructure$ find /home/suberti/ -iname *36iz96klk8*

Where this file was been generated ? i don't find it on /tmp directory or on current partition.

wruffine-csgroup commented 2 years ago

No worries.

The tmp file is generated on the platform.

When you deploy applications, Ansible uses kustomize to create the files that will be deployed on the platform. Ansible then copies these files on the gateway and deploy them from there. Then, if the debug tag is not added, Ansible deletes the files.

suberti-ads commented 2 years ago

This issue was due to wrong parameters on infrastructure/inventory/sample/group_vars/all/generated_inventory_vars.yaml which not target my rs-config application directory.

Thanks for you help to find this. Configuration change successfully deployed.

COPRS / rs-issues

[BUG] Graylog is not able to locate IP (geolocalisation error). #403