docker - logstash cant find conf files

[lcia-projects]

i can get pfelk to run natively (not docker) just fine.. but i'd really prefer a docker installation. I've watched the youtube videos..

I get this error: logstash | [INFO ] 2021-03-20 15:39:28.963 [Agent thread] configpathloader - No config files found in path {:path=>"/etc/pfelk/conf.d/*.conf"}

where do i need to put the folders for it to locate..

• ./etc/logstash/config/:/usr/share/logstash/config:ro
• ./etc/logstash/conf.d/:/etc/pfelk/conf.d:ro
• ./etc/logstash/conf.d/patterns/:/etc/pfelk/patterns:ro
• ./etc/logstash/conf.d/databases/:/etc/pfelk/databases:ro so its my thinking that it should just look in your local folder where you pulled the git.

where am i missing?

[a3ilson]

Thanks for the feedback! I added to my list of things to do and/or improve but will clarify in the future. To answer your question and forgive me if this is trivial:

The first portion is the location on your host followed by where within the docker container the files/folders will be accessible

This is my current setup:

- /etc/pfelk/config:/usr/share/logstash/config:ro

The /etc/pfelk/config is were the folder is on my actual host separated by : followed by where it needs to be for pfelk to work /usr/share/logstash/config:ro

invoking a ls * on my system results in the following where the path is /etc/pfelk:

conf.d:
01-inputs.conf  03-filter.conf  20-interfaces.conf  35-rules-desc.conf  45-cleanup.conf
02-types.conf   05-apps.conf    30-geoip.conf       36-ports-desc.conf  50-outputs.conf

config:
logstash.yml  pipelines.yml

databases:
private-hostnames.csv  rule-names.csv  service-names-port-numbers.csv

patterns:
openvpn.grok  pfelk.grok

This repository was defined for TravisCI to validate a working instance. I am using this docker-compose.yml file.

[lcia-projects]

thank you for the quick response. to clarify.

I need to make a folder in my host machine (server) /etc called "pfelk" so.. /etc/pfelk

within that folder will be the files and folders listed in your response? i swore i tried that with no luck, but i'll try again.

thanks again, great product, thank you for your hard work

[a3ilson]

Correct, all files are located within the /etc/pfelk/ path:

etc/pfelk/
├── conf.d
│   ├── 01-inputs.conf
│   ├── 02-types.conf
│   ├── 03-filter.conf
│   ├── 05-apps.conf
│   ├── 20-interfaces.conf
│   ├── 30-geoip.conf
│   ├── 35-rules-desc.conf
│   ├── 36-ports-desc.conf
│   ├── 45-cleanup.conf
│   └── 50-outputs.conf
├── config
│   ├── logstash.yml
│   └── pipelines.yml
├── databases
│   ├── private-hostnames.csv
│   ├── rule-names.csv
│   └── service-names-port-numbers.csv
└── patterns
    ├── openvpn.grok
    └── pfelk.grok

[lcia-projects]

if thats the case.. how does that corrispond with whats in the docker-compose file: ...

logstash:
    container_name: logstash
    build:
      context: logstash/
      args:
        ELK_VERSION: $ELK_VERSION
    environment:
      LS_JAVA_OPTS: -Xmx1G -Xms1G
    restart: unless-stopped
    volumes:
      - ./etc/logstash/config/:/usr/share/logstash/config:ro       
      - ./etc/logstash/conf.d/:/etc/pfelk/conf.d:ro
      - ./etc/logstash/conf.d/patterns/:/etc/pfelk/patterns:ro
      - ./etc/logstash/conf.d/databases/:/etc/pfelk/databases:ro

i got it working but i had to change the "./etc/logstash/" to /etc/pfelk/...

[a3ilson]

It does not and apologize for the confusion. Let me clarify:

The docker-compose.yml for this repo (pfelk/docker) works if following the instructions (1-4).

I am, in the future, going to combine both repos (pfelk/docker and pfelk/pfelk). I also have another docker-compose.yml that I am using on the pfelk/pfelk repository which is my current setup and what my prior response was related too - sorry for the confusion.

So you can download the files from the pfelk/pfelk and utilize that docker-compose.yml file or your current setup from the pfelk/docker repo. I'll attempt to explain what/how the docker-compose.yml is referencing the files:

Within the docker-compose.yml file volumes are referencing files (folders) on your host to the corresponding docker container(s). In the docker-compose.yml from the pfelk/docker repo, lines 82-85 define these. You'll note that each path contains a preceding dot. The preceding dot represents the current path.

• Example: you are at /usr/home/user/ and within that path is your docker-compose.yml (e.g. /usr/home/user/docker-compose.yml). Those references simple mean that you should also have etc/logstash/config/, etc/logstash/conf.d/ and so forth:

  /usr/home/user/
  ├── docker-compose.yml
  ├── conf.d
  │   ├── 01-inputs.conf
  │   ├── 02-types.conf
  │   ├── 03-filter.conf
  │   ├── 05-apps.conf
  │   ├── 20-interfaces.conf
  │   ├── 30-geoip.conf
  │   ├── 35-rules-desc.conf
  │   ├── 36-ports-desc.conf
  │   ├── 45-cleanup.conf
  │   └── 50-outputs.conf
  ├── config
  │   ├── logstash.yml
  │   └── pipelines.yml
  ├── databases
  │   ├── private-hostnames.csv
  │   ├── rule-names.csv
  │   └── service-names-port-numbers.csv
  └── patterns
  ├── openvpn.grok
  └── pfelk.grok

If you wish to relocate those files at an alternate location, you'll need to define that in the docker-compose.yml file. Example:

      - /usr/myfolder/:/usr/share/logstash/config:ro       
      - /usr/myfolder/conf.d/:/etc/pfelk/conf.d:ro
      - /usr/myfolder/conf.d/patterns/:/etc/pfelk/patterns:ro
      - /usr/myfolder/conf.d/databases/:/etc/pfelk/databases:ro

In the example above we elected to use an absolute path and those files need to reside in the specified location for docker-compose.yml (i.e. Logstash) to properly run.

Final note: the pfelk/docker and pfelk/pfelk files are completely interchangeable - use one or the other.

Feel free to shot me an email (support@pfelk.com) and discuss via audio means if needed.

Linux Paths: / absolute path . Relative path - current directory .. Relative path - parent directory

[lcia-projects]

that makes total sense. i was just using the wrong compose file. Thank you for your quick responses and explanation.

[a3ilson]

No worries and let us know if you have any additional feedback/issues.