search

NginxProxyManager / nginx-proxy-manager

Docker container for managing Nginx proxy hosts with a simple, powerful interface
https://nginxproxymanager.com
MIT License
20.86k stars 2.42k forks source link

Integration with crowdsecurity/cs-nginx-bouncer #1131

Open Sparkxxx opened 3 years ago

Sparkxxx commented 3 years ago

Is your feature request related to a problem? Please describe. I would like to add protection for sites to NPM and be able to block IP's based on reputation/scenario using Crowdsecurity https://github.com/crowdsecurity which is a log analyzer and reactive firewall, something like fail2ban but in a much modern architecture and suitable for the containers world and having a distributed banlist generated from users around the world.

The question/feature request is how/where to add the https://github.com/crowdsecurity/cs-nginx-bouncer module to nginx config. This bouncer leverages nginx lua's API, namely access_by_lua_file. New/unknown IPs are checked against crowdsec API, and if request should be blocked, a 403 is returned to the user, and put in cache. The exact desired action can be configured in Crowdsec, like displaying captcas etc.

Describe the solution you'd like I would like to have an easy way (example maybe) to deploy the bouncer without breaking things in npm.

Describe alternatives you've considered None so far since it might take you minutes to offer a solution and it would take me hours to tweak things and break them :)

Additional context Please have a look at https://crowdsec.net/ to understand what is crowdsec and how it works. I use jc21/nginx-proxy-manager:latest for NPM and https://github.com/crowdsecurity/crowdsec/tree/master/docker for Crowdsec containers analyzing the nginx logs allready written by NPM on disk (docker map volumes ./npm_data/logs/:/var/log/nginx/ and have a quick look at acquis.yaml which tells crowdsec which logs to parse, mine looks like this): `filenames:

Thank you and hope you'd consider this as a valuable addition to NPM

chaptergy commented 3 years ago

Referencing https://github.com/jc21/nginx-proxy-manager/issues/39.

Sparkxxx commented 3 years ago

I have read that feature request but, let me justify better my request. As far as I know fail2ban works only on one machine so it would have to manipulate some iptables/ipsets or scripts to work over a network and this implies many moving parts. It is basically a stand alone application from before the container world and I'm not even speaking here about swarms. I know it's usefull, I'm using it but not in the container world where in my opinion it looks like a dinosaur. Crowdsec (runs on docker and has 3.3K stars on github) on the other hand is a more modern and advanced way of achieving the same goals as with fail2ban but in a distributed infrastructure with networking, VM's, containers, barebones, detecting atacks, redirecting, banning, serving captchas, etc., with a graphical interface and we love the GUIs. Bouncers and detectors don't even have to be on the same machine/container. Have a look at their agents/bouncers HUB https://hub.crowdsec.net/ and BLOG https://crowdsec.net/blog/ for what is already available.

My requested integration ( https://hub.crowdsec.net/author/crowdsecurity/bouncers/cs-nginx-bouncer ) is already a lua script that works in OpenResty, so it would only be, in my opinion, a matter of copy-paste in the config file of NPM and not writing a new module or script to integrate it.

Please allow me to cite from Crowdsec's website:

I understand that Crowdsec is a new tool for many but it is able to do even more than fail2ban and the learning curve is low and in the long run it would overtake f2b and all you have to do is configure it where to read the logs and it is already packed with detectors and ready to take action trough bouncers on new detected atacker or the ones downloaded from the community list. It is a set and forget application that can be further integrated with others, being prometheus or your own application over the network trough it's api.

chaptergy commented 3 years ago

Yeah, it was not my intention to say this issue was the same thing as the fail2ban one, I just wanted to cross-link these issues, since this they seem mutually exclusive.

Sparkxxx commented 3 years ago

I didn't thought you were saying that, just wanted to clarify a little bit since with appearance of crowdsec (v0.0.1 - May 15, 2020), supporting an old tool and reinventing the wheel and wasting time building around f2b seeamd like a bad idea. Hope you like my proposal and looking forward to see it integrated in NPM.

luismanson commented 2 years ago

Hello, I'll add my vote for this feature. I thinks this is very different from fail2ban in which follows the crowdsourcing approach.

Thanks!

nisargjoshi95 commented 2 years ago

@Sparkxxx do you have the manual steps to do this for the time being? I'm not very familiar with crowdsec so forgive my naivete, I think I have setup crowdsec and metabse in the same dockerfile using the guide here and here, but when i test with nikto from another machine on the local network I do not see a new decision and logs don't appear to show traffic. There isn't an official docker image for the nginx bouncer either, but based on your comment it seems like there may not be a need for an extra bouncer container. Here's the setup I have so far:

version: '3'
services:
  app:
    image: 'jc21/nginx-proxy-manager:latest'
    restart: unless-stopped
    ports:
      - '80:80'
      - '81:81'
      - '443:443'
    environment:
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "npm"
      DB_MYSQL_NAME: "npm"
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
      - logs:/var/log/nginx
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: unless-stopped
    environment:
      MYSQL_ROOT_PASSWORD: 'npm'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'npm'
    volumes:
      - ./data/mysql:/var/lib/mysql

  #crowdsec : it will be fed nginx's logs
  #and later we're going to plug a firewall bouncer to it
  crowdsec:
    image: crowdsecurity/crowdsec:latest
    restart: always
    environment:
      #this is the list of collections we want to install
      #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
      COLLECTIONS: "crowdsecurity/nginx"
      GID: "${GID-1000}"
    depends_on:
      - 'app'
    volumes:
      - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
      - logs:/var/log/nginx
      - crowdsec-db:/var/lib/crowdsec/data/
      - crowdsec-config:/etc/crowdsec/

  #metabase, because security is cool, but dashboards are cooler
  dashboard:
    #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
    build: ./crowdsec/dashboard
    restart: always
    ports:
      - 3000:3000
    environment:
      MB_DB_FILE: /data/metabase.db
      MGID: "${GID-1000}"
    depends_on:
      - 'crowdsec'
    volumes:
      - crowdsec-db:/metabase-data/

volumes:
  logs:
  crowdsec-db:
  crowdsec-config:
klausagnoletti commented 2 years ago

I am head of community at CrowdSec (and an avid user myself). I think it's a great idea to add support for CrowdSec as well. Also I would like to offer my assistance and a way into the CrowdSec dev team if you need any help implementing it.

I would also like to point any technical questions regarding CrowdSec to our discourse where all devs are available to help out.

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

klausagnoletti commented 2 years ago

@nisargjoshi95 Did you ever get this working? If not, feel free to ask on the discourse :-)

klausagnoletti commented 2 years ago

@Sparkxxx please give me a buzz on the discourse, on twitter @klausagnoletti or send me a mail klaus@crowdsec.net. I'd like to ask about your experience with CrowdSec etc.

baudneo commented 2 years ago

I am here to cast my vote, I haven't started looking in depth yet but does anyone have crowdsec setup with the bouncer and NPM?

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

2Wanderer commented 2 years ago

@nisargjoshi95

I'll think your docker-compose wont work because nginx proxymanager is storing it's logs in other than the default nginx folders.

change your acquis.yml to - /var/log/nginx/*.log and then try this docker-compose: it reads the logs folder which is mounted by nginx-proxymananger container. it will read the logs but the parsing isn't that good



services:
 #crowdsec : it will be fed nginx's logs
 #and later we're going to plug a firewall bouncer to it
 crowdsec:
   image: crowdsecurity/crowdsec:v1.2.1
   restart: always
   environment:
     #this is the list of collections we want to install
     #https://hub.crowdsec.net/author/crowdsecurity/collections/nginx
     COLLECTIONS: "crowdsecurity/nginx"
     GID: "${GID-1000}"

   volumes:
     - ./crowdsec/acquis.yaml:/etc/crowdsec/acquis.yaml
     - /fullpath-to-nginx-proxymanager/data/logs:/var/log/nginx
     - crowdsec-db:/var/lib/crowdsec/data/
     - crowdsec-config:/etc/crowdsec/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.4
  #metabase, because security is cool, but dashboards are cooler
 dashboard:
   #we're using a custom Dockerfile so that metabase pops with pre-configured dashboards
   build: ./crowdsec/dashboard
   restart: always
   ports:
     - 3000:3000
   environment:
     MB_DB_FILE: /data/metabase.db
     MGID: "${GID-1000}"
   depends_on:
     - 'crowdsec'
   volumes:
     - crowdsec-db:/metabase-data/
   networks:
     crowdsec_test:
       ipv4_address: 172.20.0.5

volumes:
 logs:
 crowdsec-db:
 crowdsec-config:

networks:
 crowdsec_test:
   ipam:
     driver: default
     config:
       - subnet: 172.20.0.0/24

INFO[26-11-2021 04:58:54 PM] Buckets Metrics:
+--------------------------------------+---------------+-----------+--------------+--------+---------+
|                BUCKET                | CURRENT COUNT | OVERFLOWS | INSTANCIATED | POURED | EXPIRED |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
| crowdsecurity/http-bad-user-agent    | -             | -         |            2 |      2 |       2 |
| crowdsecurity/http-crawl-non_statics | -             | -         |           77 |     77 |      77 |
+--------------------------------------+---------------+-----------+--------------+--------+---------+
INFO[26-11-2021 04:58:54 PM] Acquisition Metrics:
+---------------------------------------------+------------+--------------+----------------+------------------------+
|                   SOURCE                    | LINES READ | LINES PARSED | LINES UNPARSED | LINES POURED TO BUCKET |
+---------------------------------------------+------------+--------------+----------------+------------------------+
| file:/var/log/nginx/default-host_access.log |         37 |           30 |              7 |                     32 |
| file:/var/log/nginx/fallback-access.log     |         34 | -            |             34 | -                      |
| file:/var/log/nginx/fallback_error.log      |         23 |           23 | -              |                     20 |
| file:/var/log/nginx/proxy-host-1_access.log |        216 | -            |            216 | -                      |
| file:/var/log/nginx/proxy-host-1_error.log  |         28 |           28 | -              |                     27 |
+---------------------------------------------+------------+--------------+----------------+------------------------+
INFO[26-11-2021 04:58:54 PM] Parser Metrics:
+--------------------------------+------+--------+----------+
|            PARSERS             | HITS | PARSED | UNPARSED |
+--------------------------------+------+--------+----------+
| child-crowdsecurity/http-logs  |  243 |    100 |      143 |
| child-crowdsecurity/nginx-logs |  646 |     81 |      565 |
| crowdsecurity/dateparse-enrich |   81 |     81 | -        |
| crowdsecurity/geoip-enrich     |   30 |     30 | -        |
| crowdsecurity/http-logs        |   81 |     14 |       67 |
| crowdsecurity/nginx-logs       |  338 |     81 |      257 |
| crowdsecurity/non-syslog       |  338 |    338 | -        |
| crowdsecurity/whitelists       |   81 |     81 | -        |
+--------------------------------+------+--------+----------+
INFO[26-11-2021 04:58:54 PM] Local Api Metrics:
+----------------------+--------+------+
|        ROUTE         | METHOD | HITS |
+----------------------+--------+------+
| /v1/alerts           | GET    |    1 |
| /v1/decisions/stream | GET    | 7021 |
| /v1/watchers/login   | POST   |    3 |
+----------------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Machines Metrics:
+-----------+------------+--------+------+
|  MACHINE  |   ROUTE    | METHOD | HITS |
+-----------+------------+--------+------+
| localhost | /v1/alerts | GET    |    1 |
+-----------+------------+--------+------+
INFO[26-11-2021 04:58:54 PM] Local Api Bouncers Metrics:
+---------------------+----------------------+--------+------+
|       BOUNCER       |        ROUTE         | METHOD | HITS |
+---------------------+----------------------+--------+------+
| HostFirewallBouncer | /v1/decisions/stream | GET    | 7021 |
+---------------------+----------------------+--------+------+
jakern commented 2 years ago

@baudneo did this work for you? I would have thought there would be issues with NPM's custom log format.

klausagnoletti commented 2 years ago

@klausagnoletti - Should I just go to the discourse and ask there or are there any crowdsec docs on how to set it all up while utilizing NPM? Thanks.

Sorry for my late reply. I don't get notifications from Github even though I enabled it :-/

Just go to the Discorse and ask. To my knowledge noone has tried integrating with npm yet.

baudneo commented 2 years ago

@jakern, it is a strange situation. If NPM is already running, proxying requests and I start up crowdsec with the newest config that @2Wanderer posted, everything is ok. When I reboot, NPM no longer will come up, citing an error binding to port :80 and :443 due to the crowdsec nginx bouncer using port :80 or :443. I am playing around trying to get things to mesh properly. Once I get that part stable I will start trying to integrated the logs structure.

klausagnoletti commented 2 years ago

I tried creating a docker container extending the npm container and installing crowdsec-nginx-bouncer. It didn't work. Turns out npm doesn't use nginx but openresty, wghich we don't (yet) support. But it's on the roadmap. So I guess it'a a bit of a PITA to get working before then.

leon1995 commented 2 years ago

Can you give a rough estimation when this will probably happen?

klausagnoletti commented 2 years ago

No, it is not scheduled. Maybe the community will contribute it. It's open source after all :-)

leon1995 commented 2 years ago

I already tried it with a custom dockerfile:

FROM jc21/nginx-proxy-manager:latest

RUN curl -s https://packagecloud.io/install/repositories/crowdsec/crowdsec/script.deb.sh | bash
RUN apt install -y crowdsec crowdsec-nginx-bouncer
RUN sed -i '1s/^/filenames:\n - \/data\/logs\/*.log\nlabels:\n  type:nginx/' /etc/crowdsec/acquis.yaml

but I get some weird errors. here are some of these:

Unpacking crowdsec-nginx-bouncer (0.0.7) ...
Setting up lua-logging (1.3.0-1) ...
Setting up libxpm4:amd64 (1:3.5.12-1) ...
Setting up nginx-common (1.14.2-2+deb10u4) ...

Configuration file '/etc/nginx/mime.types'
 ==> File on system created by you or by a script.
 ==> File also in package provided by package maintainer.
   What would you like to do about it ?  Your options are:
    Y or I  : install the package maintainer's version
    N or O  : keep your currently-installed version
      D     : show the differences between the versions
      Z     : start a shell to examine the situation
 The default action is to keep your current version.
*** mime.types (Y/I/N/O/D/Z) [default=N] ? dpkg: error processing package nginx-common (--configure):
 end of file on stdin at conffile prompt
Setting up lua-socket:amd64 (3.0~rc1+git+ac3201d-4) ...
dpkg: dependency problems prevent configuration of nginx-full:
 nginx-full depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package nginx-full (--configure):
 dependency problems - leaving unconfigured
Setting up libjbig0:amd64 (2.1-3.1+b2) ...
Setting up libicu63:amd64 (63.1-6+deb10u2) ...
Setting up lua-sql-sqlite3:amd64 (2.3.4-1+b1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-xslt-filter:
 libnginx-mod-http-xslt-filter depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-xslt-filter (--configure):
 dependency problems - leaving unconfigured
Setting up libjpeg62-turbo:amd64 (1:1.5.2-2+deb10u1) ...
dpkg: dependency problems prevent configuration of libnginx-mod-http-auth-pam:
 libnginx-mod-http-auth-pam depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-auth-pam (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-geoip:
 libnginx-mod-http-geoip depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

dpkg: error processing package libnginx-mod-http-geoip (--configure):
 dependency problems - leaving unconfigured
dpkg: dependency problems prevent configuration of libnginx-mod-http-ndk:
 libnginx-mod-http-ndk depends on nginx-common (= 1.14.2-2+deb10u4); however:
  Package nginx-common is not configured yet.

maybe you can help? or is this the error you mentioned, that not the real nginx but openresty is used?

klausagnoletti commented 2 years ago

This is the error I mentioned. Openresty != nginx (and they're not compatible). So the .deb complains that dependencies are not installed. So it won't work, unfortunately.

klausagnoletti commented 2 years ago

Anyways I have good news @leon1995. I asked the developers just to be sure. And it turns out they're working on it as we speak. Only a new bouncer is needed as logfiles are the same, apparently. I'll ping you here once it's out in some usable shape or form. Are you willing to do a beta test?

sanderdatema commented 2 years ago

@klausagnoletti I'm running on unRaid with NPM and willing to beta test if you need more people. I have to be honest: I have a pretty private server, so there won't be many visitors. I can imagine you'd rather have a beta tester with some spicey logs.

klausagnoletti commented 2 years ago

I can imagine you'd rather have a beta tester with some spicey logs.

Not gonna lie: That would be more interesting. But it won't prevent you to join the fun :-)

leon1995 commented 2 years ago

@klausagnoletti I would also join the beta. However, mine is also a private server with multiple services and just a few people using it (maybe up to ten I think)

sanderdatema commented 2 years ago

So... @leon1995 and I will visit each other's servers like crazy til steam's coming out of our logs. Will that work for you, @klausagnoletti? 🤪

klausagnoletti commented 2 years ago

Sure, hit me up at klaus (at) crowdsec (dot) net. You'd have to do a Docker container on your own (but would be happy to help :-)

leon1995 commented 2 years ago

I sent you an email

dhernan3 commented 2 years ago

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container. I have installed Crowdsec last week, so my knowledge about it is very limited.

klausagnoletti commented 2 years ago

I can help testing if you wish. I have a host with Dietpi for x86, several dockers containers, one with Nginx proxy manager, other with crowdsec with the logs from Nginx proxy manager logs mapped, and also the firewall bouncer installed on the host and connected to Crowdsec's docker container. I have installed Crowdsec last week, so my knowledge about it is very limited.

Cool, please send me an email so I have your information in my inbox :-)

klausagnoletti commented 2 years ago

Now there's interesting news! @dhernan3 @leon1995 @sanderdatema @baudneo @2Wanderer @nisargjoshi95 @Sparkxxx

https://docs.crowdsec.net/docs/bouncers/openresty/

The bouncer is out. It should be easy to extend the existing npm Dockerfile with that since it's available as a .deb and npm is based on Debian. For log parsing and scenarios the normal nginx collection can be used. Contrary to my former believe, the log format is indeed the same. Find it here: https://hub.crowdsec.net/author/crowdsecurity/collections/nginx

Let me know what comes out of experimenting with it :-)

Revorge commented 2 years ago

Would be great if anyone can get this working and write some quick instructions :)

klausagnoletti commented 2 years ago

Actually there's many ways to use it. The quick and dirty way is to have the agent and firewall installed on the host (which I have on my own instance as of a couple of days), point to the log files in the /etc/crowdsec/aquis.yaml (of course adjusted to your environment):

#NPM - Openresty logs
filenames:
  - /home/klaus/containers/npm/data/logs/*.log
    #  - /home/klaus/containers/npm/data/logs/proxy-host-2_access.log
    #  - /home/klaus/containers/npm/data/logs/proxy-host-3_access.log
labels:
  type: nginx
---

Protip: remember the --- as a separator between entries or you get weird errors :-)

After this, install the nginx collection: sudo cscli collections install crowdsecurity/nginx and reload crowdsec sudo systemctl reload crowdsec.

Another approach would be to extend the npm container with the new openresty bouncer and run the CrowdSec agent either on the docker host or in its own container (and maybe share data between the containers using docker volumes. This is not yet tested but knock yourselves out :-)

Let me know what comes out of it!

Revorge commented 2 years ago

Actually there's many ways to use it. The quick and dirty way is to have the agent and firewall installed on the host (which I have on my own instance as of a couple of days), point to the log files in the /etc/crowdsec/aquis.yaml (of course adjusted to your environment):

Let me know what comes out of it!

Perfect, thanks - I'll give it at go as soon as i get the change :)

klausagnoletti commented 2 years ago
  • I'll probaly try using the docker container, seeing as pretty much everything else in my setup is containerized.

You're most welcome - only issue with that approach is that you'd have to either containerize the openresty bouncer or the firewall bouncer (which would be messy since it would need root permissions) or install just the firewall bouncer on the host and find a way for the CrowdSec container to talk to it in order to be able to block anything. Feel free to post at https://discourse.crowdsec.net if you run into problems.

klausagnoletti commented 2 years ago

Hey! So there has been a recent development in the case. We have modificed log parsers and have a PR ready: https://github.com/crowdsecurity/hub/pull/358

The big question - and what we need your help for - is input to how this should be integrated with NPM to be as usable to the NPM community as possible. If you have good suggestions wither write them here or in our Discord (in #dev-general)

Also I added an issue on logging Admin panel by default. Currently it's disabled which makes it hard (e.g. impossible for CrowdSec or anything else to protect it): https://github.com/NginxProxyManager/nginx-proxy-manager/issues/1800

Let me know what you think!

baudneo commented 2 years ago

I have been trying and trying to get the openresty bouncer to work with no success. All my other bouncers work fine except for openresty. I added the LOG_FILE directive to the openresty bouncer file and the log file is never created. cscli bouncers list shows this

`

NAME IP ADDRESS VALID LAST API PULL TYPE VERSION

FirewallBouncer-1643162803 ::1 ✔️ 2022-01-26T02:40:25Z crowdsec-firewall-bouncer v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714 zm-iptables 10.0.0.30 ✔️ 2022-01-26T02:40:22Z crowdsec-firewall-bouncer v0.0.21-debian-pragmatic-eebd0b02ac6bb8cb330b3fcacc1f400595fbc6f7 local-cloudflare ::1 ✔️ 2022-01-26T02:40:30Z crowdsec-cloudflare-bouncer v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705 local-openresty ✔️ 2022-01-26T02:13:59Z ---------------------------------------------------------------------------------------------------------------------------------------------------------------------`

openresty and openresty-opm installed by the openresty bouncer do not have logging enabled and I am uncertain how to enable it properly as the openresty docs only point towards nginx docs.

I cant debug the bouncer without logging, can someone point me in the right direction to enable logging in openresty so that the bouncer can attempt to output log files for debug?

Can anyone post a working non-docker config where the openresty bouncer is actually working and confirm, that someone has the openresty bouncer actually working?

Edit: Also it seems the nginx collection and its nginx parser do not accept the NPM logs

NPM log message:

[26/Jan/2022:03:27:25 +0000] - 200 200 - GET https xxx.EXAMPLE.com \"/zm/index.php?view=login\" [Client IPADDRESS] [Length 2321] [Gzip -] [Sent-to 10.0.0.30] \"Dalvik/2.1.0 (Linux; U; Android 11; SM-N975W Build/RP1A.200720.012)\" \"-\"

nginx parser patterns

pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'

custom crowdsec GROK patterns:

NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}
DAY2 \d{2}

IT APPEARS THE 'MONTHNUM2' PATTERN IS MISSING!

grep -R MONTHNUM2 /etc/crowdsec
/etc/crowdsec/patterns/nginx:#NGINXERRTIME %{YEAR:year}/%{MONTHNUM2:month}/%{DAY2:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
/etc/crowdsec/patterns/nginx:NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}

I will write a parser for the NPM format, but it would be great if someone could confirm they have the openresty bouncer working.

klausagnoletti commented 2 years ago

I have been trying and trying to get the openresty bouncer to work with no success. All my other bouncers work fine except for openresty. I added the LOG_FILE directive to the openresty bouncer file and the log file is never created. cscli bouncers list shows this

`

NAME IP ADDRESS VALID LAST API PULL TYPE VERSION

FirewallBouncer-1643162803 ::1 ✔️ 2022-01-26T02:40:25Z crowdsec-firewall-bouncer v0.0.22-debian-pragmatic-f64e94b59a948717c3dc848f9abebb27b5974714 zm-iptables 10.0.0.30 ✔️ 2022-01-26T02:40:22Z crowdsec-firewall-bouncer v0.0.21-debian-pragmatic-eebd0b02ac6bb8cb330b3fcacc1f400595fbc6f7 local-cloudflare ::1 ✔️ 2022-01-26T02:40:30Z crowdsec-cloudflare-bouncer v0.0.8-debian-pragmatic-38768ad6e47bc7ce058668ef286de303f897d705 local-openresty ✔️ 2022-01-26T02:13:59Z ---------------------------------------------------------------------------------------------------------------------------------------------------------------------`

openresty and openresty-opm installed by the openresty bouncer do not have logging enabled and I am uncertain how to enable it properly as the openresty docs only point towards nginx docs.

I cant debug the bouncer without logging, can someone point me in the right direction to enable logging in openresty so that the bouncer can attempt to output log files for debug?

Can anyone post a working non-docker config where the openresty bouncer is actually working and confirm, that someone has the openresty bouncer actually working?

Edit: Also it seems the nginx collection and its nginx parser do not accept the NPM logs

NPM log message:

[26/Jan/2022:03:27:25 +0000] - 200 200 - GET https xxx.EXAMPLE.com \"/zm/index.php?view=login\" [Client IPADDRESS] [Length 2321] [Gzip -] [Sent-to 10.0.0.30] \"Dalvik/2.1.0 (Linux; U; Android 11; SM-N975W Build/RP1A.200720.012)\" \"-\"

nginx parser patterns

pattern: '(%{IPORHOST:target_fqdn} )?%{IPORHOST:remote_addr} - (%{NGUSER:remote_user})? \[%{HTTPDATE:time_local}\] "%{WORD:verb} %{DATA:request} HTTP/%{NUMBER:http_version}" %{NUMBER:status} %{NUMBER:body_bytes_sent} "%{NOTDQUOTE:http_referer}" "%{NOTDQUOTE:http_user_agent}"( %{NUMBER:request_length} %{NUMBER:request_time} \[%{DATA:proxy_upstream_name}\] \[%{DATA:proxy_alternative_upstream_name}\])?'

pattern: '(%{IPORHOST:target_fqdn} )?%{NGINXERRTIME:time} \[%{LOGLEVEL:loglevel}\] %{NONNEGINT:pid}#%{NONNEGINT:tid}: (\*%{NONNEGINT:cid} )?%{GREEDYDATA:message}, client: %{IPORHOST:remote_addr}, server: %{DATA:target_fqdn}, request: "%{WORD:verb} %{URIPATHPARAM:request} HTTP/%{NUMBER:http_version}", host: "%{IPORHOST}"'

custom crowdsec GROK patterns:

NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}
DAY2 \d{2}

IT APPEARS THE 'MONTHNUM2' PATTERN IS MISSING!

grep -R MONTHNUM2 /etc/crowdsec
/etc/crowdsec/patterns/nginx:#NGINXERRTIME %{YEAR:year}/%{MONTHNUM2:month}/%{DAY2:day} %{HOUR:hour}:%{MINUTE:minute}:%{SECOND:second}
/etc/crowdsec/patterns/nginx:NGINXERRTIME %{YEAR}/%{MONTHNUM2}/%{DAY2} %{HOUR}:%{MINUTE}:%{SECOND}

I will write a parser for the NPM format, but it would be great if someone could confirm they have the openresty bouncer working.

Did you get anywhere with this? If not, I'll arrange for you to get in touch with the dev. Ping me on Discord :-)

LePresidente commented 2 years ago

Ok got the openresty bouncer working in jlesage/docker-nginx-proxy-manager docker image and created a pull

I'm self hosting the image currently at dockerhub.com/lepresidente/nginx-proxy-manager which is based off my fork

You will need to set the Environment Variable CROWDSEC_BOUNCER to 1 and edit the /config/crowdsec-openresty-bouncer.conf once it has been started at least once with your API and URL to crowdsec

baudneo commented 2 years ago

Good job!

baudneo commented 2 years ago

I have forked and successfully created a docker image that incorporates openresty bouncer into NPM as it is. This means you can docker pull baudneo/nginx-proxy-manager:latest and directly replace the existing jc21/nginx_proxy_manager images. I also added the GeoIP2 with lib-MindMax DB module and modsecurity directly to openresty.

Note

New features

  1. Enable logging for the admin dashboard, if enabled the logs will be in /data/logs/adminpanel[access/error].log.
  2. Enable DEBUG level logging for the default openresty error.log (/data/logs/fallback_error.log) file (default level is warn).

New ENV vars

Forked repos:

Docker hub

Docker pull

cscli output

baudneo commented 2 years ago

Example docker-compose.yaml, literally just change the image: and add the new ENV vars.

version: "3"
services:
  npm:
    #image: 'jc21/nginx-proxy-manager:latest'
    image: 'baudneo/nginx-proxy-manager:latest'
    restart: always
    container_name: npm-crowdsec
    ports:
      # Public HTTP Port:
      - '80:80'
      # Public HTTPS Port:
      - '443:443'
      # Admin Web Port:
      - '81:81'
    environment:
      ADMIN_PANEL_LOG: "1"
      CROWDSEC_BOUNCER: "1"
      OPENRESTY_DEBUG: "0"

      CROWDSEC_LAPI: "http://IP TO CROWDSEC LOCAL API:8080"
      CROWDSEC_KEY: "xxxxxxxxxxxxxxxxxxxxxxxx"

      # These are the settings to access your db
      DB_MYSQL_HOST: "db"
      DB_MYSQL_PORT: 3306
      DB_MYSQL_USER: "npm"
      DB_MYSQL_PASSWORD: "PASSw0rD"
      DB_MYSQL_NAME: "npm"
      # If you would rather use Sqlite uncomment this
      # and remove all DB_MYSQL_* lines above
      # DB_SQLITE_FILE: "/data/database.sqlite"
      # Uncomment this if IPv6 is not enabled on your host
      # DISABLE_IPV6: 'true'
    volumes:
      - ./data:/data
      - ./letsencrypt:/etc/letsencrypt
    depends_on:
      - db
  db:
    image: 'jc21/mariadb-aria:latest'
    restart: always
    container_name: npm_db
    environment:
      MYSQL_ROOT_PASSWORD: 'xxXXxxXXXxxxXXX'
      MYSQL_DATABASE: 'npm'
      MYSQL_USER: 'npm'
      MYSQL_PASSWORD: 'DB Passw0rd'
    volumes:
      - ./data/mysql:/var/lib/mysql
gatesry commented 2 years ago

@baudneo - This is excellent. Thank you for everyone's contributions. I will be testing this in my lab soonish.

baudneo commented 2 years ago

I have added ModSecurity to a new image with CrowdSec

Now CrowdSec and ModSecurity are installed and working. The new image that has modsec is docker pull baudneo/nginx-proxy-manager:cs-modsec. I installed OWASP-CRS for modsec. The modsec audit log is in /data/logs/modsec_audit.log by default, you can turn it off by editing /data/modsec/modsecurity.conf.

NOTE

To enable modsec you need to add these 2 directives to the config somewhere.

modsecurity on;
modsecurity_rules_file /etc/nginx/modsec/main.conf;

Here are some pointers.

Some logs from modsec_audit.log to confirm modsecurity is working. This was a test using ?exec=/bin/bash in the URI.

---C3KCgGTx---F--
HTTP/2.0 403
Server: nginx
Date: Tue, 22 Feb 2022 06:16:02 GMT
Content-Length: 552
Content-Type: text/html
Connection: close
Strict-Transport-Security: max-age=63072000;includeSubDomains; preload

---C3KCgGTx---H--
ModSecurity: Warning. Matched "Operator `PmFromFile' with parameter `unix-shell.data' against variable `ARGS:exec' (Value: `/bin/bash' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-932-APPLICATION-ATTACK-RCE.conf"] [line "480"] [id "932160"] [rev ""] [msg "Remote Command Execution: Unix Shell Code Found"] [data "Matched Data: bin/bash found within ARGS:exec: /bin/bash"] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-shell"] [tag "platform-unix"] [tag "attack-rce"] [tag "paranoia-level/1"] [tag "OWASP_CRS"] [tag "capec/1000/152/248/88"] [tag "PCI/6.5.2"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref "o1,8v14,9t:urlDecodeUni,t:cmdLine,t:normalizePath,t:lowercase"]
ModSecurity: Access denied with code 403 (phase 2). Matched "Operator `Ge' with parameter `5' against variable `TX:ANOMALY_SCORE' (Value: `5' ) [file "/etc/nginx/modsec/ruleset/rules/REQUEST-949-BLOCKING-EVALUATION.conf"] [line "80"] [id "949110"] [rev ""] [msg "Inbound Anomaly Score Exceeded (Total Score: 5)"] [data ""] [severity "2"] [ver "OWASP_CRS/3.3.2"] [maturity "0"] [accuracy "0"] [tag "application-multi"] [tag "language-multi"] [tag "platform-multi"] [tag "attack-generic"] [hostname "172.18.0.4"] [uri "/zm/"] [unique_id "1645510562"] [ref ""]

I don't know JavaScript so I am unable to add to the UI to make things pretty. Ideally there would be a ModSecurity tab that has a toggle switch for on/off and a text input box for the rules file. This tab would be in the proxyhost modal where you set what domain/target host/target port etc. It would be nice to have some sort of UI for corwdsec as well to turn it off and on and to edit the config file. I don't think having a rule editor for modsec would matter much, but it would be nice to have it.

I might take the time to teach myself some JS just to implement this to make it all pretty. Things are working for CrowdSec and ModSecurity though. The people who want that functionality should not have an issue with configuring things from the CLI anyways, for now.

klausagnoletti commented 2 years ago

Will this also work with the CrowdSec agent installed directly on Docker host?

baudneo commented 2 years ago

Will this also work with the CrowdSec agent installed directly on Docker host?

Yes just edit the bouncer config file and put the ip:port of the docker host in the API_URL. Do not use "localhost:8080" use the actual IP for your docker host. If you have any issues when starting the container issue docker logs --follow <container name> to see what the logs are saying. I also recommend tail /path/to/data/logs/*_error.log to see if any Lua or crowdsec errors are in those logs.

If you mean using docker networks I am not sure as I do not have my system setup that way. I am sure there is a way to make that work.

I am currently building the multiarch image base images and will be pushing multi arch images soon.

klausagnoletti commented 2 years ago

Thanks. I am using a Docker bridged network totally standardized. If one configures the agent to listen on 0.0.0.0 it can be reached from the container using it's docker network ip. Remember to firewall the port so it's not internet exposed (which reminds me that I asked the devs if there's a CrowdSec parser + scenario for CrowdSec. There should be. Very meta :-)

erdoukki commented 2 years ago

I just ugrade my NPM with "baudneo/nginx-proxy-manager:cs-modsec" on my OpenWrt server (with NextCloud). so all broken now : 

stderr: standard_init_linux.go:228: exec user process caused: exec format error
# uname -ar
Linux STARGATE 5.4.124 #0 SMP Sun Jun 13 22:02:19 2021 aarch64 GNU/Linux

Back to the 'jc21/nginx-proxy-manager:latest' okay again...

My own use for Nginx-Proxy-Manager is to check the logs externally from the dockerized side ! just with:

    volumes:
      - /srv/NGINX/data/nginx-proxy-manager:/data
      - /srv/NGINX/letsencrypt:/etc/letsencrypt
      - /srv/NGINX/logs:/var/log/nginx

ad in the main OpenWrt host :

$ cat /etc/crowdsec/acquis.d/npm.yaml 
# NginxProxyManager
filenames:
#  - ~/data/logs/*.log
# Dockerized NGINX PROXY MANAGER
  - /srv/NGINX/data/nginx-proxy-manager/logs/*.log
labels:
  type: nginx-proxy-manager

with intalled collection : crowdsecurity/nginx-proxy-manager

gatesry commented 2 years ago

@erdoukki - What's the CPU architecture for your OpenWRT server?

erdoukki commented 2 years ago

architecture : aarch64 cortex-a53 arm64 Marvell armada37xx

gatesry commented 2 years ago

I don't see an ARM build for baudneo/nginx_proxy_manager on dockerhub . Which is most likely causing your issue. You could try editing the Dockerfile to build an ARM image or ask the owner to include ARM architecture.

erdoukki commented 2 years ago

Good idea. Thanks.