search

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders
Other
826 stars 181 forks source link

[INSTALLATION] Missing envariable expression for SetEnvIfNoCase / [PANELS] Froxlor Usage #134

Closed Tealk closed 4 years ago

Tealk commented 5 years ago

Describe the problem you are experiencing If i try to load the config i get the following error:

apache2ctl configtest
AH00526: Syntax error on line 28 of /etc/apache2/custom.d/whitelist-domains.conf:
Missing envariable expression for SetEnvIfNoCase
Action 'configtest' failed.

Server (please complete the following information):

mitchellkrogza commented 5 years ago

Please post your /etc/apache2/custom.d/whitelist-domains.conf

Tealk commented 5 years ago

i didn't want to publish all my domains here now here a few examples SetEnvIfNoCase Referer anzahcraft.de SetEnvIfNoCase Referer discord.anzahcraft.de SetEnvIfNoCase Referer rollenspiel.wiki

mitchellkrogza commented 5 years ago

You did not follow the examples properly.

SetEnvIfNoCase Referer ~*yourdomain\.com good_ref

so

SetEnvIfNoCase Referer ~*anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*discord\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*rollenspiel\.wiki good_ref
Tealk commented 5 years ago

I followed the example from the description.

# Make sure any domains have dots and special characters escaped as per the Regex examples below.
# For example myfirstowndomainname.com should be entered as myfirstowndomainname\.com
# and my-second-owndomainname.com should be entered as my\-second\-owndomainname\.com

but it works now, thanks.

only I seem to have done something wrong.

# curl -I https://anzahcraft.de -e http://100dollars-seo.com
HTTP/2 200
date: Wed, 21 Aug 2019 16:03:46 GMT
server: Apache
x-powered-by: PHP/7.3.8-1+0~20190807.43+debian10~1.gbp7731bf
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: private, no-cache, max-age=0
strict-transport-security: max-age=31536000
set-cookie: anzah_csrf=55NccpkOy65pP3p5; path=/; secure; HttpOnly
last-modified: Wed, 21 Aug 2019 16:03:46 GMT
content-length: 151993
x-xss-protection: 1; mode=block
x-robots-tag: all
x-download-options: noopen
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
expect-ct: enforce, max-age=86400,report-uri="https://anzahreport.uriports.com/reports/report"
feature-policy: accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'
report-to: {"group":"default","max_age":10886400,"endpoints":[{"url":"https://anzahreport.uriports.com/reports"}],"include_subdomains":true}
nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":1.0}
content-type: text/html; charset=utf-8
# curl -IA "80legs" https://anzahcraft.de
HTTP/2 200
date: Wed, 21 Aug 2019 16:04:14 GMT
server: Apache
x-powered-by: PHP/7.3.8-1+0~20190807.43+debian10~1.gbp7731bf
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: private, no-cache, max-age=0
strict-transport-security: max-age=31536000
set-cookie: anzah_csrf=x4D2vHKMxJMYeIMU; path=/; secure; HttpOnly
last-modified: Wed, 21 Aug 2019 16:04:14 GMT
content-length: 151993
x-xss-protection: 1; mode=block
x-robots-tag: all
x-download-options: noopen
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
expect-ct: enforce, max-age=86400,report-uri="https://anzahreport.uriports.com/reports/report"
feature-policy: accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'
report-to: {"group":"default","max_age":10886400,"endpoints":[{"url":"https://anzahreport.uriports.com/reports"}],"include_subdomains":true}
nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":1.0}
content-type: text/html; charset=utf-8
mitchellkrogza commented 5 years ago

Need to post all your config files before I can help you more. You've not done something crucial to make it work you can strip any domain names from your config files

mitchellkrogza commented 5 years ago

Most likely Step 6 https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/Apache_2.4

GitHub
mitchellkrogza/apache-ultimate-bad-bot-blocker
Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders - mitc...
Tealk commented 5 years ago

but I would have inserted it image

mitchellkrogza commented 5 years ago

Somewhere it is just not loading the blocker. Go through your config carefully

Compare your config files with these from my working build tests https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/tree/master/.dev-tools/_test_results/_conf_files_2.4

GitHub
mitchellkrogza/apache-ultimate-bad-bot-blocker
Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders - mitc...
Tealk commented 5 years ago

so the configs are created by froxler. i have added the include to all vhost files

i have set ssl forwarding for all domains and only in this vhost file is the include, but that shouldn't be a problem?

mitchellkrogza commented 5 years ago

Post your apache2.conf and vhost configs please

Tealk commented 5 years ago

apache2.conf

# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.4/ for detailed information about
# the directives and /usr/share/doc/apache2/README.Debian about Debian specific
# hints.
#
#
# Summary of how the Apache 2 configuration works in Debian:
# The Apache 2 web server configuration in Debian is quite different to
# upstream's suggested way to configure the web server. This is because Debian's
# default Apache2 installation attempts to make adding and removing modules,
# virtual hosts, and extra configuration directives as flexible as possible, in
# order to make automating the changes and administering the server as easy as
# possible.

# It is split into several files forming the configuration hierarchy outlined
# below, all located in the /etc/apache2/ directory:
#
#   /etc/apache2/
#   |-- apache2.conf
#   |   `--  ports.conf
#   |-- mods-enabled
#   |   |-- *.load
#   |   `-- *.conf
#   |-- conf-enabled
#   |   `-- *.conf
#   `-- sites-enabled
#       `-- *.conf
#
#
# * apache2.conf is the main configuration file (this file). It puts the pieces
#   together by including all remaining configuration files when starting up the
#   web server.
#
# * ports.conf is always included from the main configuration file. It is
#   supposed to determine listening ports for incoming connections which can be
#   customized anytime.
#
# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/
#   directories contain particular configuration snippets which manage modules,
#   global configuration fragments, or virtual host configurations,
#   respectively.
#
#   They are activated by symlinking available configuration files from their
#   respective *-available/ counterparts. These should be managed by using our
#   helpers a2enmod/a2dismod, a2ensite/a2dissite and a2enconf/a2disconf. See
#   their respective man pages for detailed information.
#
# * The binary is called apache2. Due to the use of environment variables, in
#   the default configuration, apache2 needs to be started/stopped with
#   /etc/init.d/apache2 or apache2ctl. Calling /usr/bin/apache2 directly will not
#   work with the default configuration.

# Global configuration
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the Mutex documentation (available
# at <URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#Mutex file:${APACHE_LOCK_DIR} default

#
# The directory where shm and other runtime files will be stored.
#

DefaultRuntimeDir ${APACHE_RUN_DIR}

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 500

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog ${APACHE_LOG_DIR}/error.log

#
# LogLevel: Control the severity of messages logged to the error_log.
# Available values: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the log level for particular modules, e.g.
# "LogLevel info ssl:warn"
#
LogLevel warn

# Include module configuration:
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

# Include list of ports to listen on
Include ports.conf

# Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
<Directory />
    Options FollowSymLinks
    AllowOverride None
    Require all denied
</Directory>

<Directory /usr/share>
    AllowOverride None
    Require all granted
</Directory>

<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride None
    Require all granted
</Directory>

#<Directory /srv/>
#   Options Indexes FollowSymLinks
#   AllowOverride None
#   Require all granted
#</Directory>

# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht">
    Require all denied
</FilesMatch>

#
# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# These deviate from the Common Log Format definitions in that they use %O
# (the actual bytes sent including headers) instead of %b (the size of the
# requested file), because the latter makes it impossible to detect partial
# requests.
#
# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
# Use mod_remoteip instead.
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
IncludeOptional conf-enabled/*.conf

# Include the virtual host configurations:
IncludeOptional sites-enabled/*.conf
IncludeOptional sites-froxlor/*.conf

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

ServerSignature Off
ServerTokens Prod

# [For HTTPS Support]
Protocols h2 http/1.1

# [For HTTP Support]
Protocols h2c http/1.1

SSLOpenSSLConfCmd DHParameters "/etc/ssl/certs/dhparams4096.pem"
SSLOpenSSLConfCmd ECDHParameters secp384r1
SSLOpenSSLConfCmd Curves secp521r1:secp384r1

35_froxlor_normal_vhost_anzahcraft.de.conf

# 35_froxlor_normal_vhost_anzahcraft.de.conf
# Created 21.08.2019 18:10
# Do NOT manually edit this file, all changes will be deleted after the next domain change at the panel.

# Domain ID: 23 - CustomerID: 1 - CustomerLogin: Anzah
<VirtualHost 185.170.112.156:80 [2a03:4000:15:68f::b9aa:709c]:80>
  ServerName anzahcraft.de
  ServerAlias www.anzahcraft.de
  ServerAdmin MAIL
  <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge
    RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [R=301;L,NE]
  </IfModule>
  <IfModule !mod_rewrite.c>
    Redirect 301 / https://anzahcraft.de/
  </IfModule>
</VirtualHost>

35_froxlor_ssl_vhost_anzahcraft.de.conf

# 35_froxlor_ssl_vhost_anzahcraft.de.conf
# Created 21.08.2019 18:10
# Do NOT manually edit this file, all changes will be deleted after the next domain change at the panel.

# Domain ID: 23 (SSL) - CustomerID: 1 - CustomerLogin: Anzah
<VirtualHost 185.170.112.156:443 [2a03:4000:15:68f::b9aa:709c]:443>
  ServerName anzahcraft.de
  ServerAlias www.anzahcraft.de
  ServerAdmin MAIL
  SSLEngine On
  SSLProtocol -ALL +TLSv1.2 +TLSv1.3
 Protocols h2 http/1.1
  SSLCompression Off
  SSLHonorCipherOrder On
  SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256
  SSLVerifyDepth 10
  SSLCertificateFile /etc/ssl/froxlor-custom/anzahcraft.de.crt
  SSLCertificateKeyFile /etc/ssl/froxlor-custom/anzahcraft.de.key
  SSLCACertificateFile /etc/ssl/froxlor-custom/anzahcraft.de_CA.pem
  SSLCertificateChainFile /etc/ssl/froxlor-custom/anzahcraft.de_chain.pem
  SSLUseStapling on
  <IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=31536000"
  </IfModule>
  DocumentRoot "/var/customers/webs/Anzah/anzahcraft.de/"
  SuexecUserGroup "USER" "USER"
  <FilesMatch \.(php)$>
  SetHandler proxy:unix:/var/lib/apache2/fastcgi/4-anzah-anzahcraft.de-php-fpm.socket|fcgi://localhost
  </FilesMatch>
  <Directory "/var/customers/webs/Anzah/anzahcraft.de/">
    Require all granted
    AllowOverride All
  </Directory>
  Alias /webalizer "/var/customers/webs/Anzah/webalizer/anzahcraft.de"
  LogLevel warn
  ErrorLog "/var/customers/logs/Anzah-anzahcraft.de-error.log"
  CustomLog "/dev/null" combined
<Directory "/var/customers/webs/Anzah/anzahcraft.de/">
        AllowOverride All
        Options FollowSymLinks
        Include custom.d/globalblacklist.conf
    </Directory>
</VirtualHost>
mitchellkrogza commented 5 years ago

If you compare your apache2.conf with the one I told you look at in the test results folder it will hit you in the face.

Your main Directory block must have AllowOverride All

<Directory />
    Options FollowSymLinks
    AllowOverride All
    Require all denied
</Directory>
Tealk commented 5 years ago

but still the same:

# curl -IA "80legs" https://anzahcraft.de
HTTP/2 200
date: Wed, 21 Aug 2019 18:11:08 GMT
server: Apache
x-powered-by: PHP/7.3.8-1+0~20190807.43+debian10~1.gbp7731bf
x-frame-options: SAMEORIGIN
x-content-type-options: nosniff
expires: Thu, 19 Nov 1981 08:52:00 GMT
cache-control: private, no-cache, max-age=0
strict-transport-security: max-age=31536000
set-cookie: anzah_csrf=iGXiIOVINd9HVuZ0; path=/; secure; HttpOnly
last-modified: Wed, 21 Aug 2019 18:11:08 GMT
content-length: 152921
x-xss-protection: 1; mode=block
x-robots-tag: all
x-download-options: noopen
x-permitted-cross-domain-policies: none
referrer-policy: same-origin
expect-ct: enforce, max-age=86400,report-uri="https://anzahreport.uriports.com/reports/report"
feature-policy: accelerometer 'none'; camera 'none'; geolocation 'none'; gyroscope 'none'; magnetometer 'none'; microphone 'none'; payment 'none'; usb 'none'
report-to: {"group":"default","max_age":10886400,"endpoints":[{"url":"https://anzahreport.uriports.com/reports"}],"include_subdomains":true}
nel: {"report_to":"default","max_age":2592000,"include_subdomains":true,"failure_fraction":1.0}
content-type: text/html; charset=utf-8

But my web directory is also "/var/customers/webs" and there are many subfolders for each website.

mitchellkrogza commented 5 years ago

I'll have a look in the morning very hard to diagnose on my mobile. Very silly question but you have restarted Apache every time

mitchellkrogza commented 5 years ago

Are you testing from the machine that is whitelisted? Have you tested externally?

mitchellkrogza commented 5 years ago

Your main Directory Block for www also must have AllowOverride All. Please compare with this config file https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/.dev-tools/_test_results/_conf_files_2.4/apache2.conf

GitHub
mitchellkrogza/apache-ultimate-bad-bot-blocker
Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders - mitc...
mitchellkrogza commented 5 years ago

You also have duplicate Directory blocks

<Directory "/var/customers/webs/Anzah/anzahcraft.de/">
    Require all granted
    AllowOverride All
  </Directory>
  Alias /webalizer "/var/customers/webs/Anzah/webalizer/anzahcraft.de"
  LogLevel warn
  ErrorLog "/var/customers/logs/Anzah-anzahcraft.de-error.log"
  CustomLog "/dev/null" combined
<Directory "/var/customers/webs/Anzah/anzahcraft.de/">
        AllowOverride All
        Options FollowSymLinks
        Include custom.d/globalblacklist.conf
    </Directory>
Tealk commented 5 years ago

i have tested from the machine that is whitelisted

i have fixed the Directory Block for www

the first block is created by the froxlor script, i don't know how to intervene

Tealk commented 5 years ago

so i wrote the whole thing in apache2.conf now and still get status 200 when i run the test

<Directory />
    Options FollowSymLinks
    AllowOverride All
    Require all denied
</Directory>

<Directory /usr/share>
    AllowOverride None
    Require all granted
</Directory>

<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

#<Directory /srv/>
#   Options Indexes FollowSymLinks
#   AllowOverride None
#   Require all granted
#</Directory>

<Directory "/var/customers/webs">
    AllowOverride All
    Options FollowSymLinks
    Include custom.d/globalblacklist.conf
</Directory>
mitchellkrogza commented 5 years ago

This is very strange, can you try specifying the full path to the blocker?

Include /etc/apache2/custom.d/globalblacklist.conf it appears it is just not being loaded or not being seen by Apache.

Tealk commented 5 years ago

it is loaded, because if I enter an error at the "whitelist-domains.conf", apache does not start

even with full path i get a status 200

curl -IA "80legs" https://anzahcraft.de
HTTP/2 200
mitchellkrogza commented 5 years ago

Please post your output of apache2ctl -M

mitchellkrogza commented 5 years ago

Also your contents of whitelist-domains and whitelist-ips

Tealk commented 5 years ago 
apache2ctl -M
Loaded Modules:
 core_module (static)
 so_module (static)
 watchdog_module (static)
 http_module (static)
 log_config_module (static)
 logio_module (static)
 version_module (static)
 unixd_module (static)
 access_compat_module (shared)
 actions_module (shared)
 alias_module (shared)
 auth_basic_module (shared)
 authn_core_module (shared)
 authn_file_module (shared)
 authz_core_module (shared)
 authz_host_module (shared)
 authz_user_module (shared)
 autoindex_module (shared)
 cgi_module (shared)
 deflate_module (shared)
 dir_module (shared)
 env_module (shared)
 expires_module (shared)
 filter_module (shared)
 geoip_module (shared)
 headers_module (shared)
 http2_module (shared)
 mime_module (shared)
 mpm_event_module (shared)
 negotiation_module (shared)
 perl_module (shared)
 proxy_module (shared)
 proxy_balancer_module (shared)
 proxy_fcgi_module (shared)
 proxy_http_module (shared)
 proxy_wstunnel_module (shared)
 reqtimeout_module (shared)
 rewrite_module (shared)
 setenvif_module (shared)
 slotmem_shm_module (shared)
 socache_shmcb_module (shared)
 ssl_module (shared)
 status_module (shared)
 suexec_module (shared)
# Add One Entry Per Line

##############################################################################
#        ___                 __                                              #
#       / _ | ___  ___ _____/ /  ___                                         #
#      / __ |/ _ \/ _ `/ __/ _ \/ -_)                                        #
#     /_/ |_/ .__/\_,_/\__/_//_/\__/                                         #
#        __/_/        __   ___       __     ___  __         __               #
#       / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____     #
#      / _  / _ `/ _  /  / _  / _ \/ __/  / _  / / _ \/ __/  '_/ -_) __/     #
#     /____/\_,_/\_,_/  /____/\___/\__/  /____/_/\___/\__/_/\_\\__/_/        #
#                                                                            #
##############################################################################

# BY DEFAULT THE EXAMPLES BELOW ARE COMMENTED OUT AND HENCE NOT ENABLED
# ADD ONLY ONE "Require ip" COMMAND PER LINE !

#Require ip 192.168.1.0 
#Require ip 192.168.2.0 

# !!!!!! ********************************************************
# DO NOT EVER USE 127.0.0.1 only real public facing IP addresses.
# !!!!!! ********************************************************

Require ip 185.170.112.156
# EDIT THIS FILE AS YOU LIKE TO WHITELIST YOUR OWN DOMAIN NAMES AND SPARE THEM FROM ANY REFERRER CHECKING ###

##############################################################################
#        ___                 __                                              #
#       / _ | ___  ___ _____/ /  ___                                         #
#      / __ |/ _ \/ _ `/ __/ _ \/ -_)                                        #
#     /_/ |_/ .__/\_,_/\__/_//_/\__/                                         #
#        __/_/        __   ___       __     ___  __         __               #
#       / _ )___ ____/ /  / _ )___  / /_   / _ )/ /__  ____/ /_____ ____     #
#      / _  / _ `/ _  /  / _  / _ \/ __/  / _  / / _ \/ __/  '_/ -_) __/     #
#     /____/\_,_/\_,_/  /____/\___/\__/  /____/_/\___/\__/_/\_\\__/_/        #
#                                                                            #
##############################################################################

# Add One Entry Per Line - List all your own domains of the sites you host on the server
# This file must exist on your system or Nginx will fail a reload due to a missing file
# Automatic updates will never be able to remove this custom list of yours 
# Add One Entry Per Line

# Make sure any domains have dots and special characters escaped as per the Regex examples below.
# For example myfirstowndomainname.com should be entered as myfirstowndomainname\.com
# and my-second-owndomainname.com should be entered as my\-second\-owndomainname\.com

# BY DEFAULT ALL THE EXAMPLES BELOW ARE COMMENTED OUT AND HENCE NOT ENABLED

    #SetEnvIfNoCase Referer ~*yourdomain\.com good_ref
    #SetEnvIfNoCase Referer ~*your\-domain\.com good_ref

SetEnvIfNoCase Referer ~*anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*analytics\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*ark\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*discord\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*img\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*linkus\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*pap\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*papdiscord\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*poll\.anzahcraft\.de good_ref
SetEnvIfNoCase Referer ~*anzah\.network good_ref
SetEnvIfNoCase Referer ~*search\.anzah\.network good_ref
SetEnvIfNoCase Referer ~*anzah\.tools good_ref
SetEnvIfNoCase Referer ~*rollenspiel\.monster good_ref
SetEnvIfNoCase Referer ~*discord\.rollenspiel\.monster good_ref
SetEnvIfNoCase Referer ~*poll\.rollenspiel\.monster good_ref
SetEnvIfNoCase Referer ~*rollenspiel\.wiki good_ref
mitchellkrogza commented 5 years ago

This is too weird. The only difference I can find between the working tests and my server is you have access_compat enabled which "should not" make a difference but please try

sudo a2dismod access_compat
sudo service apache2 restart

Let me know.

If that does not work, then please try disabling any whitelisting or mods made to any of the includes. BEST to just back up the entire /custom.d folder and grab a fresh copy of everything without making a single modification to anything.

Tealk commented 5 years ago

wow it shouldn't but somehow it does influence.

root:~# a2dismod access_compat
Module access_compat disabled.
To activate the new configuration, you need to run:
  systemctl restart apache2
root:~# service apache2 restart
root:~# curl -IA "80legs" https://anzahcraft.de                           HTTP/2 500
date: Thu, 22 Aug 2019 14:17:37 GMT
server: Apache
strict-transport-security: max-age=31536000
content-type: text/html; charset=iso-8859-1

but then I can no longer use my geoip block :( .htaccess: Invalid command 'Deny', perhaps misspelled or defined by a module not included in the server configuration

<IfModule mod_geoip.c>

    GeoIPEnable On
    # https://dev.maxmind.com/geoip/legacy/codes/iso3166/
    SetEnvIf GEOIP_COUNTRY_CODE CN DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KR DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KP DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RU DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RO DenyCountry
    deny from env=DenyCountry

    deny from 193.31

    <RequireAll>
        Require all granted
    </RequireAll>
</IfModule>
mitchellkrogza commented 5 years ago

That's very weird indeed, my tests though with access_compat disabled / enabled yield the same results, a working blocker.

It's likely you have a conflict, perhaps even the geoip module that is causing a problem?

Try disabling the mod_geoip and then testing with access_compat enabled and disabled.

mitchellkrogza commented 5 years ago

This block in geoip, if loaded in the wrong place "could" break the entire workings of the blocker. 

    <RequireAll>
        Require all granted
    </RequireAll>

Any <RequireAll> or <RequireAny> block in the wrong place could break things horribly, not only the blocker but all of Apache's security. I spent months figuring out the new Apache 2.4 mechanisms to get this blocker to work properly so I know how one <RequireAny> or <RequireAll> block can break everything.

Really worth disabling any extra modules you have loaded, start with getting the blocker working, and then one by one bring in an extra module at a time, if things break then you know where to look.

Tealk commented 5 years ago

Is it wrong how I used the ? I need to see how I can do this because the whole thing is a live system I don't want to be hanged by my users.

mitchellkrogza commented 5 years ago

The blocker will never break a live site it will only block what is has been told to block. Start with getting the blocker working without access_compat enabled. When the blocker is working then we can look at your geoip module.

You must know access_compat is not good to use on Apache 2.4 because it leads to using old security directives mixed with new.

mitchellkrogza commented 5 years ago

This guide for using geoip makes no use of any <RequireAll> blocks https://znil.net/index.php/Ubuntu_14.04.x_Apache_2.4.7_GeoIP_Blocking_mit_IPv4_und_IPv6_einrichten

Ubuntu 14.04.x Apache 2.4.7 GeoIP Blocking mit IPv4 und IPv6 einrichten – znilwiki
mitchellkrogza commented 5 years ago

Tested and Working 100% !!! Does not break blocker and DOES block the countries. I tested through a VPN.

Add this to your vhost config just before the closing </Virtualhost>

<Location />
    GeoIPEnable On
    # https://dev.maxmind.com/geoip/legacy/codes/iso3166/
    SetEnvIf GEOIP_COUNTRY_CODE CN DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KR DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KP DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RU DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RO DenyCountry
        Deny from env=DenyCountry
</Location>
mitchellkrogza commented 5 years ago

:point_up: requires mod access_compat

Tealk commented 5 years ago

I haven't finished everything yet but I wanted to say thank you that you invest so much time and effort to help me.

The blocker will never break a live site i

I didn't mean to say that either. I meant disabling all modules.

Tealk commented 5 years ago

geoip is in vhost according to your model and status remains at 200

curl -IA "80legs" https://anzahcraft.de
HTTP/2 200
mitchellkrogza commented 5 years ago

Post your latest apache2.conf and virtualhost config I've tested with geoip and blocker and it works 100% so you have something else wrong

Tealk commented 5 years ago 
# This is the main Apache server configuration file.  It contains the
# configuration directives that give the server its instructions.
# See http://httpd.apache.org/docs/2.4/ for detailed information about
# the directives and /usr/share/doc/apache2/README.Debian about Debian specific
# hints.
#
#
# Summary of how the Apache 2 configuration works in Debian:
# The Apache 2 web server configuration in Debian is quite different to
# upstream's suggested way to configure the web server. This is because Debian's
# default Apache2 installation attempts to make adding and removing modules,
# virtual hosts, and extra configuration directives as flexible as possible, in
# order to make automating the changes and administering the server as easy as
# possible.

# It is split into several files forming the configuration hierarchy outlined
# below, all located in the /etc/apache2/ directory:
#
#   /etc/apache2/
#   |-- apache2.conf
#   |   `--  ports.conf
#   |-- mods-enabled
#   |   |-- *.load
#   |   `-- *.conf
#   |-- conf-enabled
#   |   `-- *.conf
#   `-- sites-enabled
#       `-- *.conf
#
#
# * apache2.conf is the main configuration file (this file). It puts the pieces
#   together by including all remaining configuration files when starting up the
#   web server.
#
# * ports.conf is always included from the main configuration file. It is
#   supposed to determine listening ports for incoming connections which can be
#   customized anytime.
#
# * Configuration files in the mods-enabled/, conf-enabled/ and sites-enabled/
#   directories contain particular configuration snippets which manage modules,
#   global configuration fragments, or virtual host configurations,
#   respectively.
#
#   They are activated by symlinking available configuration files from their
#   respective *-available/ counterparts. These should be managed by using our
#   helpers a2enmod/a2dismod, a2ensite/a2dissite and a2enconf/a2disconf. See
#   their respective man pages for detailed information.
#
# * The binary is called apache2. Due to the use of environment variables, in
#   the default configuration, apache2 needs to be started/stopped with
#   /etc/init.d/apache2 or apache2ctl. Calling /usr/bin/apache2 directly will not
#   work with the default configuration.

# Global configuration
#

#
# ServerRoot: The top of the directory tree under which the server's
# configuration, error, and log files are kept.
#
# NOTE!  If you intend to place this on an NFS (or otherwise network)
# mounted filesystem then please read the Mutex documentation (available
# at <URL:http://httpd.apache.org/docs/2.4/mod/core.html#mutex>);
# you will save yourself a lot of trouble.
#
# Do NOT add a slash at the end of the directory path.
#
#ServerRoot "/etc/apache2"

#
# The accept serialization lock file MUST BE STORED ON A LOCAL DISK.
#
#Mutex file:${APACHE_LOCK_DIR} default

#
# The directory where shm and other runtime files will be stored.
#

DefaultRuntimeDir ${APACHE_RUN_DIR}

#
# PidFile: The file in which the server should record its process
# identification number when it starts.
# This needs to be set in /etc/apache2/envvars
#
PidFile ${APACHE_PID_FILE}

#
# Timeout: The number of seconds before receives and sends time out.
#
Timeout 300

#
# KeepAlive: Whether or not to allow persistent connections (more than
# one request per connection). Set to "Off" to deactivate.
#
KeepAlive On

#
# MaxKeepAliveRequests: The maximum number of requests to allow
# during a persistent connection. Set to 0 to allow an unlimited amount.
# We recommend you leave this number high, for maximum performance.
#
MaxKeepAliveRequests 500

#
# KeepAliveTimeout: Number of seconds to wait for the next request from the
# same client on the same connection.
#
KeepAliveTimeout 5

# These need to be set in /etc/apache2/envvars
User ${APACHE_RUN_USER}
Group ${APACHE_RUN_GROUP}

#
# HostnameLookups: Log the names of clients or just their IP addresses
# e.g., www.apache.org (on) or 204.62.129.132 (off).
# The default is off because it'd be overall better for the net if people
# had to knowingly turn this feature on, since enabling it means that
# each client request will result in AT LEAST one lookup request to the
# nameserver.
#
HostnameLookups Off

# ErrorLog: The location of the error log file.
# If you do not specify an ErrorLog directive within a <VirtualHost>
# container, error messages relating to that virtual host will be
# logged here.  If you *do* define an error logfile for a <VirtualHost>
# container, that host's errors will be logged there and not here.
#
ErrorLog ${APACHE_LOG_DIR}/error.log

#
# LogLevel: Control the severity of messages logged to the error_log.
# Available values: trace8, ..., trace1, debug, info, notice, warn,
# error, crit, alert, emerg.
# It is also possible to configure the log level for particular modules, e.g.
# "LogLevel info ssl:warn"
#
LogLevel warn

# Include module configuration:
IncludeOptional mods-enabled/*.load
IncludeOptional mods-enabled/*.conf

# Include list of ports to listen on
Include ports.conf

# Sets the default security model of the Apache2 HTTPD server. It does
# not allow access to the root filesystem outside of /usr/share and /var/www.
# The former is used by web applications packaged in Debian,
# the latter may be used for local directories served by the web server. If
# your system is serving content from a sub-directory in /srv you must allow
# access here, or in any related virtual host.
<Directory />
    Options FollowSymLinks
    AllowOverride All
    Require all denied
</Directory>

<Directory /usr/share>
    AllowOverride None
    Require all granted
</Directory>

<Directory /var/www/>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

#<Directory /srv/>
#   Options Indexes FollowSymLinks
#   AllowOverride None
#   Require all granted
#</Directory>

<Directory "/var/customers/webs">
    AllowOverride All
    Options FollowSymLinks
    Include /etc/apache2/custom.d/globalblacklist.conf
</Directory>

# AccessFileName: The name of the file to look for in each directory
# for additional configuration directives.  See also the AllowOverride
# directive.
#
AccessFileName .htaccess

#
# The following lines prevent .htaccess and .htpasswd files from being
# viewed by Web clients.
#
<FilesMatch "^\.ht">
    Require all denied
</FilesMatch>

#
# The following directives define some format nicknames for use with
# a CustomLog directive.
#
# These deviate from the Common Log Format definitions in that they use %O
# (the actual bytes sent including headers) instead of %b (the size of the
# requested file), because the latter makes it impossible to detect partial
# requests.
#
# Note that the use of %{X-Forwarded-For}i instead of %h is not recommended.
# Use mod_remoteip instead.
#
LogFormat "%v:%p %h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" vhost_combined
LogFormat "%h %l %u %t \"%r\" %>s %O \"%{Referer}i\" \"%{User-Agent}i\"" combined
LogFormat "%h %l %u %t \"%r\" %>s %O" common
LogFormat "%{Referer}i -> %U" referer
LogFormat "%{User-agent}i" agent

# Include of directories ignores editors' and dpkg's backup files,
# see README.Debian for details.

# Include generic snippets of statements
IncludeOptional conf-enabled/*.conf

# Include the virtual host configurations:
IncludeOptional sites-enabled/*.conf
IncludeOptional sites-froxlor/*.conf

# vim: syntax=apache ts=4 sw=4 sts=4 sr noet

ServerSignature Off
ServerTokens Prod

# [For HTTPS Support]
Protocols h2 http/1.1

# [For HTTP Support]
Protocols h2c http/1.1

SSLOpenSSLConfCmd DHParameters "/etc/ssl/certs/dhparams4096.pem"
SSLOpenSSLConfCmd ECDHParameters secp384r1
SSLOpenSSLConfCmd Curves secp521r1:secp384r1
# 35_froxlor_normal_vhost_anzahcraft.de.conf
# Created 22.08.2019 19:55
# Do NOT manually edit this file, all changes will be deleted after the next domain change at the panel.

# Domain ID: 23 - CustomerID: 1 - CustomerLogin: Anzah
<VirtualHost 185.170.112.156:80 [2a03:4000:15:68f::b9aa:709c]:80>
  ServerName anzahcraft.de
  ServerAlias www.anzahcraft.de
  ServerAdmin MAIL
  <IfModule mod_rewrite.c>
    RewriteEngine On
    RewriteCond %{HTTPS} off
    RewriteCond %{REQUEST_URI} !^/\.well-known/acme-challenge
    RewriteRule ^/(.*) https://%{HTTP_HOST}/$1 [R=301;L,NE]
  </IfModule>
  <IfModule !mod_rewrite.c>
    Redirect 301 / https://anzahcraft.de/
  </IfModule>
</VirtualHost>
# 35_froxlor_ssl_vhost_anzahcraft.de.conf
# Created 22.08.2019 19:55
# Do NOT manually edit this file, all changes will be deleted after the next domain change at the panel.

# Domain ID: 23 (SSL) - CustomerID: 1 - CustomerLogin: Anzah
<VirtualHost 185.170.112.156:443 [2a03:4000:15:68f::b9aa:709c]:443>
  ServerName anzahcraft.de
  ServerAlias www.anzahcraft.de
  ServerAdmin MAIL
  SSLEngine On
  SSLProtocol -ALL +TLSv1.2 +TLSv1.3
 Protocols h2 http/1.1
  SSLCompression Off
  SSLHonorCipherOrder On
  SSLCipherSuite ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-CHACHA20-POLY1305:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-RSA-AES128-GCM-SHA256
  SSLVerifyDepth 10
  SSLCertificateFile /etc/ssl/froxlor-custom/anzahcraft.de.crt
  SSLCertificateKeyFile /etc/ssl/froxlor-custom/anzahcraft.de.key
  SSLCACertificateFile /etc/ssl/froxlor-custom/anzahcraft.de_CA.pem
  SSLCertificateChainFile /etc/ssl/froxlor-custom/anzahcraft.de_chain.pem
  SSLUseStapling on
  <IfModule mod_headers.c>
    Header always set Strict-Transport-Security "max-age=31536000"
  </IfModule>
  DocumentRoot "/var/customers/webs/Anzah/anzahcraft.de/"
  SuexecUserGroup "USER" "USER"
  <FilesMatch \.(php)$>
  SetHandler proxy:unix:/var/lib/apache2/fastcgi/4-anzah-anzahcraft.de-php-fpm.socket|fcgi://localhost
  </FilesMatch>
  <Directory "/var/customers/webs/Anzah/anzahcraft.de/">
    Require all granted
    AllowOverride All
  </Directory>
  Alias /webalizer "/var/customers/webs/Anzah/webalizer/anzahcraft.de"
  LogLevel warn
  ErrorLog "/var/customers/logs/Anzah-anzahcraft.de-error.log"
  CustomLog "/dev/null" combined
<Location />
    GeoIPEnable On
    # https://dev.maxmind.com/geoip/legacy/codes/iso3166/
    SetEnvIf GEOIP_COUNTRY_CODE CN DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KR DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KP DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RU DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RO DenyCountry
        Deny from env=DenyCountry
</Location>
</VirtualHost>
Tealk commented 5 years ago

You must know access_compat is not good to use on Apache 2.4 because it leads to using old security directives mixed with new.

I didn't know that at all but unfortunately it is needed for the geoip :'(

mitchellkrogza commented 5 years ago

Your main Directory block in the SSL host is missing the blocker include

<Directory "/var/customers/webs/Anzah/anzahcraft.de/">
    Require all granted
    AllowOverride All
  </Directory>
Tealk commented 5 years ago

I wrote this in the apache2.conf because froxlor doesn't give me the possibility to intervene there.

mitchellkrogza commented 5 years ago

I don't think it will work properly from apache2.conf not sure I'd have to test that but I doubt it, should be enclosed in the <Virtualhost> block

Tealk commented 5 years ago

yes only unfortunately I have no possibility to install it there :(

mitchellkrogza commented 5 years ago

I will look tomorrow it will need a mod to their templates

Tealk commented 5 years ago

If it helps you with that: https://github.com/Froxlor/Froxlor

GitHub
Froxlor/Froxlor
The server administration software for your needs - The official Froxlor development Git repository - Froxlor/Froxlor
mitchellkrogza commented 5 years ago

Thanks I already forked a copy to look through tomorrow

mitchellkrogza commented 5 years ago

@Tealk please try this.

Backup this config file (wherever froxlor is installed?) change froxlordir below to the correct folder

sudo cp froxlordir/lib/Froxlor/Cron/Http/Apache.php froxlordir/lib/Froxlor/Cron/Http/Apache.bak

Then edit the original file

sudo nano froxlordir/lib/Froxlor/Cron/Http/Apache.php

Change lines 653-655

FROM THIS

            if (Settings::Get('system.apache24') == '1') {
                $webroot_text .= '    Require all granted' . "\n";
                $webroot_text .= '    AllowOverride All' . "\n";

TO THIS

            if (Settings::Get('system.apache24') == '1') {
                $webroot_text .= '    Require all granted' . "\n";
                $webroot_text .= '    AllowOverride All' . "\n";
                $webroot_text .= '    Include /etc/apache2/custom.d/globalblacklist.conf' . "\n";

Let me know.

NOTE: This is very temporary hack and should work just fine, when you update Froxlor however you will need to re-do this step. Unfortunately I don't have time to contribute any proper changes to Froxlor to have it as an option in the admin menu but it really should not be difficult at all for someone to do.

If this works then you can further edit that block to add your geoip block as follows.

            if (Settings::Get('system.apache24') == '1') {
                $webroot_text .= '    Require all granted' . "\n";
                $webroot_text .= '    AllowOverride All' . "\n";
                $webroot_text .= '    Include /etc/apache2/custom.d/globalblacklist.conf' . "\n";
                $webroot_text .= '    <Location />' . "\n";
                $webroot_text .= '    GeoIPEnable On' . "\n";
                $webroot_text .= '    SetEnvIf GEOIP_COUNTRY_CODE CN DenyCountry' . "\n";
                $webroot_text .= '    SetEnvIf GEOIP_COUNTRY_CODE KR DenyCountry' . "\n";
                $webroot_text .= '    SetEnvIf GEOIP_COUNTRY_CODE KP DenyCountry' . "\n";
                $webroot_text .= '    SetEnvIf GEOIP_COUNTRY_CODE RU DenyCountry' . "\n";
                $webroot_text .= '    SetEnvIf GEOIP_COUNTRY_CODE RO DenyCountry' . "\n";
                $webroot_text .= '    Deny from env=DenyCountry' . "\n";
                $webroot_text .= '    </Location>' . "\n";

NOTE: This will apply it to every single Vhost.

Tealk commented 5 years ago

NOTE: This will apply it to every single Vhost. Yes that would be a problem, I would like to use the GEOIP only with certain web pages.

i have now entered this in line 681 but i have nothing in the vhost

image

mitchellkrogza commented 5 years ago

Did you re-generate the host file?

Tealk commented 5 years ago

yes I have rebuilt all configs. geoip I entered in the webinterface that was taken over

  <Directory "/var/customers/webs/Anzah/anzahcraft.de/">
    Require all granted
    AllowOverride All
  </Directory>
  Alias /webalizer "/var/customers/webs/Anzah/webalizer/anzahcraft.de"
  LogLevel warn
  ErrorLog "/var/customers/logs/Anzah-anzahcraft.de-error.log"
  CustomLog "/dev/null" combined
<Location />
    GeoIPEnable On
    # https://dev.maxmind.com/geoip/legacy/codes/iso3166/
    SetEnvIf GEOIP_COUNTRY_CODE CN DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KR DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE KP DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RU DenyCountry
    SetEnvIf GEOIP_COUNTRY_CODE RO DenyCountry
        Deny from env=DenyCountry
</Location>
</VirtualHost>
mitchellkrogza commented 5 years ago

Try make the very same changes on lines 73-75