search

mitchellkrogza / apache-ultimate-bad-bot-blocker

Apache Block Bad Bots, (Referer) Spam Referrer Blocker, Vulnerability Scanners, Malware, Adware, Ransomware, Malicious Sites, Wordpress Theme Detectors and Fail2Ban Jail for Repeat Offenders
Other
800 stars 173 forks source link

[HELP] I can write the robots.txt file, when i test the bIRLbot badbot name #129

Closed ZerooCool closed 5 years ago

ZerooCool commented 5 years ago

I have add the list to my htaccess website file, for Apache 2.4 : https://github.com/mitchellkrogza/apache-ultimate-bad-bot-blocker/blob/master/_htaccess_versions/htaccess-mod_rewrite.txt

To my second website, i add a file badbot-request.php with :

<?php
ini_set("user_agent","bIRLbot");
$c2 = file_get_contents("https://www.visionduweb.fr/robots.txt");
?>
<html>
<body>
<h1>Done ...</h1>
<?php
echo c2;
echo $c2;
?>

I can write the robots.txt file, when i test the bIRLbot badbot name.

I may have something wrong? The request should not be blocked by the script?

ZerooCool commented 5 years ago

The same with : curl -A "bIRLbot" https://www.visionduweb.fr/robots.txt

mitchellkrogza commented 5 years ago 
curl -A "bIRLbot" -I https://www.visionduweb.fr
HTTP/1.1 200 OK
Date: Fri, 28 Jun 2019 17:02:16 GMT
Server: Apache
Expires: Wed, 17 Aug 2005 00:00:00 GMT
Cache-Control: no-store, no-cache, must-revalidate, post-check=0, pre-check=0
Pragma: no-cache
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
Set-Cookie: f4e8b27a5048454220760e16fe525d93=9e8eouvnpc8m9m1rc4o07ompmk; path=/; HttpOnly;HttpOnly;Secure
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
Feature-Policy: geolocation none;midi none;notifications none;push none;sync-xhr self;microphone none;camera none;magnetometer none;gyroscope none;speaker self;vibrate none;fullscreen self;payment none;
Last-Modified: Fri, 28 Jun 2019 17:02:19 GMT
X-XSS-Protection: 1; mode=block
X-Content-Type-Options: nosniff
Access-Control-Allow-Origin: default-src 'self' https://www.visionduweb.fr
Content-Security-Policy: default-src 'self' https://www.visionduweb.fr; script-src 'self' 'unsafe-eval' 'unsafe-inline' https://www.visionduweb.fr https://www.youtube.com https://unpkg.com https://s.ytimg.com https://www.google.com https://www.gstatic.com; object-src 'self' https://www.visionduweb.fr; style-src 'self' 'unsafe-inline' https://fonts.googleapis.com; img-src 'self' https://www.visionduweb.fr https://i.ytimg.com https://secure.gravatar.com https://avatars3.githubusercontent.com; media-src 'self' https://www.visionduweb.fr https://youtu.be; frame-src 'self' https://www.visionduweb.fr https://www.youtube.com https://www.coingecko.com https://hackbbs.org:7777 https://www.spreaker.com https://widget.spreaker.com https://www.google.com; font-src 'self' 'unsafe-inline' https://www.visionduweb.fr https://fonts.gstatic.com data:; connect-src 'self' https://www.visionduweb.fr https://api.github.com; frame-ancestors 'self'
Content-Type: text/html; charset=utf-8
mitchellkrogza commented 5 years ago 
curl -A "Nutch" -I https://www.visionduweb.fr
HTTP/1.1 302 Found
Date: Fri, 28 Jun 2019 17:03:12 GMT
Server: Apache
Strict-Transport-Security: max-age=63072000; includeSubDomains; preload
X-Frame-Options: SAMEORIGIN
Referrer-Policy: no-referrer-when-downgrade
Feature-Policy: geolocation none;midi none;notifications none;push none;sync-xhr self;microphone none;camera none;magnetometer none;gyroscope none;speaker self;vibrate none;fullscreen self;payment none;
Location: https://www.visionduweb.fr/403-forbidden.php
Cache-Control: max-age=604800
Expires: Fri, 05 Jul 2019 17:03:12 GMT
Content-Type: text/html; charset=iso-8859-1

shows blocker / htaccess is working

mitchellkrogza commented 5 years ago

Not sure what you are doing but I can't support any custom scripts, only whether the blocker works or doesn't which it does.

ZerooCool commented 5 years ago

Thank you very much for your answer. I tried to check that the blockage is functional. I'm learning and reviewing the method to test a user agent. I realized from your answer that my test had an error!

When I looked at one of the proposed lines: RewriteCond% {HTTP_USER_AGENT} \ bIRLbot \ b [NC, OR]

I was wrong in writing the query: curl -A "bIRLbot" -I https://www.visionduweb.fr

This query works: curl -A "IRLbot" -I https://www.visionduweb.fr

So I can integrate this script, knowing that it works. Thank you !

Vision du web - La Réponse Collaborative
Accueil
Accueil
ZerooCool commented 5 years ago

You can add me to the group, to feed the wiki, I could translate an example of use, in English, to explain the integration in .htacces or VirtualHost, and, how to test the proper functioning.

https://wiki.visionduweb.fr/index.php?title=Configurer_le_fichier_.htaccess#Une_liste_de_plus_de_7000_bots_bloqu.C3.A9s_avec_.htaccess

Analyse Développement Hacking
Configurer le fichier .htaccess