add Redis support

[SoledaD208]

Hi kyprizel, I would like to say that this module is really incredible. I'm using it for almost all of my system which i manage. But, imo, there's still a small problem that, sometimes we need this module work in other way. Normally, the idea for verifying bot is to block all but allow some in whitelist, sometimes we will need it in opposite way, allow all, but block only some in blacklist, and this blacklist will need to be added/removed dynamically without reloading Nginx. Thinking about this, i did some changes to integrate this great module with Redis, we will store blacklist IP in Redis, then enable bot verification only if request's remote_add in this blacklist. Of course, this new feature is not conflict with the famous original one, we can use original for some locations, and Redis feature for others, which need "allow all but block some" idea. I tested successfully with Nginx 1.16.0 and 1.14.2, other versions should work well, too. So, please consider this change, for a more flexible module. Thanks!

On branch redis-support Changes to be committed: modified: README modified: src/ngx_http_testcookie_access_module.c

[Mecanik]

Amazing contribution! Unfortunately the author does not bother anymore... even to answer.

On a side note, this feature is not really helpful unless we have a way of automatically add IP's into the list.

PS: Does IPv6 also work in the list ?

[Mecanik]

Also in addition to the above, I see you are opening a connection on each request to redis... why not use a persistent connection ?

[kyprizel]

I don't think this feature is mature enough to be merged. Almost the same functionality can be achieved using https://github.com/kyprizel/testcookie-nginx-module/#testcookie_pass directive combined with existing redis modules.

[Mecanik]

I don't think this feature is mature enough to be merged. Almost the same functionality can be achieved using https://github.com/kyprizel/testcookie-nginx-module/#testcookie_pass directive combined with existing redis modules.

Indeed the feature is not developed properly, however testcookie_pass is quite (very) limited as well. You can only use 1 time this variable, and you cannot map combinations ofr IP/UA and other stuff.

[SoledaD208]

@kyprizel thanks for your reply, it's just my idea and it may not need for now. @Mecanik we can make use of some other analysis solutions to detect malicious IPs and add to Redis (for example: i use Elasticsearch to store nginx log, and write a small script to dig into ES, find out bot IPs and add to Redis). One more advantage of using Redis is you can set the expired time for each IP easily, so you may not need to care about releasing IPs after add it to blacklist. About the connection, for now it opens and uses only 1 connection for each worker, if this connection goes down, nginx will failed at first 1-2 requests but then ignore this module and let next requests go through, you will need to reload nginx when connection is back to normal. if you interested in my customization, let contribute here: https://github.com/SoledaD208/testcookie-nginx-module. actually i'm not really good at C/C++, so i think my code need to be improved more

[Mecanik]

It's absolutely pointless to use elastic search for this, one can just use fail2ban (with ssd storage) and achieve faster and better result without additional cost of money.

It would be better to integrate automatic ip "greylisting" using limit_req module, perhaps using the one you modified on your repo.

This idea with redis is very good and I shall make an input in this, hopefully kyprizel will accept it on the official repo.