Inquiry and Suggestions for Enhancing a Database Module's Functionality

[MiguelFGM]

Pretty nice module, but there is a question that remains in my head after reading the documentation:

1. Will each connection lead to a database lookup or are those maintained in some sort of caching?
2. Did any benchmark was done over the module?

Being that, obviously, for users who aim to build their databases this module has very potential for future usage, therefore I would request some kind of features such as:

1. Database reload: Such as the MaxMind module (geoip2), an auto reload time can be set, where, it will verify if the database has any change and apply those changes if they exist (See: https://github.com/leev/ngx_http_geoip2_module "auto_reload")
2. IP found behavior: Bringing flexibility to this database means, we can have a few other uses for the database such as databases where we enable Captchas if the IP is found on the database, to bring flexibility, in this case, the information (1 if found, 0 if not) could be saved into a variable that we could access (It's important that the "ipfilter_denied_url" keeps existing and it brings also some flexibility).

Those are both features that could be discussed, do you have some platform such as Skype, Telegram, Slack, or Discord where I could reach you to discuss further details?

Thank you!

[janbar]

Hi,

• The database is mounted in vitual memory (mmap). So if you have free memory all search read in hot page memory.

• I made benchmarks, and that is why I publish my code. On first run query response gets less than 100 usec, and next run get less than 5 usec. The memory pages are cached by the backend FS or swap. Response time doesn't depends on the number of stored rules. Even the database is full, you will get the same very short response time.

• Database reload ? You don't need ! The database can be updated on the fly using the CLI. Once you have started NGINX with the module enabled, you can update the database(s): Not by replacing the file, but using the CLI, as the following:

Launch ipfiltercli. >>> mount /etc/nginx/my_database.db Here you have mounted the db read-write, and so you can allow or deny or even purge the db to reload new rules. All changes are instantly available. >>> allow x.x.x.x/y >>> load deny cidr.txt (etc) >>> exit

[janbar]

Also you can update the database using any scheduler (cron or what ever) to run the cli in a script. Example:

#!/bin/sh
ipfiltercli <<EOF
mount /etc/nginx/my_database.txt
deny 0.0.0.0/0
load allow list_of_cidr_ok.txt
exit
EOF

Or the reverse (allow all, then denied few)

#!/bin/sh
ipfiltercli <<EOF
mount /etc/nginx/my_database.txt
allow 0.0.0.0/0
load deny list_of_cidr_ko.txt
exit
EOF

I let you imagine, what you could do ...

[MiguelFGM]

Also you can update the database using any scheduler (cron or what ever) to run the cli in a script. Example:

#!/bin/sh
ipfiltercli <<EOF
mount /etc/nginx/my_database.txt
deny 0.0.0.0/0
load allow list_of_cidr_ok.txt
exit
EOF

Or the reverse (allow all, then denied few)

#!/bin/sh
ipfiltercli <<EOF
mount /etc/nginx/my_database.txt
allow 0.0.0.0/0
load deny list_of_cidr_ko.txt
exit
EOF

I let you imagine, what you could do ...

Hey Hey!

Thanks for your explanation! Everything seems alright regarding all the questions I made, just a small one missing regarding this.

In my case I'm not looking to deny, allow, or provide a page for the IP, I'm looking to do a database for malicious connections (flagged by my system), import them into the database (your module in this case) and captcha all the connections that matches with the database.

Now this works like NGINX testcookie using a directive such as "captcha on;" and this directive works either in server block, location block, or if block. I could indeed create an internal location for the "ipfilter_denied_url" and captcha on there, but I had to add an extra verification to see if the client has the correct captcha cookies before sending him to the internal location and this would lead to more resources usage, so I'm thinking in a way that I could direct use the "captcha on;" for all the flagged IPs, that's why I gave the variable example that could be 1 or 0, for example:

• If the module would check if the IP is in the database by adding 1 or 0 in the variable "test", I had simply to do the:

  if ($test = 1) {
  captcha on;
  }

Do you believe that with current integration we could achieve a similar behavior?

[janbar]

I guess you mean the module could set a nginx shared variable. Yes it is possible. For now the module redirect to the denied url. But it could set a variable instead, and so depending of a new directive or if the directive "denied url" hasn't been specified.

the value could be : 0=not_found , 1=allow, 2=deny OR only 0=not_found, 1=(allow or deny)

even 2 variables, as: ipfilter_state and ipfilter_found

[MiguelFGM]

@janbar Thank you for your fast responses! Yea, that could bring a big flexibility, would it be too much if I ask for such implementation?

[janbar]

I made some changes to add the variable "$ipfilter". Please check the README file, or have a look of the commit c2ff8a1ea70178a692ee6d319342df7a57838e06.

Note that, the database with default size (seg size=256) can stores up to 300K IP (CIDR with prefix=32). If you need more then create a database with seg size of 512 (500K IP) or 1024 (1M IP). But attention, for 512 you need 64MB of virtual memory, and 1024 -> 128MB.