search

tlhackque / BlockCountries

iptables manager for IP blocking by country
Other
32 stars 6 forks source link

centos 5.11 only issue #10

Closed budgierless closed 8 years ago

budgierless commented 8 years ago

hi I am using this script on two ubuntu systems with no issue, but i install it on contos, and all is working even the list but i do get this error:

~]# /etc/init.d/BlockCountries start -update Starting blocked countries IP filter: No new IP data available from apnic No new IP data available from lacnic No new IP data available from afrinic Updated IP zone data from ripe No new IP data available from arin ip6tables-restore v1.3.5: ip6tables-restore: unable to initializetable 'filter'

Error occurred at line: 1 Try `ip6tables-restore -h' or 'ip6tables-restore --help' for more information.

tlhackque commented 8 years ago

I'm glad that you have had some success.

That message says that the ip6tables filter that BlockCountries generated doesn't load.

The 'filter' table is internal to your kernel and provided by netfilter.

Is netfilter configured in your kernel?

Is IPv6 running? What does ifconfig report?

Is netfilter (iptables) working? What does ip6tables -nvL report?

redirect those to a file and post them -- as ATTACHMENTs.

If you just start BlockCountries on IPv4, does it work?

If not, what are the errors? What does iptables -nvL report?

I don't know anything about 'contos'. If you mean 'centos', that's the same as fedora, which I use for most of my machines. BlockCountries runs here on kernels as old as 2.6.17, Fedora Core 4. (though IPv6 has issues on kernels that old.)

What are the versions of your OS, iptables, ip6tables?

uname -a

cat /proc/sys/kernel/osrelease

iptables --version ip6tables --version

tlhackque commented 8 years ago

Also: post, again as an ATTACHMENT, the output of

BlockCountries start -d -6 -no4 2>ipv6.table.

This will be large, but it includes exactly what is fed to ip6tables-restore

budgierless commented 8 years ago

hi, sorry i have not had the change to look into it yet as today is very busy, but thanks for your post as it will give me good ideas of what to check, but from the research i already did, it seems that i maybe missing (xtables-addons-common) this may not be the case, but when i get a chance later I will reply with your requested outputs.

budgierless commented 8 years ago

uname -a Linux ams 2.6.18-408.el5 #1 SMP Tue Jan 19 09:14:52 EST 2016 x86_64 x86_64 x86_64 GNU/Linux cat /proc/sys/kernel/osrelease 2.6.18-408.el5 iptables --version iptables v1.3.5 ip6tables --version ip6tables v1.3

/etc/init.d/BlockCountries start -d -6 -no4 2>ipv6.table Starting blocked countries IP filter: Read /root/blockips/cn.cdb Read /root/blockips/tr.cdb[root@ams ~]

tlhackque commented 8 years ago

2.6.18 is a very old kernel. I'm not sure it's new enough for reliable operation with IPv6, especially IPv6 with ip6tables. I know 2.6.17 has IPv6-related bugs in netfilter - but it does get further than you report.

Do you actually have IPv6 connectivity with this machine? If not, remove the -6 from your configuration file and BlockCountries will ignore IPv6, which should solve your problem.

If you do have IPv6 connectivity, you should update your kernel to something much more current in any case.

You didn't attach ipv6.table.

You didn't include the output of ifconfig

You didn't attach the output of ip6tables -nvL and iptables -nvL

You didn't tell me if IBlockCountries works with just IPv4. (BlockCountries start -no6 -4)

budgierless commented 8 years ago

yes maybe issue id because of ole kernel, and as on that system i don't use iv6 i have disabled it in the config like your said, and the issue is gone, thanks for you help, I have your script running on three systems now, two ubuntu and one centOS.

tlhackque commented 8 years ago

If you don't use iPv6, disabling it is a good idea in any case. The BlockCountries processing costs something - more importantly, the iptables rules occupy kernel memory. No point in that if you have no traffic.

You could consider adding an iptables rule that blocks all IPv6 traffic (except ::1) just in case IPv6 connectivity is added unexpectedly.

I'm closing this issue since you are running and haven't provided the additional data.

By the way, if you're going to install on more machines, get the latest release. bcinstall has been updated to be more helpful...

budgierless commented 8 years ago

[root@ams ~]# /etc/init.d/BlockCountries stop Removing blocked countries IP filteriptables-restore: line 1511 failed Table update failed: 256 [root@ams ~]# /etc/init.d/BlockCountries start -no6 -4 Starting blocked countries IP filter: iptables-restore: line 7065 failed [root@ams ~]#

[root@ams ~]# ifconfig eth0 Link encap:Ethernet HWaddr 08:00:27:97:A5:35 inet addr:81.138.252.211 Bcast:81.138.252.215 Mask:255.255.255.248 UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1 RX packets:17688 errors:0 dropped:0 overruns:0 frame:0 TX packets:27649 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:1000 RX bytes:1799956 (1.7 MiB) TX bytes:3449579 (3.2 MiB)

lo Link encap:Local Loopback inet addr:127.0.0.1 Mask:255.0.0.0 UP LOOPBACK RUNNING MTU:16436 Metric:1 RX packets:2513 errors:0 dropped:0 overruns:0 frame:0 TX packets:2513 errors:0 dropped:0 overruns:0 carrier:0 collisions:0 txqueuelen:0 RX bytes:3769658 (3.5 MiB) TX bytes:3769658 (3.5 MiB)

[root@ams ~]# ip6tables -nvL ip6tables v1.3.5: can't initialize ip6tables table `filter': Address family not supported by protocol Perhaps ip6tables or your kernel needs to be upgraded.

iptables -nvL = too big to copy but look good to me

ipv6.table = dont know how to get this data sorry,

tlhackque commented 8 years ago

You created ipv6.table in your current directory when you executed: /etc/init.d/BlockCountries start -d -6 -no4 2>ipv6.table

If you're administering Unix systems, you ought to take a course on the shell. Or see man sh or man bash

ip6tables -nvL
 ip6tables v1.3.5: can't initialize ip6tables table `filter': Address family not supported by protocol
 Perhaps ip6tables or your kernel needs to be upgraded.

This rules out BlockCountries as an issue. It says that ip6tables by itself can't even list IPv6 filters. That means that either your startup for doesn't include ip6tables, or your kernel doesn't support it.

This is NOT good:

/etc/init.d/BlockCountries start -no6 -4
 Starting blocked countries IP filter: iptables-restore: line 7065 failed

This means the filter was not installed.

If you ever see a message that says 'error', 'failed', 'warning' or the like, something is wrong. Report it.

One thing to try first - you are running an old iptables, but it might have been patched by redhat.

So if you're using -conntrack, remove it and see if you get the error. Otherwise, add it ...

If that doesn't solve the problem, I need the result of /etc/init.d/BlockCountries start -d -no6 -4 2>ipv4.table. Which, as you've figured out by now, is the file ipv4.table. Please post this as an attachment.

I may need iptables -nvL. iptables -nvL >ipv4.iptables would capture that in a file (ipv4.iptables), which you would attach. But let's hold off until I can see ipv4.table

budgierless commented 8 years ago

[root@ams ~]# /etc/init.d/BlockCountries start -d -6 -no4 2>ipv6.table Starting blocked countries IP filter: Read /root/blockips/cn.cdb Read /root/blockips/tr.cdb[root@ams ~]# /etc/init.d/BlockCountries start -d -no6 -4 2>ipv4.table Starting blocked countries IP filter: Read /root/blockips/cn.cdb Read /root/blockips/tr.cdb[root@ams ~]#

result of /etc/init.d/BlockCountries start -d -no6 -4 2>ipv4.table

tr.txt cn.txt

budgierless commented 8 years ago

-conntrack = no change when added with -ipv6 on config

budgierless commented 8 years ago

with -ipv4 only in config [root@ams ~]# /etc/init.d/BlockCountries start -update Starting blocked countries IP filter: No new IP data available from apnic No new IP data available from lacnic No new IP data available from afrinic No new IP data available from ripe No new IP data available from arin iptables-restore: line 7065 failed Table update failed: 256

note: line 7065 same line error that showed up before when still using -ipv6, maybe the iptable-restore need a flush?

tlhackque commented 8 years ago

I asked for ipv4.table and you attached tr.txt and cn.txt. They appear to be renamed versions of cn.cdb and tr.cdb, which are compiled binary data for Turkey and China from apnic. This is not helpful at this time.

-conntrack is something to try -no6 -4 (that is, IPv4-only). The iptables-restore: line 7065 failed is more important than the IPv6 mystery. It means the IPv4 filter is not loading.

For some reason your iptables-restore doesn't like the data. -conntrack mismatch is one possibility.

Unfortunately, iptables-restore provides no useful diagnostics - even the line number is useless. So I need the data file. If I can reproduce your issue here, I can diagnose it. If not, you'll have to help.

budgierless commented 8 years ago

/etc/init.d/BlockCountries start -conntrack = same line 7065 error on output Had to rename to be able to be allowed to upload, as when i was trying .zip it still failed, but file are untouched so you can rename again if you like. ipv4.table.txt ipv6.table.txt

tlhackque commented 8 years ago

Renaming Is fine, sending another file isn't. Next time we'll create the file with a .zip extension.

Both files look like reasonable data. Some is missing, but that may be due to your version of iptables-restore interfering with my logging.

I've looked at the IPv4 table in some detail yesterday.

It appears that iptables-restore is failing to delete a rule chain, probably because there is a reference to that chain that shouldn't exist.

If BlockCountries is the only software touching its chains, this is impossible - at least in theory.

I decided that it's too complicated for you to selectively gather the data that I need to debug this.

I have added data collection to BlockCountries that should simplify gathering what I need to debug this.

Please download and install the latest version. (2.17)

Run its bcinstall as some new modules are required.

/etc/init.d/BlockCountries start -d -z 2>debug.zip

Post the terminal output, and the debug.zip file that's created.

Depending on your configuration, the (unzipped) debug log may be several MB in size. It contains some data about your system; I don't think it's sensitive. It's all text; you are welcome to review its contents.

You can post it somewhere else if you prefer.

The new version also logs start and stop to syslog. I recommend it, but -nosyslog will turn it off.

tlhackque commented 8 years ago

FYI, I found my note on IPv6 - the minimum kernel version required for stateful IPv6 filters is 2.6.20.

Although BlockCountries will start on earlier kernels, IPv6 connections will not pass the firewall reliably.

This is a kernel restriction, not a BlockCountries issue.

This is not to say that you should run a kernel as old as 2.6.20 or that netfilter in that version is bug-free. Just that prior versions, which include the one you reported running, are known not to work with IPv6.

2.6.20 was released in 2007. 2.6.18 was released in 2006, 10 years ago. Both are EOL.

If you can collect the necessary data (use the latest BlockCountries with -d -z 2>debug.zip, I still want to get to the bottom of the issue with iptables-restore. The latest version collects more data than my previous (deleted) post requested, but requires much less work from you.

budgierless commented 8 years ago

I've been away for few days and won't be back home for another 36 hours, but will be sure to send the data you have requested.

Sent from my iPhone

On 16 Feb 2016, at 01:40, Timothe Litt notifications@github.com wrote:

FYI, I found my note on IPv6 - the minimum kernel version required for stateful IPv6 filters is 2.6.20.

Although BlockCountries will start on earlier kernels, IPv6 connections will not pass the firewall reliably.

This is a kernel restriction, not a BlockCountries issue.

This is not to say that you should run a kernel as old as 2.6.20 or that netfilter in that version is bug-free. Just that prior versions, which include the one you reported running, are known not to work with IPv6.

2.6.20 was released in 2007. 2.6.18 was released in 2006, 10 years ago. Both are EOL.

If you can collect the necessary data (use the latest BlockCountries with -d -z 2>debug.zip, I still want to get to the bottom of the issue with iptables-restore. The latest version collects more data than my previous (deleted) post requested, but requires much less work from you.

— Reply to this email directly or view it on GitHub.

budgierless commented 8 years ago

/etc/init.d/BlockCountries start -update

Starting blocked countries IP filter: Updated IP zone data from apnic Updated IP zone data from lacnic No new IP data available from afrinic Updated IP zone data from ripe Updated IP zone data from ariniptables-restore: line 7068 failed Rules update failed: Broken pipe at line 0 [ FAILED ] [root@ams BlockCountries]# /etc/init.d/BlockCountries start -d -z 2>debug.zip Debug log will be written as a .zip archive Starting blocked countries IP filter: Read /root/blockips/cn.cdb Read /root/blockips/tr.cdb iptables-restore: line 7068 failed Rules update failed: Broken pipe at line 0 [ FAILED ]

Again zip will not upload for some reason, please down file from my website link

tlhackque commented 8 years ago

Thanks for the data.

.zip should upload per https://help.github.com/articles/file-attachments-on-issues-and-pull-requests/, contact github for help on that.

I've downloaded it from your website & will analyze it shortly.

tlhackque commented 8 years ago

The good news: With this data, I can reproduce your issue. Should have a solution shortly.

budgierless commented 8 years ago

Ok please remember that the data dose not include -IPv6 in config

Sent from my iPhone

On 18 Feb 2016, at 11:25, Timothe Litt notifications@github.com wrote:

The good news: With this data, I can reproduce your issue. Should have a solution shortly.

— Reply to this email directly or view it on GitHub.

tlhackque commented 8 years ago

Would you please post your /etc/sysconfig/iptables-config?

Also, please run (and post) 

# locale

I can reproduce the symptom, but it's not clear how it happens....

budgierless commented 8 years ago 
# Load additional iptables modules (nat helpers)
#   Default: -none-
# Space separated list of nat helpers (e.g. 'ip_nat_ftp ip_nat_irc'), which
# are loaded after the firewall rules are applied. Options for the helpers are
# stored in /etc/modprobe.conf.
IPTABLES_MODULES="ip_conntrack_netbios_ns"

# Unload modules on restart and stop
#   Value: yes|no,  default: yes
# This option has to be 'yes' to get to a sane state for a firewall
# restart or stop. Only set to 'no' if there are problems unloading netfilter
# modules.
IPTABLES_MODULES_UNLOAD="yes"

# Save current firewall rules on stop.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets stopped
# (e.g. on system shutdown).
IPTABLES_SAVE_ON_STOP="no"

# Save current firewall rules on restart.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/iptables if firewall gets
# restarted.
IPTABLES_SAVE_ON_RESTART="no"

# Save (and restore) rule and chain counter.
#   Value: yes|no,  default: no
# Save counters for rules and chains to /etc/sysconfig/iptables if
# 'service iptables save' is called or on stop or restart if SAVE_ON_STOP or
# SAVE_ON_RESTART is enabled.
IPTABLES_SAVE_COUNTER="no"

# Numeric status output
#   Value: yes|no,  default: yes
# Print IP addresses and port numbers in numeric format in the status output.
IPTABLES_STATUS_NUMERIC="yes"

# Verbose status output
#   Value: yes|no,  default: yes
# Print info about the number of packets and bytes plus the "input-" and
# "outputdevice" in the status output.
IPTABLES_STATUS_VERBOSE="no"

# Status output with numbered lines
#   Value: yes|no,  default: yes
# Print a counter/number for every rule in the status output.
IPTABLES_STATUS_LINENUMBERS="yes"

# Reload sysctl settings on start and restart
#   Default: -none-
# Space separated list of sysctl items which are to be reloaded on start.
# List items will be matched by fgrep.
#IPTABLES_SYSCTL_LOAD_LIST=".ip_conntrack .bridge-nf"
budgierless commented 8 years ago

[root@ams ~]# locale LANG=en_US.UTF-8 LC_CTYPE="en_US.UTF-8" LC_NUMERIC="en_US.UTF-8" LC_TIME="en_US.UTF-8" LC_COLLATE="en_US.UTF-8" LC_MONETARY="en_US.UTF-8" LC_MESSAGES="en_US.UTF-8" LC_PAPER="en_US.UTF-8" LC_NAME="en_US.UTF-8" LC_ADDRESS="en_US.UTF-8" LC_TELEPHONE="en_US.UTF-8" LC_MEASUREMENT="en_US.UTF-8" LC_IDENTIFICATION="en_US.UTF-8" LC_ALL=

budgierless commented 8 years ago 
**IV6-Config**

# Load additional ip6tables modules (nat helpers)
#   Default: -none-
# Space separated list of nat helpers (e.g. 'ip_nat_ftp ip_nat_irc'), which
# are loaded after the firewall rules are applied. Options for the helpers are
# stored in /etc/modprobe.conf.
IP6TABLES_MODULES=""

# Unload modules on restart and stop
#   Value: yes|no,  default: yes
# This option has to be 'yes' to get to a sane state for a firewall
# restart or stop. Only set to 'no' if there are problems unloading netfilter
# modules.
IP6TABLES_MODULES_UNLOAD="yes"

# Save current firewall rules on stop.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/ip6tables if firewall gets stopped
# (e.g. on system shutdown).
IP6TABLES_SAVE_ON_STOP="no"

# Save current firewall rules on restart.
#   Value: yes|no,  default: no
# Saves all firewall rules to /etc/sysconfig/ip6tables if firewall gets
# restarted.
IP6TABLES_SAVE_ON_RESTART="no"

# Save (and restore) rule and chain counter.
#   Value: yes|no,  default: no
# Save counters for rules and chains to /etc/sysconfig/ip6tables if
# 'service ip6tables save' is called or on stop or restart if SAVE_ON_STOP or
# SAVE_ON_RESTART is enabled.
IP6TABLES_SAVE_COUNTER="no"

# Numeric status output
#   Value: yes|no,  default: yes
# Print IP addresses and port numbers in numeric format in the status output.
IP6TABLES_STATUS_NUMERIC="yes"

# Verbose status output
#   Value: yes|no,  default: yes
# Print info about the number of packets and bytes plus the "input-" and
# "outputdevice" in the status output.
IP6TABLES_STATUS_VERBOSE="no"

# Status output with numbered lines
#   Value: yes|no,  default: yes
# Print a counter/number for every rule in the status output.
IP6TABLES_STATUS_LINENUMBERS="yes"

# Reload sysctl settings on start and restart
#   Default: -none-
# Space separated list of sysctl items which are to be reloaded on start.
# List items will be matched by fgrep.
#IP6TABLES_SYSCTL_LOAD_LIST=".ip_conntrack .bridge-nf"
tlhackque commented 8 years ago

Thanks for the configuration & locale. That rules out two possibilities.

I have spent several hours on this. (IPv4 only).

The problem, as I said earlier, is that iptables-restore is unable to delete an old ruleset because netfilter thinks it's still in use. However, iptables-restore accepts a command to delete the only user. So it can't be in use.

I have used the oldest available machine - which is not as old as yours.

I am using exactly your config file (which is in the data that you sent).

I am setting up iptables exactly as they are before the start command executes. (That's in the data too.)

I am using the output of the iptables command as captured on your system.

It turns out that I can not reproduce the problem unless I make a mistake in the process. I did, which is why I was optimistic earlier.

I can successfully put the data that BlockCountries is sending to iptables-restore into iptables-restore here. So BlockCountries is generating good data.

The only difference that I can see is that you are using iptables version 1.3.5, and the oldest version that I can build here is 1.4.15. I have 1.3.0, but that version does not support BlockCountries. I think the latest version is something like 1.6.0.

At this point, I think you should update your system.

You can try to just update iptables, but I think you would do better to update the entire OS.

If you really want to try to make iptables work:

But I do think that updating this machine - or moving its functions to another one - would be a much better use of your time.

budgierless commented 8 years ago

thanks for your feedback I will look into updating iptables to 1.4, and see how it goes from there.

budgierless commented 8 years ago

not had a good time, the link you posted should of worked for me but didn't as build worked be with restore area issues as the kernel effected the build, in the end i reinstalled stock version of iptables, which seem to fix issues with that error, as (start -update) had no issues in output, i think their may still be another with table-restore but its dose seem to be running OK, anyway just in-case their is better data i have added an updated debug file for you. check the link

tlhackque commented 8 years ago

It's not running OK. BlockCountries uses iptables-restore to load the rules that it generates.

"Rules update failed" means just that. BlockCountries is unable to update the firewall with the latest rules. Do not ignore the error.

You are still/again running iptables 1.3.5. It's clearly broken. I can not fix iptables. You have to run a version that works.

The start command will work the first time the rules are loaded, because there is no old ruleset to remove. Thereafter, it will fail. and you will run a stale ruleset. This is not a BlockCountries problem. It's iptables.

This is easy to see in the debug data.

Before BlockCountries starts, we have a set of BlockCountries rules installed. (as reported by iptables -n -L):

Chain INPUT (policy ACCEPT)
target     prot opt source               destination         
BLOCKCC0-I  all  --  0.0.0.0/0            0.0.0.0/0           

Chain BLOCKCC0-I (1 references)
target     prot opt source               destination         
RETURN     all  --  0.0.0.0/0            0.0.0.0/0           state RELATED,ESTABLISHED 
RETURN     tcp  --  0.0.0.0/0            0.0.0.0/0           multiport dports 25,587,465,53

This shows that the BLOCKCC0-I chain has (as expected) exactly one reference, from the INPUT chain.

BlockCountries generates a new set of rules, based on the latest data. It needs to install the new rules, and remove the old ones.

To do this, BlockCountries includes these commands (among thousands, but only these matter):

-D INPUT -j BLOCKCC0-I
-F BLOCKCC0-I
-X BLOCKCC0-I

These commands

The third command fails. That's the one at line 7074.

The only reasons for chain deletion to fail are a rule referencing it, or rules in the chain. But the preceding two commands ensure that neither should be true.

Thus, either iptables-restore is broken, or the netfilter service in the kernel is. They're part of the same package. iptables startup installs netfilter.

So netfilter/iptables is broken. You need to update. You need to update.

I can't spend any more time chasing a broken environment.

What I have done is add a check to bcinstall to verify that a recent version of iptables is installed. That should prevent anyone else from stumbling across this issue. The check is in V2.19.

The oldest version of iptables that I'm running here is 1.14.10.

As I've said before, an internet-facing machine should be running more current software. But that's your business.

budgierless commented 8 years ago

ok, I have upgraded the OS to the latest version that i must use production reasons (centOS 6.7), this uses iptables version 1.4.7 and it has installed with no issues at all, the only bad message i got was from bcinstall saying it wants version 1.14.10, but centOS will be supported uptil sept 2021 i think and its not even that old at all, so i think your script is being a bit unkind with the requirements, anyway if you have any questions please let me know.

tlhackque commented 8 years ago

I set the required version based on what I have here and knew works.

Based on your feedback, I'll update the requirement.

To make sure that the rest of the checks pass ,you can change the value of IPTV on line 9 of bcinstall to 1.4.7 in the meantime.

If you're not using IPv6, you can run bcinstall with -S to disable the IPv6 checks.

-v if you want to see the analysis. (and -h for all the options)

I'm glad that you've updated and that it solved your problems.