Closed Dude4Linux closed 6 years ago
Thanks @Dude4Linux
I believe the correct behavior would be to default to the internal
apt-cacher-ng
athttp://192.168.121.1:3142
.
Considering that apt-cacher-ng
is pre-installed in the LXC host appliance, it makes sense that it should be the apt-cache host too! Please excuse my ignorance, but I assume that http://192.168.121.1
will always resolve to the LXC host? If so, that sounds like a fine plan to me.
One thing that will need testing though is that the additional security measures we've taken with third party apt repos (see #1033) don't break the apt cache (I don't think it should, but not 100% sure).
While investigating the issue of caching apt requests, the question was raised about the value of using Polipo to cache requests for
wget
,curl
, andgit
.
AFAIK polipo dev is essentially dead as the value of a plain http (only) caching tool is questionable in this day and age of (nearly) everything being https.
I wouldn't be against it being implemented, but I'm not convinced that it would really provide much value.
Alternatives that occur to me are:
If we go with option 2, we should think really hard about ensuring that the user(s - of both LXC host and guest) is (are) aware of the security/privacy implications.
FWIW Debian Stretch has Squid v3.5.23. Here's a few links that look relevant: https://www.ssltrust.com.au/blog/setup-squid-proxy/ https://wiki.squid-cache.org/Features/HTTPS https://wiki.squid-cache.org/Features/SslPeekAndSplice
Reality has a way of being more complicated than anticipated. After spending several frustrating hours trying to get curl
and wget
to work seamlessly with Polipo, I decided that the potential benefits weren't worth the added complexity. I previously tried to use a Squid proxy on my portable development environment (PDE) with little success. For now, I suggest we just focus on getting apt-caching to work.
Next I discovered that the simple one line fix only works for NAT'ed containers; getting bridged containers to work will require some additional work. I'm now looking at what changes are needed in iptables to allow connections from the br0
interface to the apt-cacher-ng server which is bound to the natbr0
interface. An alternate approach would be to bind apt-cacher-ng
to both interfaces, but that gets ugly since the br0
ip address is set by dhcp which means it can change over time. The only practical way is to use the default setting, which binds to all addresses. I think that might provide security concerns in some situations, i.e. AWS. It also adds the complication of configuring different proxy addresses for bridged and nat'ed containers.
[...] For now, I suggest we just focus on getting apt-caching to work.
:+1:
Re apt cache stuff, it sounds like you have a good understanding of the issues. Even if we only provided a minimalist MVP implementation that only supported NAT'ed containers, that would be acceptable at this point IMO. We could always as it as a feature request for v15.1 if need be. Having said that, while you are happy to continue working on it, please be my guest! :smile:
Priorities for using apt-cacher-ng
in LXC appliance.
1) Maintain backward compatibility
As much as possible, the proxy should behave as in previous versions when specified on the command line, eg -x --apt-proxy
2) Allow pre-seeding APT_PROXY in inithooks.conf
3) Use internal apt-cacher-ng by default if not specified in inithooks.conf
or on CLI
.
4) Maintain security by allowing only containers to access internal server.
5) Specify proxy for upsteam proxy/cache
6) Work in nested containers
7) Dynamically configure apt-cacher-ng
8) Fail gracefully
If the user supplied proxy, or the dynamically selected proxy is not available or not functioning, resort to direct downloading
At first I thought that this would be a trivial matter of just setting the default value and allowing the user to override it on the command line. I soon discovered, however, that while this worked fine for nat'ed containers in the 192.168.121.0/24
address space, the bridged (br0
) containers could access the proxy server on the BindAddress: 192.168.121.1
. I tried adding every iptables rule I could think of to allow the containers to connect, but finally I had to admit defeat and look for an alternate approach. I tried binding apt-cacher-ng
to both natbr0
and br0
. That worked, but required different addresses be used depending on the type of container. I decided that since I would have to deal with potentially changing IP addresses (br0
gets it's IP from DHCP) I would just focus on binding to the br0
interface. Happily both bridged and nat'ed containers can access the br0
address.
I ended up needing to add three separate scripts:
1) firstboot.d/10proxy
grabs the inithooks.conf
during firstboot and if it defines $APT_PROXY
, it takes the value and adds it permanently to the root environment using Alon's trick for handling $FAB_APT_PROXY
, by adding it to /root/.bashrc.d/proxy
.
2) /etc/network/if-up.d/apt-cacher-ng
handles the task of detecting br0's IP address
and configuring the host server accordingly.
3) /etc/network/if-up.d/apt-proxy
handles the client (container) by fetching the proxy address from the environment and setting /etc/apt/apt.conf.d/01proxy
accordingly.
@JedMeister - I've posted my work in progress over at https://github.com/Dude4Linux/lxc/tree/add-inithooks-proxy. Haven't issued a PR yet because I'm still working on a few items. When it's ready, should I bump the version to 15.0rc2
?
@Dude4Linux
That all sounds good to me.
TL;DR - you can bump the version to 15.0
(i.e. no rc).
I probably should have caught the changelog version earlier when I did the merge. There should only be separate changelog entries for releases. So seeing as this hasn't been released as an RC (nor will it most likely) any intermediate versions are for your purposes only.
Also, as you're not using buildtasks, then it probably doesn't matter, but if you were, then the format you're using (15.0-rc1
) isn't valid (apologies if you picked that up from me, I inadvertently used that incorrect format for both tkldev and core initially). The correct format is 15.0rc1
(i.e. no dash).
@JedMeister @OnGle @qrntz - After testing the proposed changes to the LXC appliance to improve the handling of APT_PROXY and fix several issues I noted along the way, it seems to be working fairly smoothly. The change of binding apt-cacher-ng
from natbr0
to br0
allows both types of containers access to apt-cacher-ng
. One side effect of this change is that access to the Apt-Cacher NG maintenance page is now exposed to the world. This is both a benefit (page provides useful statics and ability to initiate maintenance actions) and a liability (possible security vulnerability e.g. CVE-2014-4510).
I have tried to lock down access to apt-cacher-ng
server port 3142 by using /etc/hosts.{allow|deny}
to restrict access to the two CIDR networks attached to natbr0
and br0
respectively, but this still leaves open access from external hosts that share the same CIDR as the bridged br0
containers. Next, I would like to restrict access using iptables
so that only the containers in that CIDR are allowed access to port 3142. So here is where a could use a few extra eyeballs to help me see what, if anything, I'm doing wrong.
Let's start with the current code in /etc/network/if-up.d/iptables-lxc
.
1:#!/bin/bash
2:
3:iptables_add() {
4: args=$@
5: if ! iptables -C $args >/dev/null 2>&1; then iptables -A $args; fi
6:}
7:
8:#Creating default policies
9:iptables -P INPUT DROP
10:iptables -P OUTPUT ACCEPT
11:iptables -P FORWARD DROP # we're not a router
12:
13:if [ "$IFACE" = "lo" ]; then
14: # Allow traffic on loopback
15: iptables_add INPUT -i lo -j ACCEPT
16: iptables_add OUTPUT -o lo -j ACCEPT
17: # Allow established connections to continue
18: iptables_add INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
19: iptables_add OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
20: iptables_add FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
21:fi
22:
23:if [ "$IFACE" = "br0" ]; then
24: # Allow inbound connections
25: iptables_add INPUT -i br0 -j ACCEPT
26: iptables_add INPUT -p tcp --dport 22 -j ACCEPT #SSH
27: iptables_add INPUT -p tcp --dport 53 -j ACCEPT #DNS
28: iptables_add INPUT -p tcp --dport 80 -j ACCEPT #HTTP
29: iptables_add INPUT -p tcp --dport 443 -j ACCEPT #HTTPS
30: iptables_add INPUT -p UDP --dport 67:68 -j ACCEPT #DHCP
31: iptables_add INPUT -p udp --dport 53 -j ACCEPT #DNS
32: iptables_add INPUT -p tcp --dport 12320 -j ACCEPT #HTTPS
33: iptables_add INPUT -p tcp --dport 12321 -j ACCEPT #HTTPS
34: iptables_add INPUT -p tcp --dport 12322 -j ACCEPT #HTTPS
35: iptables_add FORWARD -i br0 -j ACCEPT
36: iptables_add POSTROUTING -t nat -o br0 -j MASQUERADE
37:fi
38:
39:if [ "$IFACE" = "natbr0" ]; then
40: CIDR=192.168.121.0/24
41: iptables_add INPUT -s $CIDR -i natbr0 -j ACCEPT
42: iptables_add OUTPUT -d $CIDR -o natbr0 -j ACCEPT
43: iptables_add FORWARD -i natbr0 -s $CIDR -j ACCEPT
44: iptables_add POSTROUTING -t nat -s $CIDR ! -o natbr0 -j MASQUERADE
45: iptables_add POSTROUTING -t mangle -p udp --dport bootpc -s $CIDR -j CHECKSUM --checksum-fill
46:fi
My first question is about the purpose of line 36. I cannot think of a reason why MASQUERADE would be used on the bridged br0
interface. I think maybe this is an artifact from editing that was left behind by mistake. I have experimented with removing it and saw no bad side-effects. If anyone knows of a reason why it should be kept, please let me know.
The second question is about lines 26-34. These rules allow access to the well-known ports used by all appliances. I'm wondering why they are included in the IFACE = br0
section when none of them are specific to br0
. It would make more sense to me to include them in the IFACE = lo
section.
After making these changes and adding extra rules for port 3142, I have the following:
1:#!/bin/bash
2:
3:iptables_add() {
4: args=$@
5: if ! iptables -C $args >/dev/null 2>&1; then iptables -A $args; fi
6:}
7:
8:#Creating default policies
9:iptables -P INPUT DROP
10:iptables -P OUTPUT ACCEPT
11:iptables -P FORWARD DROP # we're not a router
12:
13:if [ "$IFACE" = "lo" ]; then
14: # Allow traffic on loopback
15: iptables_add INPUT -i lo -j ACCEPT
16: iptables_add OUTPUT -o lo -j ACCEPT
17: # Allow connections to apt-cacher-ng from containers, reject others
18: iptables_add INPUT -i br0 -p tcp --dport 3142 -j ACCEPT
19: iptables_add INPUT -i natbr0 -p tcp --dport 3142 -j ACCEPT
20: iptables_add INPUT -p tcp --dport 3142 -j REJECT --reject-with tcp-reset
21: # Allow connections to well known ports
22: iptables_add INPUT -p tcp --dport 22 -j ACCEPT #SSH
23: iptables_add INPUT -p tcp --dport 53 -j ACCEPT #DNS
24: iptables_add INPUT -p tcp --dport 80 -j ACCEPT #HTTP
25: iptables_add INPUT -p tcp --dport 443 -j ACCEPT #HTTPS
26: iptables_add INPUT -p udp --dport 67:68 -j ACCEPT #DHCP
27: iptables_add INPUT -p udp --dport 53 -j ACCEPT #DNS
28: iptables_add INPUT -p tcp --dport 12320 -j ACCEPT #HTTPS
29: iptables_add INPUT -p tcp --dport 12321 -j ACCEPT #HTTPS
30: iptables_add INPUT -p tcp --dport 12322 -j ACCEPT #HTTPS
31: # Allow established connections to continue
32: iptables_add INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
33: iptables_add OUTPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
34: iptables_add FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
35:fi
36:
37:if [ "$IFACE" = "br0" ]; then
38: # Allow inbound connections
39: iptables_add INPUT -i br0 -j ACCEPT
40: iptables_add FORWARD -i br0 -j ACCEPT
41:# iptables_add POSTROUTING -t nat -o br0 -j MASQUERADE
42:fi
43:
44:if [ "$IFACE" = "natbr0" ]; then
45: CIDR=192.168.121.0/24
46: iptables_add INPUT -s $CIDR -i natbr0 -j ACCEPT
47: iptables_add OUTPUT -d $CIDR -o natbr0 -j ACCEPT
48: iptables_add FORWARD -i natbr0 -s $CIDR -j ACCEPT
49: iptables_add POSTROUTING -t nat -s $CIDR ! -o natbr0 -j MASQUERADE
50: iptables_add POSTROUTING -t mangle -p udp --dport bootpc -s $CIDR -j CHECKSUM --checksum-fill
51:fi
The port 3142 rules, lines 17-20 were placed early in the list so that the REJECT rule would take precedence over later ACCEPT rules. Regardless of their position though, the rules don't seem to be having the desired effect, i.e. blocking access from other hosts on the same CIDR as br0
. If anyone can see a reason why these rules aren't having the desired effect, please let me know. TIA
Tomorrow, I'll post the results of a test run of all 100 appliances in containers.
I shortened the test to twelve appliances to save time. The IPtables verbose listings before and after the test run were:
IPtables verbose listing immediately after restarting host VM
root@lxc ~# iptables -L -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
29 4273 f2b-sshd tcp -- any any anywhere anywhere multiport dports ssh
32 2512 ACCEPT all -- lo any anywhere anywhere
107 10452 ACCEPT tcp -- br0 any anywhere anywhere tcp dpt:3142
0 0 ACCEPT tcp -- natbr0 any anywhere anywhere tcp dpt:3142
0 0 REJECT tcp -- any any anywhere anywhere tcp dpt:3142 reject-with tcp-reset
0 0 DROP tcp -- any any anywhere anywhere tcp dpt:3142
29 4273 ACCEPT tcp -- any any anywhere anywhere tcp dpt:ssh
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:domain
16 832 ACCEPT tcp -- any any anywhere anywhere tcp dpt:http
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:https
3 984 ACCEPT udp -- any any anywhere anywhere udp dpts:bootps:bootpc
0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:domain
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12320
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12321
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12322
211 22617 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- br0 any anywhere anywhere
0 0 ACCEPT all -- natbr0 any 192.168.121.0/24 anywhere
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- br0 any anywhere anywhere
0 0 ACCEPT all -- natbr0 any 192.168.121.0/24 anywhere
Chain OUTPUT (policy ACCEPT 8 packets, 480 bytes)
pkts bytes target prot opt in out source destination
32 2512 ACCEPT all -- any lo anywhere anywhere
347 108K ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- any natbr0 anywhere 192.168.121.0/24
Chain f2b-sshd (1 references)
pkts bytes target prot opt in out source destination
29 4273 RETURN all -- any any anywhere anywhere
root@lxc ~# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 237 packets, 38231 bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 36 packets, 2700 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 35 packets, 2412 bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 35 packets, 2412 bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any !natbr0 192.168.121.0/24 anywhere
IPtables verbose listing after testing 12 bridged br0
appliances
root@lxc ~# iptables -L -v
Chain INPUT (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
16726 18M f2b-sshd tcp -- any any anywhere anywhere multiport dports ssh
1352 135K ACCEPT all -- lo any anywhere anywhere
11531 788K ACCEPT tcp -- br0 any anywhere anywhere tcp dpt:3142
0 0 ACCEPT tcp -- natbr0 any anywhere anywhere tcp dpt:3142
0 0 REJECT tcp -- any any anywhere anywhere tcp dpt:3142 reject-with tcp-reset
0 0 DROP tcp -- any any anywhere anywhere tcp dpt:3142
16726 18M ACCEPT tcp -- any any anywhere anywhere tcp dpt:ssh
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:domain
16 832 ACCEPT tcp -- any any anywhere anywhere tcp dpt:http
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:https
54 18959 ACCEPT udp -- any any anywhere anywhere udp dpts:bootps:bootpc
0 0 ACCEPT udp -- any any anywhere anywhere udp dpt:domain
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12320
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12321
0 0 ACCEPT tcp -- any any anywhere anywhere tcp dpt:12322
665K 4586M ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
22 3005 ACCEPT all -- br0 any anywhere anywhere
0 0 ACCEPT all -- natbr0 any 192.168.121.0/24 anywhere
Chain FORWARD (policy DROP 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- br0 any anywhere anywhere
0 0 ACCEPT all -- natbr0 any 192.168.121.0/24 anywhere
Chain OUTPUT (policy ACCEPT 1244 packets, 83537 bytes)
pkts bytes target prot opt in out source destination
1352 135K ACCEPT all -- any lo anywhere anywhere
914K 651M ACCEPT all -- any any anywhere anywhere ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- any natbr0 anywhere 192.168.121.0/24
Chain f2b-sshd (1 references)
pkts bytes target prot opt in out source destination
16726 18M RETURN all -- any any anywhere anywhere
root@lxc ~# iptables -t nat -L -v
Chain PREROUTING (policy ACCEPT 2303 packets, 371K bytes)
pkts bytes target prot opt in out source destination
Chain INPUT (policy ACCEPT 213 packets, 23506 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 1626 packets, 113K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 1626 packets, 113K bytes)
pkts bytes target prot opt in out source destination
0 0 MASQUERADE all -- any !natbr0 192.168.121.0/24 anywhere
Unfortunately, I don't know anywhere near enough about iptables as I probably should... I suspect that @qrntz is the man for this job! :smile:
After many failed attempts to limit access to apt-cacher-ng
using iptables
, I finally thought of using ebtables
. Found it only requires one line to block requests from external hosts. Gotta use the right tool for the job :). Of course I had to add ebtables
to plan/main
.
Along the way, I completely revised iptables-lxc
, eliminating several unneeded rules, changed the order of some rules to achieve the desired outcome, and then added a few new rules to enhance security. The new version looks like this:
#!/bin/bash
# append iptables rule at end of table, unless it exists
iptables_add() {
args=$@
if ! iptables -C $args >/dev/null 2>&1; then iptables -A $args; fi
}
# insert iptables rule at top of table, unless it exists
iptables_insert() {
args=$@
if ! iptables -C $args >/dev/null 2>&1; then iptables -I $args; fi
}
cidr() {
ip route list scope link dev $1 | sed -rn '{ s/^(\S+).*/\1/p }'
return
}
#Creating default policies
iptables -P INPUT DROP
iptables -P OUTPUT ACCEPT
iptables -P FORWARD DROP # we're not a router
if [ "$IFACE" = "lo" ]; then
# Drop invalid packets
iptables_add INPUT -m state --state INVALID -j DROP
# Allow traffic on loopback
iptables_add INPUT -i lo -j ACCEPT
# Allow established connections to continue
iptables_add INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
iptables_add FORWARD -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
fi
if [ "$IFACE" = "eth0" ]; then
# Use ebtables to block external traffic to apt-cacher-ng
ebtables -t filter -A INPUT -p IPv4 -i eth0 --ip-proto tcp --ip-dport 3142 -j DROP
fi
if [ "$IFACE" = "br0" ]; then
CIDR=$(cidr br0)
# Accept all packets from br0 containers
iptables_add INPUT -s $CIDR -i br0 -m conntrack --ctstate NEW -j ACCEPT
iptables_add FORWARD -s $CIDR -i br0 -m conntrack --ctstate NEW -j ACCEPT
fi
if [ "$IFACE" = "natbr0" ]; then
CIDR=$(cidr natbr0)
# Drop external connections to natbr0 containers
iptables_insert FORWARD -d 192.168.121.0/24 -p tcp -m conntrack --ctstate NEW -j DROP
# Drop external pings to natbr0 containers
iptables_insert FORWARD -d 192.168.121.0/24 -p icmp -m icmp --icmp-type 8 -j DROP
# Accept all packets from natbr0 containers
iptables_add INPUT -s $CIDR -i natbr0 -m conntrack --ctstate NEW -j ACCEPT
iptables_add FORWARD -s $CIDR -i natbr0 -m conntrack --ctstate NEW -j ACCEPT
# Accept broadcasts from DHCP
iptables_add INPUT -i natbr0 -p udp -m udp --sport 68 --dport 67 -j ACCEPT #DHCP
# Masquerade natbr0 containers
iptables_add POSTROUTING -t nat -s $CIDR ! -d $CIDR -j MASQUERADE
# Correct checksums for bootpc
iptables_add POSTROUTING -t mangle -o natbr0 -p udp --dport bootpc -j CHECKSUM --checksum-fill
fi
Great work @Dude4Linux - from a quick glance it looks good.
FYI, The first batch of v15.0 appliances (~30) are built to ISO (almost) ready for publishing. I'm in the final QA testing stage and hope to finalise them this week, ready to publish next week (I had hoped to get it done this week, but fast running out of week).
The LXC host appliance will be one of the last v15.0 appliances to ship so we can do some solid "real world" testing with the v15.0 templates. FWIW Ansible won't be in this first batch, but I hope to add it to the next batch.
This looks closed to me, feel free to reopen if necessary.
The LXC turnkey template,
lxc-turnkey
, accepts a command line option for--aptproxy
. If the option is specified, it will properly configure the container to use an external apt proxy (cached), but otherwise it does nothing. I believe the correct behavior would be to default to the internalapt-cacher-ng
athttp://192.168.121.1:3142
. This should be a simple one line fix.While investigating the issue of caching apt requests, the question was raised about the value of using Polipo to cache requests for
wget
,curl
, andgit
.This discussion is being moved here from https://github.com/turnkeylinux-apps/lxc/pull/14 which is now closed.