IPSec (XFRM) is not working on the CoreOS

uzytkownik commented 7 years ago

CoreOS Version

All I've tried (stable/alpha/beta).

Environment

DigitalOcean and Vagrant (virtualbox)

Expected Behavior

After the tunnel is established the encrypted packets get through.

Actual Behavior

__xfrm_policy_check seems to drop the packages (XfrmInTmplMismatch increases). After getting in touch with libreswan ML it looks like the problem is in CoreOS kernel. However because it is hard to debug the kernel on CoreOS (bug #1679) I haven't manage to find out what is wrong.

Reproduction Steps

Unpack ipsec.zip
Run script inside
Start ipsec service (systemctl start ipsec)
Generate key by logging into guest (machinectl login root@ipsec) and running ipsec newhostkey --output /etc/ipsec.d/<hostname>.secrets
Show the keys and create configuration file
Restart ipsec service to load the private key (and also changed configuration)
Start the tunnel

Additional information

I've managed to get the connection on Fedora using the same runtime/configuration. So it is almost certainly CoreOS kernel/configuration.

crawford commented 7 years ago

Sorry for the silence on this. Which version of Container Linux did this affect? Have you tried again with a newer version?

uzytkownik commented 7 years ago

Yes - I have been trying for some time now. I tried on all channels (stable/alpha/beta) at the time of submission and I just tried on 1235.12.0 (latest stable).

I just spotted it in dmesg - not sure if this is relevant and quick googleing did not reveal what it is about (I think it is benign):

[ 1800.914748] alg: No test for echainiv(authenc(hmac(sha1),cbc(aes))) (echainiv(authenc(hmac(sha1-avx),cbc-aes-aesni)))

dm0- commented 7 years ago

I've set up an IPSec tunnel between two containers on alpha and stable, and I am not seeing any errors.

Can you further explain your setup? In particular, what are your other end points? Did you address issues raised by ipsec verify? Can you just paste your configuration file from step 5 with sensitive data replaced?

This is what I ran to test: https://gist.github.com/dm0-/205bfb4a1b5144dc2e9615742b910d7e

uzytkownik commented 7 years ago

I haven't seen the XfrmInTmplMismatch in recent versions but I still see the packets not going through. I haven't tried to do it on single host though.

Regarding setup - I try to set up tunnel between two instances of coreos. I have fixed the issues from ipsec verify.

uzytkownik commented 7 years ago

I run the script and it worked for me. The config seems to be identical (other then the ordering):

conn core-01--core-02
        leftid=@core-01
        left=<core-01 ip>
        leftrsasigkey=<core-01 RSA>
        rightid=@core-02
        right=<core-02 ip>
        rightrsasigkey=<core-02 RSA>
        authby=rsasig
        auto=add

coreos / bugs