Auto CRL check

[y0no]

Hello, I try to revoke a client certificate using the commands shown in the documentation:

docker run --rm -i --volumes-from vpn -e "EASYRSA_BATCH=1" kylemanna/openvpn easyrsa revoke toto
docker run --rm -i --volumes-from vpn kylemanna/openvpn easyrsa gen-crl

The process seems to be ok, but when I try to connect to vpn with my revocated certificate, it works... The docker-openvpn logs:

Fri Jan 29 23:17:04 2016 172.17.0.1:41187 CRL CHECK OK: CN=blah
Fri Jan 29 23:17:04 2016 172.17.0.1:41187 VERIFY OK: depth=1, CN=blah
Fri Jan 29 23:17:04 2016 172.17.0.1:41187 CRL CHECK OK: CN=toto
Fri Jan 29 23:17:04 2016 172.17.0.1:41187 VERIFY OK: depth=0, CN=toto

Does this reaction normal ?

[kylemanna]

Did you restart the server after you generated the CRL? The ovpn_run script looks for the presence of the CRL and passes the path to openvpn if found. If you created the CRL and never restarted the container, the argument was never passed.

Can you run a ps and see if the crl-verify arugment is passed? If it isn't can you restart your container and confirm that it is?

[1] https://github.com/kylemanna/docker-openvpn/blob/e7d0d4e/bin/ovpn_run#L42-L49

[y0no]

If I restart the server after the CRL generation it work correctly. But in my VPN server creation script. I have done something like that:

docker run --name $VOL_NAME -v $CONFIG:/etc/openvpn busybox
docker run --volumes-from $VOL_NAME --rm synvpn ovpn_genconfig -u udp://vpn01.synhack.fr:$PORT -t
docker run --volumes-from $VOL_NAME --rm -i -e "EASYRSA_REQ_CN=synhack" -e "EASYRSA_BATCH=1" synvpn ovpn_initpki nopass <<< $'\n'
docker run --volumes-from $VOL_NAME --rm kylemanna/openvpn easyrsa gen-crl
docker run --name $CTR_NAME --volumes-from $VOL_NAME -d -p $PORT:1194/udp --cap-add=NET_ADMIN synvpn

I don't understand why it don't automatically stop to accept connection from revoked certificate after revoke command has been fired without restarting the server.

[y0no]

The problem seems to come from the hardlink between /etc/openvpn/crl.pem and /etc/openvpn/pki/crl.pem. When I check the revoked certificates with "openssl crl" command. I have revoked certificates with /etc/openvpn/pki/crl.pem but not with /etc/openvpn/crl.pem.

Did you know what can cause this issue ?

[kylemanna]

That's interesting. That would explain the problem you have. I apparently didn't do enough testing with new revocations.

Sounds like the file is being replaced by openssl breaking the hardlink? Could you verify this by running ls -li to see the inode number? I bet initially they are the same, but when you revoke another certificate they are different and hence not updated.

The solution may be a wrapper for the easyrsa revocation generation to copy the file to the correct location with the correct permissions instead of using the hardlink.

[y0no]

Effectively the inode number seems to change. Currently my script copy the file in /etc/openvpn after each revocation, But you are right, it will be better to modify easyrsa to do that.

[fabn]

@kylemanna I'm encountering the same issue. Any fix for this?

Why are you using ln in this line? Wouldn't be much simpler to change line 49 to something like

ARGS+=("--crl-verify" "$OPENVPN/pki/crl.pem")

Is there any drawback in doing that?

Also according to this thread client documentation is outdated.

[fabn]

Why are you using ln in this line

Ok, I think this is needed because /etc/openvpn/pki/ has 600 permissions and openvpn daemon drops root privileges before reading that file. So a solution might be to replace ln with cp, this however will require users to restart openvpn image for each crl change.

Thoughts?

[kylemanna]

Yes, this is a problem. Could try a hardlink to that file that the openvpn user can read?

[johnreutersward]

I'm also having trouble with revoked client certificates even after restarting the container.

After running revoke and gen-crl I see that the certificate is listed as REVOKED when running ovpn_listclients.

However, it seems that restarting the docker container does not seem to have any effect. I am still able to connect with a revoked certificate.

[kylemanna]

@rojters do you see --crl-verify /etc/openvpn/crl.pem passed to the openvpn binary using something ps waux?

[johnreutersward]

@kylemanna Yes, that appears to be the case:

Running docker exec CONTAINER ps waux shows the following output:

PID   USER     TIME   COMMAND
    1 nobody     0:03 openvpn --config /etc/openvpn/openvpn.conf --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl.pem

[kylemanna]

Hmm. This is something that should be unit tested so that it doesn't happen.

Can someone write up the exact steps to reproduce on the lastest docker image and I'll take a look at it.

[vivekprahlad]

The workaround for this issue seems to be to copy /etc/openvpn/pki/crl.pem to /etc/openvpn/crl.pem as @y0no had mentioned. My revocation script currently copies the file, and then restarts the openvpn service.

[kylemanna]

I feel like something is broken here, but haven't had time to dig in. Would be useful if someone could propose a test that could be added to the Travis-CI tests to detect this issue.

[jpfluger]

This command below will reset the hard-link without need to restart the container.

# pre-inode numbers
sudo docker run -it --rm -v $PWD/openvpn-data/conf:/etc/openvpn kylemanna/openvpn /bin/bash -c "ls -li /etc/openvpn/pki/crl.pem /etc/openvpn/crl.pem"
2752741 -rw-r--r--    1 root     root           690 Mar  9 22:09 /etc/openvpn/crl.pem
2752733 -rw-------    1 root     root           690 Mar  9 22:17 /etc/openvpn/pki/crl.pem

# reset hardlink
sudo docker run -it --rm -v $PWD/openvpn-data/conf:/etc/openvpn kylemanna/openvpn /bin/bash -c "ln -v -f /etc/openvpn/pki/crl.pem /etc/openvpn/crl.pem && chmod 644 /etc/openvpn/crl.pem"

# post-inode numbers
sudo docker run -it --rm -v $PWD/openvpn-data/conf:/etc/openvpn kylemanna/openvpn /bin/bash -c "ls -li /etc/openvpn/pki/crl.pem /etc/openvpn/crl.pem"
2752733 -rw-r--r--    2 root     root           690 Mar  9 22:17 /etc/openvpn/crl.pem
2752733 -rw-r--r--    2 root     root           690 Mar  9 22:17 /etc/openvpn/pki/crl.pem

Re. steps to reproduce the initial state where inodes are different:

1. New install (pre-running of container): new ovpn_genconfig, new clients, etc... If you run easyrsa gen-crl now, it will write /etc/openvpn/pki/crl.pem. This file is required for the arg "--crl-verify /etc/openvpn/crl.pem to be added in ovpn_run. If there is no crl file, then openvpn launches without revocations enabled. Then later when a client is revoked, openvpn must restart to create the hardlink from /etc/openvpn/crl.pem to the new /etc/openvpn/pki/crl.pem.
2. Revoke a client (verify inodes are different by running the pre-inode numbers command above)
3. openvpn is restarted but since inodes are different, openvpn does not recognize crl.pem
4. Run the reset hardlink command above and verify with the post-inode numbers command
5. The client revoked in step 2 is now denied authorization

It's really a nasty little bug. I tried a number of ways around this, even "touching" the pki/crl.pem file if not exists. But openvpn expects a pem with x509 content and not an empty file.

[jpfluger]

For my use case, I see a two-part fix but is this s/thing others would be okay with?

Part 1: Whenever ovpn_run runs, if /etc/openvpn/pki/crl.pem is not present, then generate it using easyrsa gen-crl. Then proceed with the hardlink and chmod settings.

Part 2: Whenever a client is revoked, do it through new script ovpn_revokeclient. This script would:

1. Revoke via easyrsa revoke $CLIENTNAME
2. Regenerate via easyrsa gen-crl
3. Invoke the hardlink fix command above

[jpfluger]

Actually, Part 1 doesn't work b/c it requires a password to generate but Part 2 can be done interactively.

[jpfluger]

For Part 2, here's a link to my version of ovpn_revokeclient. This works for my use case.

[kylemanna]

Hey guys, please give this a try with #251 merged. Hopefully restarting the server now will do what is expected.

[kylemanna]

For those interested in OpenVPN automatically re-loading the CRL I think it's a matter of switching back to a hardlink and updating easryrsa gen-crl to re-write the file instead of move a new file over the top to not break the hardlink. If the hardlink remains, the permissions should remain, and OpenVPN will potentially reload it. Haven't looked at the OpenVPN code yet to determine if this is true.

Description of hardlink breakage:

https://github.com/OpenVPN/easy-rsa/blob/5a429d22c78604c95813b457a8bea565a39793fa/easyrsa3/easyrsa#L742-L758

The problem is that EasyRSA:

1. creates a temporary file
2. writes the CRL to that file
3. moves the CRL over the top of the previous one (breaking hardlink)

Presumably it does this to make it an atomic operation to avoid corruption in a power failure (my understanding is that mv aka rename() syscall is typically atomic on journaling filesystems) or system crash at the exact same time.

What we could do instead is rewrite to the same file, that way the hardlink I originally used would work as expected and the permission would be preserved while the file update and OpenVPN would be able to check the connecting client against the updated CRL.