nondeterministic behaviour of nixos containers

nek0 commented 5 years ago

Issue description

when using multiple nixos-containers with private networks and forwarded ports from host to container on the same host, the ports are not forwarded deterministically to the container, as they start simultaneously and cause some kind of race condition in iptables. I've put together a small example as nixops configuration file which can easily be run on any 18.09-stable nixos host.

Steps to reproduce

download configuration file as conf.nix
nixops create -d determinism conf.nix
nixops deploy -d determinism
wait for it
nixops ssh -d determinism server
iptables -t nat -S
You should see ports 8001 8002 and 8003 forwarded to three different nixos containers, but you usually don't.
??????
non-profit

Technical details

system: "x86_64-linux"
host os: Linux 4.19.16, NixOS, 19.03pre166887.db8e3654e2c (Koi)
multi-user?: yes
sandbox: yes
version: nix-env (Nix) 2.2
channels(root): "nixos-18.09.1922.97e0d53d669, nixos-unstable-19.03pre166449.be445a9074f, nixpkgs-19.03pre166887.db8e3654e2c"
channels(nek0): ""
nixpkgs: /nix/var/nix/profiles/per-user/root/channels/nixos

stale[bot] commented 4 years ago

Thank you for your contributions.

This has been automatically marked as stale because it has had no activity for 180 days.

If this is still important to you, we ask that you leave a comment below. Your comment can be as simple as "still important to me". This lets people see that at least one person still cares about this. Someone will have to do this at most twice a year if there is no other activity.

Here are suggestions that might help resolve this more quickly:

Search for maintainers and people that previously touched the related code and @ mention them in a comment.
Ask on the NixOS Discourse.
Ask on the #nixos channel on irc.freenode.net.

nek0 commented 3 years ago

I would like to point out, that this issue still persists. when mutliple containers get started at the same time, iptables fails to create all port forwardings neccessary.

Mic92 commented 3 years ago

We currently start systemd-nspawn directly for declarative containers. Systemd-nspawn talks with the kernel directly to issue iptables rules. I think there is a race condition when multiple systemd-nspawn container are run in parallel. One solution might be to use systemd-machined or create all containers in serial instead parallel.

Mic92 commented 3 years ago

Also see https://github.com/systemd/systemd/blob/edf370af9e9fafad01393699e7a6f34bf0568dd6/src/nspawn/nspawn-expose-ports.c#L100

stale[bot] commented 3 years ago

I marked this as stale due to inactivity. → More info

Mic92 commented 3 years ago

I don't this have been solved.

stale[bot] commented 2 years ago

I marked this as stale due to inactivity. → More info

nek0 commented 1 year ago

Has this been solved yet?

NixOS / nixpkgs