Isolate does not configure loopback in the sandbox on my system

[jiridanek]

When I use --share-net, I do get a working loopback, as well as all the other network interfaces in my sandbox

/home/jdanek/bin/usr/local/bin/isolate --share-net --dir=/nix/store=/nix/store --run -- /nix/store/jangrpp1pfgr0kn7bqbl05minrj0jr84-iproute2-5.12.0/bin/ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
[...]

When I start isolate without it, however, my loopback is reported as being DOWN

# /home/jdanek/bin/usr/local/bin/isolate --dir=/nix/store=/nix/store --run -- /nix/store/jangrpp1pfgr0kn7bqbl05minrj0jr84-iproute2-5.12.0/bin/ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
OK (0.001 sec real, 0.001 sec wall)

I wish to use isolate to sandbox a process which spawns two subprocesses, one serving on localhost, and the other one connecting to the first as a client. When the loopback is down, I cannot do this.

Is this something that is supposed to work? If so, how can I debug what has gone wrong?

[jiridanek]

I can reproduce the problem with unshare as well, but there I am able to subsequently resolve it.

% unshare --user --net -- ip a
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

% unshare --user --net -- bash -c 'ifconfig lo up; ip a'
ifconfig: SIOCSIFFLAGS failed: Operation not permitted
1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

I can fix this when I add parameter --map-root-user. This permits me to run ifconfig lo up succesfully. I am unable to bring lo up under isolate.

% unshare --user --map-root-user --net -- bash -c 'ifconfig lo up; ip a'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever

Preferably, I'd like to have lo up already to begin with. The manpage of isolate suggests that is the expected behavior.

[jiridanek]

I've checked the paper from the original authors (@gollux) and it turned out that the current behavior is the intended one:

a networking namespace: this determines the set of networking devices available to the compartment, which for isolated processes should be none. Although isolated processes can still create IP sockets, they cannot use them as there are no network devices (not even the loopback interface).

See the attached patch which enables the loopback. I can integrate it better if you agree it is a welcome addition to the project. Would you want to put it under a flag? lo.patch.txt

[krulis-martin]

I tried your patch on our clone of isolate, but it is not working. It brings the lo device UP, but when an application attempts to use it, it gets socket error "Operation not permitted".

[gollux]

The patch brings the loopback interface up, but it does not configure it with an IP address. So it is currently of very limited use. Also, it defies almost all coding conventions of Isolate, so I am not willing to merge it in the current form.

Let's talk about use cases first: in which situations is a loopback interface inside the sandbox needed?

[krulis-martin]

According to ip addr, the address is properly set. However, when trying ping 127.0.0.1, for instance, it fails (operation not permitted). We need the loopback to test networking assignments (so that students can develop solutions on a real network and just by changing IP address in configuration, they will work in isolate as well).

[krulis-martin]

I have investigated further and I found the problem was in ping itself. I did not realize that ICMP socket would require special privileges (the ping tool uses suid to allow regular users to use it, but it was not working in the sandbox). When I used netcat to create a TCP connection via loopback, it seems to be working.

Although, I would humbly recommend adding this line to the patch: https://github.com/ReCodEx/isolate/blob/master/isolate.c#L737 (load the flags before setting them, so only the IFF_UP flag will change).

[jiridanek]

Hi, thanks for looking into this. I feel like I've forgot everything I've studied when I was originally dealing with this. That ping needs capabilities to actually do the pinging... obvious, but I did not think of it when reading about your problem.

I the meantime, I've cobbled together something using either Podman (but that was slow to start-up) or systemd and bwrap, which was fast enough. The problem for me was startup. Starting lots of short-lived podman containers is (for me surprisingly) resource intensive.

p = await asyncio.create_subprocess_exec(
        'podman', 'run', '--quiet', f'-v{box.name}:/box', '--network=none', '--rm', image, *command,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE
    )

async def execute_bwrap(box: tempfile.TemporaryDirectory, item: EvalRequest) -> Tuple[bytes, bytes, int]:
    """This will not kill grandchildren, --die-with-parent only handles direct child

    edit: solved with --unshare-pid"""
    alpine = '/home/evaluator/alpine'  # contains unpacked alpine docker image
    command = ['timeout', '20', 'python', '-B', f'{item.mode}.py']

    p = await asyncio.create_subprocess_exec(
        'systemd-run',
        '--user',
        '--quiet',
        '--scope',
        '-p', 'MemoryAccounting=yes', '-p', 'MemoryLimit=100M',
        '-p', 'CPUAccounting=yes', '-p', 'CPUQuota=20%',
        '-p', 'TasksAccounting=yes', '-p', 'TasksMax=10',

        '--',

        'bwrap',
        '--ro-bind', alpine, '/',
        '--tmpfs', '/tmp',
        '--bind', box.name, '/box',
        '--unshare-net',
        '--unshare-pid',
        '--chdir', '/box',
        '--die-with-parent',
        '--setenv', 'PATH', '/usr/bin:/usr/local/bin',

        '--',

        *command,
        stdout=subprocess.PIPE,
        stderr=subprocess.PIPE
    )

I completely realize that the proposed patch is not a good quality code (I mostly copy pasted it from the url mentioned in comment). On on 19th April we'll do planning for some another programming contest. If we decide that we want to have some network programming there, I'd be happy to polish the patch so that it is acceptable for you.

[gollux]

Could you please try the setup-lo branch?

[jiridanek]

The setup-lo branch does what I need, thank you!

sudo isolate --dir=/nix/store=/nix/store --run -- /nix/store/fcd0m68c331j7nkdxvnnpb8ggwsaiqac-bash-5.1-p16/bin/bash -c '/nix/store/74sgkf3q4aqaf9hdp15c6fgfl3dxr2k6-iproute2-5.17.0/bin/ip a l'
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host 
       valid_lft forever preferred_lft forever
OK (0.004 sec real, 0.005 sec wall)

% sudo isolate --processes=42 --dir=/nix/store=/nix/store --run -- /nix/store/fcd0m68c331j7nkdxvnnpb8ggwsaiqac-bash-5.1-p16/bin/bash -c '/nix/store/6fhyahx97qsk6c2nxribmxrfla8l40bz-nmap-7.92/bin/ncat -l 8888 & echo baf | /nix/store/6fhyahx97qsk6c2nxribmxrfla8l40bz-nmap-7.92/bin/ncat 127.0.0.1 8888'
baf
OK (0.011 sec real, 0.009 sec wall)

% sudo isolate --processes=42 --dir=/nix/store=/nix/store --run -- /nix/store/fcd0m68c331j7nkdxvnnpb8ggwsaiqac-bash-5.1-p16/bin/bash -c '/nix/store/6fhyahx97qsk6c2nxribmxrfla8l40bz-nmap-7.92/bin/ncat ::1 -l 8888 & echo baf | /nix/store/6fhyahx97qsk6c2nxribmxrfla8l40bz-nmap-7.92/bin/ncat ::1 8888'
baf
OK (0.011 sec real, 0.009 sec wall)

[gollux]

Fine, it's merged to master now.