cirruslabs / gitlab-tart-executor

GitLab Runner executor to run jobs in Tart VMs
MIT License
60 stars 5 forks source link

allow specifying an IP resolver #43

Closed umlaeute closed 12 months ago

umlaeute commented 12 months ago

I'm trying to run tart as a gitlab-executor on a host, that is also running qemu: as a consequence, the host has two network bridges:

$ ifconfig
[...]
bridge0: flags=8863<UP,BROADCAST,SMART,RUNNING,SIMPLEX,MULTICAST> mtu 1500
    options=63<RXCSUM,TXCSUM,TSO4,TSO6>
    ether 36:5b:95:ab:14:40 
    Configuration:
        id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
        maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
        root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
        ipfilter disabled flags 0x0
    member: en2 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 12 priority 0 path cost 0
    member: en3 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 13 priority 0 path cost 0
    member: en4 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 14 priority 0 path cost 0
    member: en5 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 15 priority 0 path cost 0
    nd6 options=201<PERFORMNUD,DAD>
    media: <unknown type>
    status: inactive
[...]
bridge100: flags=8a63<UP,BROADCAST,SMART,RUNNING,ALLMULTI,SIMPLEX,MULTICAST> mtu 1500
    options=3<RXCSUM,TXCSUM>
    ether 9e:76:0e:a3:47:64 
    inet 192.168.66.1 netmask 0xffffff00 broadcast 192.168.66.255
    inet6 fe80::9c76:eff:fea3:4764%bridge100 prefixlen 64 scopeid 0x1c 
    inet6 fdbd:ab55:9e51:a104:14a4:c78c:623:634c prefixlen 64 autoconf secured 
    Configuration:
        id 0:0:0:0:0:0 priority 0 hellotime 0 fwddelay 0
        maxage 0 holdcnt 0 proto stp maxaddr 100 timeout 1200
        root id 0:0:0:0:0:0 priority 0 ifcost 0 port 0
        ipfilter disabled flags 0x0
    member: vmenet0 flags=3<LEARNING,DISCOVER>
            ifmaxaddr 0 port 27 priority 0 path cost 0
    nd6 options=201<PERFORMNUD,DAD>
    media: autoselect
    status: active

now, for many (if not all) VMs brought up by tart, tart ip reports an address in the 192.168.66.1/24 network, although the VMs themselves report an IP in the 192.168.67.1/24 network, which means that the gitlab-runner cannot access the VM.

now I've found that tart ip --resolver arp returns the correct IP address (the one that the VM uses internally).

therefore I conclude that if I was able to tell gitlab-tart-executor to use the arp resolver rather than the default dhcp resolver, my problem could be worked around.

unfortunately, i haven't found a way to specify such a configuration option.

i guess the ideal solution would be to allow for a configuration file that is read by the executor and can be used to override options to specific tart invocations, e.g.

# ~/.tart/gitlab-tart-executor.toml
[ip]
resolver = "arp"
## in case your VMs are booting really slowly...:
wait = 120

an alternative idea (though i don't like it very much) could be using envvars. e.g.

$ cat /usr/local/bin/gitlab-tart-executor-wrapper
#!/bin/sh
export TART_IP_RESOLVER=arp
gitlab-tart-executor "$@"

$ cat .gitlab-runner/config.toml 
[...]
[[runners]]
  [runners.custom]
    prepare_exec = "/usr/local/bin/gitlab-tart-executor-wrapper"
    prepare_args = ["prepare"]
    run_exec = "/usr/local/bin/gitlab-tart-executor-wrapper"
    run_args = ["run"]
    cleanup_exec = "/usr/local/bin/gitlab-tart-executor-wrapper"
    cleanup_args = ["cleanup"]

I think the only important thing is to be able to set this on a per runner basis (rather than in the .gitlab-ci.yml configuration, as the users of the runner should be oblivious of the actual host specifics)

This is a follow up of https://github.com/cirruslabs/tart/issues/668

edigaryev commented 12 months ago

Let's address this via https://github.com/cirruslabs/tart/pull/669 and re-open if that doesn't help.