Add Support to specific a NAT Gateway IP that will be used as default route

[mueller-tobias]

Current SDK Version

6.0.1

Use-cases

When using private Networks, it's no possible for the new Nodes to connect to the internet to fetch packages, images or helm charts needed for the installation. To gain internet access the nodes need to have a route to the nat gateway. The nat gateway will be responsible for the internet access of the private lan.

Attempted Solutions

Add a NAT Gateway Parameter in the Node Template Configuration. This value will then be used to configure the static router inside the vm build by the docker machine driver

Proposal

Add the ability to configure the IP of the NAT Gateway via Node Template.

[avirtopeanu-ionos]

For this issue to be closed, would you also need the driver to support NAT Gateway Rules and NAT Flow Logs? Or is adding support for standalone NAT Gateways enough?

NAT Rules allow controlling traffic through the NAT Gateway by specifying a desired protocol, a source and target subnet, and a whitelisted port range. NAT Flow logs monitors traffic through the NAT on the specified direction for accepted/rejected packets (or all). NAT Flow logs output to an S3 bucket.

[mueller-tobias]

The optimal solution would be to support NAT Gateway Rules and NAT Flow Logs. But in the first step, support for a standalone, already existing and configured NAT Gateway would also be ok.

[grieshaber]

I'd be really interested in this feature aswell. My use case is a GItlab CI environment based on the docker machine executor. The worker nodes, spawned by the docker-machine driver, should be in a private network but have access to the internet.

I create the NAT Gateway, the SNAT Rule and the private lan beforehand but the worker do miss the default route to the network. Additionally, on the worker instances spawned that way, the worker nodes do not have any nameserver configured. So far, i help myself with cloud-init scripts. But, because of the order of execution of the cloud-init parts, addind the route and ns server is to late; some things (as installing docker for example) are broken by then already.

Any chance, this issue will be picked up soon?

[mueller-tobias]

I'd be really interested in this feature aswell. My use case is a GItlab CI environment based on the docker machine executor. The worker nodes, spawned by the docker-machine driver, should be in a private network but have access to the internet.

I create the NAT Gateway, the SNAT Rule and the private lan beforehand but the worker do miss the default route to the network. Additionally, on the worker instances spawned that way, the worker nodes do not have any nameserver configured. So far, i help myself with cloud-init scripts. But, because of the order of execution of the cloud-init parts, addind the route and ns server is to late; some things (as installing docker for example) are broken by then already.

Any chance, this issue will be picked up soon?

We've switchted to our own DHCP and Firewall to mitigate the NAT Gateway Problem, but something that worked to configure the network was to run the commands via bootcmd.

Here's a example from our terraform cloud-config template that you can maybe adapt to your needs:

bootcmd:
  - |
    # Note the "z" in the file name. This is needed to run in after 99_ionos.cfg
    echo -e "# $(date)\nnetwork: {config: disabled}" > /etc/cloud/cloud.cfg.d/99_z_disable_network_config.cfg
  - |
    if [ ! -e /etc/netplan/50-custom-network-init.yaml ]; then
    FIRST_NIC=$(ip -o link | grep -o 'ens[0-9]\+' | sed -n 1p)
    echo "
    # Custom network configuration
    network:
      ethernets:
        $${FIRST_NIC}:
          dhcp6: false
%{ if ip_address != "" ~}
          dhcp4: false
          addresses:
          - ${ip_address}
          routes:
%{ if custom_routes == "" ~}
            - to: default
              via: ${gateway}
%{ else ~}
            ${indent(12,custom_routes)}
%{ endif ~}
          nameservers:
            addresses:
%{ for nameserver in nameservers ~}
              - ${nameserver}
%{ endfor ~}
%{ else ~}
          dhcp4: true
%{ endif ~}
        otherInterfaces:
          match:
            name: ens*
          dhcp4: true
      renderer: networkd
      version: 2
    " > /etc/netplan/50-custom-network-init.yaml
    rm /etc/netplan/50-cloud-init.yaml
    netplan generate
    fi

Now you just need the correct netplan configuration for your needs.

[grieshaber]

@mueller-tobias thanks - i got it up and running with a slightly adapted version of your bootcmd! :)

[rmocanu-ionos]

Hello, glad you managed to solve the issue! We're hoping to add the option to customize flowlogs and rules this week!

[grieshaber]

Hi @rmocanu-ionos - what exactly will be tackled with that addition? Will we be able to add routes through the nat gateway at machine-bootup automatically? Or do we still need to use the custom-bootcmd from above?

[rmocanu-ionos]

You will be able to specify nat rules to be added to the NAT gateway when it is created

[grieshaber]

But i still need to make sure, that the nat is configured as a Gateway in the vm myself?

Then i had to high hopes for that change - i provision everything (including the nat and rules) with terraform ..

[rmocanu-ionos]

Hello! Apologies for the delayed response. Regarding your query on NAT configuration, yes, you're correct: the NAT must be manually set up as a Gateway within the VM. As for the Terraform aspect, it would be helpful if you could provide more specifics. Are the NAT and its associated rules already set up using Terraform, and your requirement is to link these to the VMs being created by the Docker machine? Looking forward to your clarification!

[grieshaber]

Hi @rmocanu-ionos thanks for coming back to my question.

Exactly - for my use case (described here https://github.com/ionos-cloud/docker-machine-driver/issues/46#issuecomment-1711107965) I want to control the lifecycle of the NAT and corresponding resources my self with terraform, which is working just fine.

I then want - by using the docker-machine driver with the GitLab Runner - to dynamically spawn new VMs in the DC, which are solely connected to the private LAN and only access the internet via the NAT Gateway created earlier (so adding a default route via NAT GW to the VMs is needed).

As i do have a "working solution" with our custom bootstrap script, we did encounter several issues with that resulting in a non-deterministic VM spawning process (e.g. sometime installation of packages etc. fail, because establishing the internet route is too late / too slow) - for now, we needed to drop the approach completely, as additionally to the sporadically failing provisioning, the provisioning process itself too long for a dynamic CI environment.

I'm curious, what's on the roadmap, that would make our use-case feasible on ionos-cloud!

[rmocanu-ionos]

Hi there, We've looked into the issue and it seems we might be able to add the necessary information to the cloud init when the NAT already exists. However, we'll need to conduct further tests to confirm this. I'll keep you updated if there are any changes or progress on our end. Best regards, Radu

[rmocanu-ionos]

Hello, We attempted to add the NAT IP using cloud-init from the driver, but the current setup does not support it. Due to limited resources, we are unable to address this now. We'll close the issue and revisit it later. Radu