OpenShift Virtualization testing—publicly expose non-http services

[computate]

• starting point https://docs.openshift.com/container-platform/4.15/virt/vm_networking/virt-networking-overview.html
  • centOS stream 9 VM, expose postgres externally (non HTTP service)
  • best: with end to end TLS & security
  • kafka
  • postgres
  • ...

[computate]

I will be working on it this week.

Here are some steps I did to install a PostgreSQL database for testing the VM.

$ sudo yum install -y postgresql-server postgresql-contrib
$ sudo systemctl start postgresql
$ sudo systemctl enable postgresql

$ sudo -u postgres psql
$ create user test password '...';
$ create database test owner test;
$ create table test(pk bigserial primary key, id text unique, val text);

$ postgres=# insert into test(id, val) values('best-linux-flavor', 'rhel9');
INSERT 0 1
$ postgres=# insert into test(id, val) values('best-linux-flavor', 'ubuntu24.04');
ERROR:  duplicate key value violates unique constraint "test_id_key"

[computate]

I added the following template label to my VirtualMachine oc -n virt-test get vm/computate-centos-stream9 for service support for my postgresql server.

spec:
  template:
    metadata:
      labels:
        computate-postgres-test: 'true'

Also add exposed ports to the default interface of the VirtualMachine.

spec:
  template:
    spec:
      domain:
        devices:
          interfaces:
            - name: default
              masquerade: {} 

Specifically add this port:

              ports: 
                - port: 5432

[computate]

Update your postgresql pg_hba.conf file to allow md5 password access to external users.

sudo vim /var/lib/pgsql/data/pg_hba.conf
host all all 0.0.0.0/0 md5
:wq

Also update your postgresql.conf file listen_addresses.

sudo vim /var/lib/pgsql/data/postgresql.conf
listen_addresses = '*'
:wq

Then restart the postgresql service.

sudo systemctl restart postgresql

[computate]

apiVersion: v1
kind: Service
metadata:
  name: computate-postgres-test
  namespace: virt-test
spec:
  selector:
    computate-postgres-test: 'true' 
  type: NodePort 
  ports: 
    - protocol: TCP
      port: 5432
      targetPort: 5432
      nodePort: 30432

Notes:

• When creating a service, be sure to keep it within the following nodePort numbers: The range of valid ports is 30000-32767" for field "spec.ports[0].nodePort".
• If you're VM is running when you add the template label, stop and start your VM.

[computate]

@jtriley says: AFAIK the ingress controller only runs on ports 80 and 443 so unless you're configuring a route to that service, that's not going to work

@larsks says: A nodeport exposes a service on a port on all the worker nodes. The address computate-postgres-test.apps.ocp-test.nerc.mghpcc.org resolves to the cluster load balancer, not to one of the worker nodes. Your nodeport is probably working; try contacting it on one of the worker node addresses. The problem is that this won't do you much good, because the worker nodes are only VPN accessible. To expose non-http services (or services on ports other than 80/443), we need to either install and configure MetalLB, or configure an external cluster load balancer with kubernetes support.

@joachimweyl we have some questions for the NERC team in a future meeting:

• Do we still wish to publicly expose non-http services in the test and prod OpenShift clusters?
• Do we want to set up MetalLB?
• Do we want to set up an external cluster load balancer with kubernetes support?

[computate]

@jtriley, @larsks, @naved001 Based in the conversation today in the NERC meeting, woud you be able to help with the MetalLB configuration?

[naved001]

Here are the steps I think we'll need to take:

1. Get an interface configured on the test cluster worker nodes to be on a public network. This means any necessary VLAN configuration and IP assignment.
2. Install the metallb operator
3. configure metallb with an ipaddresspool resource and an l2advertisement. There's also support for BGP but I am not familiar with it.
4. Take care of the asymmetric routing issue. See https://github.com/CCI-MOC/ops-issues/issues/1376 for more details.

@jtriley since I am not familiar with how public IPs are allocated in the NERC/Harvard environment we'll need your help here.

[joachimweyl]

@computate please provide an update.

[computate]

I don't have additional updates here. We'll need to configure MetalLB next.

[computate]

@jtriley do you have a timeframe on when we could do the first step that @naved001 suggested?

Get an interface configured on the test cluster worker nodes to be on a public network. This means any necessary VLAN configuration and IP assignment.

I would like to enable SSH access on some VMs in the test cluster, and also test exposing non-http services, as well as Sensu monitoring agent on a VM to demonstrate event driven remediation of issues on a virtual machine.

[computate]

Where I would love to start is actually accessing a VM internally via SSH credentials from Red Hat Ansible Automation Platform deployed in the same namespace as my virtual machines if that is easier to set up and allow. See my aap-controller route in the virt-test namespace.

[computate]

Actually, I was able to get internal ssh access to VMs working from Ansible Automation Platform in the same namespace by:

• generating a new ssh-keygen rsa key
• Adding a key when creating a VM from a template
• Configuring masquerade mode from the command line with an exposed 22 port
• Exposing a virtual machine by using a ClusterIP service
• Creating an AAP inventory, and host to the VM's internal IP address
• Creating an AAP Machine SSH private key credential

[schwesig]

For now using port forwarding, see NERC weekly Operation rolling agenda please comment/emoticon a yes