NMS is not reachable behind Ingress

NMS ingress integration is not working properly

If we deploy sdcore bundle from edge channel, NMS UI is not reachable.

To Reproduce

Deploy sdcore bundle: $ juju deploy sdcore --trust --channel=edge

Get NMS URL: $ juju run traefik-k8s/0 show-proxied-endpoints Running operation 1 with 1 task

task 2 on unit-traefik-k8s-0

Waiting for task 2... proxied-endpoints: '{"nms": {"url": "http://10.0.0.3:80/core-nms"}}'

URL starts with http, expected to be an https.

Trying to reach URL: $ curl -vvv http://10.0.0.3/core-nms

Trying 10.0.0.3:80...
Connected to 10.0.0.3 (10.0.0.3) port 80 (#0)

GET /core-nms HTTP/1.1 Host: 10.0.0.3 User-Agent: curl/7.88.1 Accept: /

< HTTP/1.1 308 Permanent Redirect < Date: Mon, 20 Nov 2023 21:15:37 GMT < Location: /network-configuration < Refresh: 0;url=/network-configuration < Transfer-Encoding: chunked <
Connection #0 to host 10.0.0.3 left intact /network-configuration

It is not routed to any backend.

Expected behavior

NMS URL should return the backend successfully.

Screenshots

http://nms-0.nms-endpoints.core.svc.cluster.local:3000/ is resolved as http://10.1.146.51:3000/

Logs

Bundle is deployed. $ juju status Model Controller Cloud/Region Version SLA Timestamp core microk8s-localhost microk8s/localhost 3.1.6 unsupported 16:38:16+03:00

App Version Status Scale Charm Channel Rev Address Exposed Message amf active 1 sdcore-amf edge 51 10.152.183.110 no
ausf active 1 sdcore-ausf edge 38 10.152.183.194 no
grafana-agent-k8s 0.32.1 waiting 1 grafana-agent-k8s latest/stable 42 10.152.183.158 no installing agent mongodb-k8s active 1 mongodb-k8s 5/edge 36 10.152.183.107 no Primary nms active 1 sdcore-nms edge 23 10.152.183.98 no
nrf active 1 sdcore-nrf edge 57 10.152.183.53 no
nssf active 1 sdcore-nssf edge 35 10.152.183.111 no
pcf active 1 sdcore-pcf edge 30 10.152.183.119 no
router active 1 sdcore-router edge 30 10.152.183.109 no
self-signed-certificates active 1 self-signed-certificates beta 33 10.152.183.216 no
smf active 1 sdcore-smf edge 33 10.152.183.48 no
traefik-k8s 2.9.6 active 1 traefik-k8s latest/stable 129 10.0.0.3 no
udm active 1 sdcore-udm edge 33 10.152.183.26 no
udr active 1 sdcore-udr edge 28 10.152.183.178 no
upf active 1 sdcore-upf edge 62 10.152.183.173 no
webui active 1 sdcore-webui edge 20 10.152.183.112 no

Unit Workload Agent Address Ports Message amf/0 active idle 10.1.146.26
ausf/0 active idle 10.1.146.45
grafana-agent-k8s/0 blocked idle 10.1.146.59 logging-consumer: off, grafana-cloud-config: off mongodb-k8s/0 active idle 10.1.146.25 Primary nms/0 active idle 10.1.146.9
nrf/0 active idle 10.1.146.44
nssf/0 active idle 10.1.146.49
pcf/0 active idle 10.1.146.35
router/0 active idle 10.1.146.5
self-signed-certificates/0 active idle 10.1.146.42
smf/0 active idle 10.1.146.19
traefik-k8s/0 active idle 10.1.146.46
udm/0 active idle 10.1.146.55
udr/0 active idle 10.1.146.13
upf/0 active idle 10.1.146.4
webui/0 active idle 10.1.146.30

We get 404 error, suspecting that Traefik does route it properly. curl -vvv https://10.0.0.3:80/core-nms -k

Trying 10.0.0.3:80...
Connected to 10.0.0.3 (10.0.0.3) port 80 (#0)
ALPN: offers h2,http/1.1
TLSv1.3 (OUT), TLS handshake, Client hello (1):
TLSv1.3 (IN), TLS handshake, Server hello (2):
TLSv1.3 (IN), TLS handshake, Encrypted Extensions (8):
TLSv1.3 (IN), TLS handshake, Certificate (11):
TLSv1.3 (IN), TLS handshake, CERT verify (15):
TLSv1.3 (IN), TLS handshake, Finished (20):
TLSv1.3 (OUT), TLS change cipher, Change cipher spec (1):
TLSv1.3 (OUT), TLS handshake, Finished (20):
SSL connection using TLSv1.3 / TLS_AES_128_GCM_SHA256
ALPN: server accepted h2
Server certificate:
subject: CN=TRAEFIK DEFAULT CERT
start date: Nov 20 13:23:41 2023 GMT
expire date: Nov 19 13:23:41 2024 GMT
issuer: CN=TRAEFIK DEFAULT CERT
SSL certificate verify result: self-signed certificate (18), continuing anyway.
using HTTP/2
h2h3 [:method: GET]
h2h3 [:path: /core-nms]
h2h3 [:scheme: https]
h2h3 [:authority: 10.0.0.3:80]
h2h3 [user-agent: curl/7.88.1]
h2h3 [accept: /]
Using Stream ID: 1 (easy handle 0x55d1bf224670)

GET /core-nms HTTP/2 Host: 10.0.0.3:80 user-agent: curl/7.88.1 accept: /
TLSv1.3 (IN), TLS handshake, Newsession Ticket (4): < HTTP/2 404 < content-type: text/plain; charset=utf-8 < x-content-type-options: nosniff < content-length: 19 < date: Mon, 20 Nov 2023 13:39:26 GMT < 404 page not found
Connection #0 to host 10.0.0.3 left intact

Environment

Juju version (output from juju --version): 3.1.6
Cloud Environment: Barematel host, ubuntu 23.04
Kubernetes version (output from kubectl version --short): Client Version: v1.27.7 Kustomize Version: v5.0.1 Server Version: v1.27.7

Additional context

This is ingress.yaml in Traefik.

root@traefik-k8s-0:/var/lib/juju/storage/configurations/0/juju# cat juju_ingress_ingress_33_nms.yaml http: middlewares: juju-sidecar-noprefix-core-nms: stripPrefix: forceSlash: false prefixes:

/core-nms routers: juju-core-nms-router: entryPoints:
- web middlewares:
- juju-sidecar-noprefix-core-nms rule: PathPrefix(/core-nms) service: juju-core-nms-service juju-core-nms-router-tls: entryPoints:
- websecure middlewares:
- juju-sidecar-noprefix-core-nms rule: PathPrefix(/core-nms) service: juju-core-nms-service tls: {} services: juju-core-nms-service: loadBalancer: servers:
url: http://nms-0.nms-endpoints.core.svc.cluster.local:3000

curl -vvv http://nms-0.nms-endpoints.core.svc.cluster.local:3000

Trying 10.1.146.51:3000...
TCP_NODELAY set
Connected to nms-0.nms-endpoints.core.svc.cluster.local (10.1.146.51) port 3000 (#0)

GET / HTTP/1.1 Host: nms-0.nms-endpoints.core.svc.cluster.local:3000 User-Agent: curl/7.68.0 Accept: /
Mark bundle as not supporting multiuse < HTTP/1.1 308 Permanent Redirect < location: /network-configuration < Refresh: 0;url=/network-configuration < Date: Mon, 20 Nov 2023 21:01:57 GMT < Connection: keep-alive < Keep-Alive: timeout=5 < Transfer-Encoding: chunked <
Connection #0 to host nms-0.nms-endpoints.core.svc.cluster.local left intact /network-configuration

Juju debug logs:

unit-traefik-k8s-0: 11:01:10 INFO unit.traefik-k8s/0.juju-log Kubernetes service 'traefik-k8s' patched successfully unit-traefik-k8s-0: 11:01:11 ERROR unit.traefik-k8s/0.juju-log invalid databag contents: expecting json. {'host': 'nms-0.nms-endpoints.core.svc.cluster.local', 'model': 'core', 'name': 'nms', 'port': '3000', 'strip-prefix': 'true'} unit-traefik-k8s-0: 11:01:11 WARNING unit.traefik-k8s/0.juju-log is using a deprecated ingress v1 protocol to talk to Traefik. Please inform the maintainers of 'nms' that they should bump to v2. unit-traefik-k8s-0: 11:01:11 WARNING unit.traefik-k8s/0.juju-log provider: <charms.traefik_k8s.v1.ingress.IngressPerAppProvider object at 0x7f3cc7e72dc0> unit-traefik-k8s-0: 11:01:11 WARNING unit.traefik-k8s/0.juju-log providing ingress over ingress v1: handling it as ingress per leader (legacy) unit-traefik-k8s-0: 11:01:11 ERROR unit.traefik-k8s/0.juju-log invalid databag contents: expecting json. {'host': 'nms-0.nms-endpoints.core.svc.cluster.local', 'model': 'core', 'name': 'nms', 'port': '3000', 'strip-prefix': 'true'}

canonical / sdcore-bundles