search

onedr0p / exportarr

AIO Prometheus Exporter for Sabnzbd, Bazarr, Prowlarr, Lidarr, Readarr, Radarr, and Sonarr
MIT License
537 stars 40 forks source link

exportarr prefers port from config.xml instead of CLI flag #276

Closed volker-raschek closed 6 months ago

volker-raschek commented 6 months ago

Hi there, I've tried to get exportarr running on kubernetes, but it seems that exportarr preferes the port of the config.xml instead of the port configured via CLI flag --url. I belive the same error also exists for radarr, readarr, sonarr and prowlarr.

I declared in my current set up a kubernetes service to forward incoming traffic to each instance of a lidarr pod. The port where the application (pod) is listening on is the default - like for lidarr 8686.

To access lidarr via app or browser I need to change the service type from ClusterIP to LoadBalancer. This is no problem for me, because the kubernetes cluster is running locally and can not access via internet. For user-friendliness I've changed the port of the service from 8686 to 80. Lidarr can therefore be opened directly in the browser without specifying a port, because port 80 is the default. The incoming data traffic is therefore forwarded from port 80 of the kubernetes service to port 8686 of the lidarr instance.

Now to exportarr. I have provided the config.xml of lidarr via volumeMount to exportarr and specified the absolute file path via the --config flag. Since exportarr now reads the config and finds the port 8686 there, exportarr tries to reach the lidarr API via this port. However, this is not possible because the port is not released by the service. 

2024-03-03T15:12:18.758Z        INFO    Starting exportarr      {"app_name": "exportarr", "version": "v1.5.3", "buildTime": "2023-06-14T18:41:40.589Z", "revision": "952a768edf22789f6b07005cdce18898c1ef3a2a"}
2024-03-03T15:12:18.760Z        INFO    Starting HTTP Server    {"interface": "0.0.0.0", "port": 9707}
2024-03-03T15:13:00.728Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/system/status"}
2024-03-03T15:13:00.730Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/health"}
2024-03-03T15:13:00.731Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/artist"}
2024-03-03T15:13:00.731Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/queue?page=1"}
2024-03-03T15:13:00.731Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/history"}
2024-03-03T15:13:00.731Z        INFO    Sending HTTP request    {"url": "http://lidarr:8686/api/v1/rootfolder"}
2024-03-03T15:13:02.778Z        ERROR   Error getting rootfolder        {"collector": "rootfolder", "error": "Failed to execute HTTP Request(http://lidarr:8686/api/v1/rootfolder): Get \"http://lidarr:8686/api/v1/rootfolder\": Error sending HTTP Request: dial tcp 10.100.75.74:8686: connect: connection refused"}

My exportarr deployment. The URL of the flag --url points to the service lidarr, port 80.

apiVersion: apps/v1
kind: Deployment
metadata:
  annotations:
    deployment.kubernetes.io/revision: "1"
    meta.helm.sh/release-name: lidarr
    meta.helm.sh/release-namespace: media
  labels:
    app.kubernetes.io/instance: lidarr
    app.kubernetes.io/managed-by: Helm
    app.kubernetes.io/name: lidarr-exportarr
    app.kubernetes.io/version: 1.4.5
    helm.sh/chart: lidarr-0.1.0-exportarr
  name: lidarr-exportarr
  namespace: media
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app.kubernetes.io/instance: lidarr
      app.kubernetes.io/name: lidarr-exportarr
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      creationTimestamp: null
      labels:
        app.kubernetes.io/instance: lidarr
        app.kubernetes.io/name: lidarr-exportarr
    spec:
      containers:
      - args:
        - lidarr
        - --config=/config/config.xml
        - --port=9707
        - --url=http://lidarr:80
        image: ghcr.io/onedr0p/exportarr:v1.5.3
        imagePullPolicy: IfNotPresent
        livenessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: monitoring
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        name: exportarr
        ports:
        - containerPort: 9707
          name: monitoring
          protocol: TCP
        readinessProbe:
          failureThreshold: 5
          httpGet:
            path: /healthz
            port: monitoring
            scheme: HTTP
          periodSeconds: 10
          successThreshold: 1
          timeoutSeconds: 1
        resources:
          limits:
            cpu: 500m
            memory: 256Mi
          requests:
            cpu: 100m
            memory: 64Mi
        terminationMessagePath: /dev/termination-log
        terminationMessagePolicy: File
        volumeMounts:
        - mountPath: /config
          name: config
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 30
      volumes:
      - name: config
        persistentVolumeClaim:
          claimName: lidarr-config

UPDATE

The bug also exists in the latest version - v1.6.2.

Workaround

  1. Remove the volume mount of the config.xml file.
  2. Start lidarr
  3. Extract the API key via GUI
  4. Define the API key via --api-key flag.
  5. Exportarr use the specified port as part of the --url flag as expected

Volker

onedr0p commented 6 months ago

You should definitely be using the cluster dns address to have exportarr find lidarr, not using the load balancer address. So the url arg should be:

--url=http://lidarr.namespace.svc.cluster.local

Then it can properly use the apikey and port defined in the config.xml

rtrox commented 6 months ago

re: ordering, that's correct. The order config is applied is:

Defaults -> Env -> Flags -> XMLConfig

We can change that pretty easily (it probably makes sense for it to be Defaults -> XML -> Env ->Flags), but the behavior's been that way for a bunch of years at this point, so it would be another breaking change, though I think it's unlikely folks were counting on that ordering. @onedr0p, not sure how you feel about changing the order

onedr0p commented 6 months ago

I think it's fine the way we have it? The issue seems more about not using the correct address for exportarr to connect to Lidarr. As OP stated they are using a load balancer svc instead of an ingress to access Lidarr from outside the cluster and not using cluster dns to access it within the cluster I suspect that is the case.

volker-raschek commented 6 months ago

Hi @rtrox and @onedr0p,

As OP stated they are using a load balancer svc instead of an ingress to access Lidarr from outside the cluster...

That is correct. An ingress is also not necessary. TLS/SSL does not need to be terminated as the k8s cluster is located locally. This makes the ingress superfluous at this point.

and not using cluster dns to access it within the cluster I suspect that is the case.

I find the terminology of Cluster DNS somewhat confusing. I explicitly specify the DNS name of the LoadBalancer in the url. This could also have been called lidarr.media or lidarr.media.svc as the service dns spec describes. The host name does not have to match the host name of the config.xml. The configured domain name of the cluster like cluster.local should be skipped, because this domain name can be different from k8s environment to k8s environment. Alternatively, I could make the domain name of the cluster configurable in the Helm chart. However, the domain name of the cluster is not known to every administrator who is able to deploy an application. For this reason I like to avoid this if not absolutely necessary.

I deployed exportarr again with the extend version of the DNS name of the service like lidarr.media.svc for testing purpose. It seems, that this issue still exists:

2024-03-04T07:53:19.750Z        INFO    Starting exportarr      {"app_name": "exportarr", "version": "v1.6.2", "buildTime": "2024-02-28T20:53:28.098Z", "revision": "0cb318682ee0733e460aae27b6acdb0aa1045162"}
2024-03-04T07:53:19.752Z        INFO    Starting HTTP Server    {"interface": "0.0.0.0", "port": 9707}
2024-03-04T07:53:57.705Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/history"}
2024-03-04T07:53:57.706Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/rootfolder"}
2024-03-04T07:53:57.706Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/queue?page=1"}
2024-03-04T07:53:57.707Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/system/status"}
2024-03-04T07:53:57.706Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/health"}
2024-03-04T07:53:57.707Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc:8686/api/v1/artist"}
2024-03-04T07:53:59.761Z        ERROR   Error getting history   {"collector": "history", "error": "Failed to execute HTTP Request(http://lidarr.media.svc:8686/api/v1/history): Get \"http://lidarr.media.svc:8686/api/v1/history\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T07:54:01.812Z        ERROR   Error getting rootfolder        {"collector": "rootfolder", "error": "Failed to execute HTTP Request(http://lidarr.media.svc:8686/api/v1/rootfolder): Get \"http://lidarr.media.svc:8686/api/v1/rootfolder\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T07:54:08.051Z        ERROR   Error getting health: Failed to execute HTTP Request(http://lidarr.media.svc:8686/api/v1/health): Get "http://lidarr.media.svc:8686/api/v1/health": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused    {"collector": "systemHealth"}
2024-03-04T07:54:08.051Z        ERROR   Error getting queue     {"collector": "queue", "error": "Failed to execute HTTP Request(http://lidarr.media.svc:8686/api/v1/queue?page=1): Get \"http://lidarr.media.svc:8686/api/v1/queue?page=1\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T07:54:12.970Z        ERROR   Error creating client   {"collector": "lidarr", "error": "Failed to execute HTTP Request(http://lidarr.media.svc:8686/api/v1/artist): Get \"http://lidarr.media.svc:8686/api/v1/artist\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}

Furthermore, I've tried deploying exportarr with the fully qualified domain name of the service lidarr.media.svc.cluster.local, but also this does not work. 

2024-03-04T07:58:44.148Z        INFO    Starting exportarr      {"app_name": "exportarr", "version": "v1.6.2", "buildTime": "2024-02-28T20:53:28.098Z", "revision": "0cb318682ee0733e460aae27b6acdb0aa1045162"}
2024-03-04T07:58:44.162Z        INFO    Starting HTTP Server    {"interface": "0.0.0.0", "port": 9707}
markus@markus-pc:~/workspace/charts/charts.cryptic.systems/lidarr/lidarr (master *=)$ kubectl logs lidarr-exportarr-5b6fbc996f-87g7t -f
2024-03-04T07:58:44.148Z        INFO    Starting exportarr      {"app_name": "exportarr", "version": "v1.6.2", "buildTime": "2024-02-28T20:53:28.098Z", "revision": "0cb318682ee0733e460aae27b6acdb0aa1045162"}
2024-03-04T07:58:44.162Z        INFO    Starting HTTP Server    {"interface": "0.0.0.0", "port": 9707}
2024-03-04T08:03:01.562Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/history"}
2024-03-04T08:03:01.562Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/queue?page=1"}
2024-03-04T08:03:01.562Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/rootfolder"}
2024-03-04T08:03:01.562Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/system/status"}
2024-03-04T08:03:01.563Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/health"}
2024-03-04T08:03:01.563Z        INFO    Sending HTTP request    {"url": "http://lidarr.media.svc.cluster.local:8686/api/v1/artist"}
2024-03-04T08:03:03.598Z        ERROR   Error creating client   {"collector": "lidarr", "error": "Failed to execute HTTP Request(http://lidarr.media.svc.cluster.local:8686/api/v1/artist): Get \"http://lidarr.media.svc.cluster.local:8686/api/v1/artist\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T08:03:09.919Z        ERROR   Error getting history   {"collector": "history", "error": "Failed to execute HTTP Request(http://lidarr.media.svc.cluster.local:8686/api/v1/history): Get \"http://lidarr.media.svc.cluster.local:8686/api/v1/history\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T08:03:12.051Z        ERROR   Error getting rootfolder        {"collector": "rootfolder", "error": "Failed to execute HTTP Request(http://lidarr.media.svc.cluster.local:8686/api/v1/rootfolder): Get \"http://lidarr.media.svc.cluster.local:8686/api/v1/rootfolder\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T08:03:12.052Z        ERROR   Error getting queue     {"collector": "queue", "error": "Failed to execute HTTP Request(http://lidarr.media.svc.cluster.local:8686/api/v1/queue?page=1): Get \"http://lidarr.media.svc.cluster.local:8686/api/v1/queue?page=1\": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused"}
2024-03-04T08:03:16.964Z        ERROR   Error getting health: Failed to execute HTTP Request(http://lidarr.media.svc.cluster.local:8686/api/v1/health): Get "http://lidarr.media.svc.cluster.local:8686/api/v1/health": Error sending HTTP Request: dial tcp 10.104.64.117:8686: connect: connection refused        {"collector": "systemHealth"}
onedr0p commented 6 months ago

The configured domain name of the cluster like cluster.local should be skipped, because this domain name can be different from k8s environment to k8s environment. Alternatively, I could make the domain name of the cluster configurable in the Helm chart. However, the domain name of the cluster is not known to every administrator who is able to deploy an application. For this reason I like to avoid this if not absolutely necessary.

99% of the time the cluster domain is going to be cluster.local unless this is changed, in most cases it is not and if it is the cluster admin should already know ahead of time because it affects deploying coredns(along with a ton of other things) which is a fundamental component to a Kubernetes cluster.

Furthermore, I've tried deploying exportarr with the fully qualified domain name of the service lidarr.media.svc.cluster.local, but also this does not work.

This is pretty confusing because you should be able to reach services via their cluster dns address and port, this is a very standard way services talk to each other in Kubernetes. With the lidarr container is it not listening on :8686 ? You made it seem like the load Balancer address is doing the :8686 to :80 port mapping but this was not clear.

One option you might not have considered is to use the containers I build for Lidarr / Radarr / Sonarr which allow you to define the API Key, PORT etc.. in the container env vars. You can pre-set the API key this way and have exportarr and Lidarr pull from the same Kubernetes secret without having to provide the config.xml

Feel free to check out the containers I build here: https://github.com/onedr0p/containers

onedr0p commented 6 months ago

Some debugging steps you might find useful using netshoot... If any of these fail you might have a misconfiguration issue or DNS issues that could be more systemic than the exportarr problems you are facing.

❯ kubectl run tmp-shell --rm -i --tty --image nicolaka/netshoot
If you don't see a command prompt, try pressing enter.
                    dP            dP                           dP
                    88            88                           88
88d888b. .d8888b. d8888P .d8888b. 88d888b. .d8888b. .d8888b. d8888P
88'  `88 88ooood8   88   Y8ooooo. 88'  `88 88'  `88 88'  `88   88
88    88 88.  ...   88         88 88    88 88.  .88 88.  .88   88
dP    dP `88888P'   dP   `88888P' dP    dP `88888P' `88888P'   dP

Welcome to Netshoot! (github.com/nicolaka/netshoot)
Version: 0.12
 tmp-shell  ~  dig +short sonarr.default.svc.cluster.local
10.43.246.132
 tmp-shell  ~  telnet sonarr.default.svc.cluster.local 8989
Connected to sonarr.default.svc.cluster.local
 tmp-shell  ~  curl sonarr.default.svc.cluster.local:8989
<!doctype html><html lang="en"><head><meta charset="utf-8"/>
....
volker-raschek commented 6 months ago

Hi @onedr0p, here is the configuration issue:

telnet sonarr.default.svc.cluster.local 8989

I don't want to publish the service ports 8989 of sonarr or port 8787 of lidarr, because than I also need to specify this port in the URL in my browser. As I already mentioned, I use the port 80 instead. This is the default port for HTTP and is not required to define in the URL.

You made it seem like the load Balancer address is doing the :8686 to :80 port mapping but this was not clear.

Exactly, I realize the port mapping via the service and exportarr does not seems to support this, because exportarr priorize the configured port of the config.xml instead of the passed CLI flag --url.

One option you might not have considered is to use the containers I build for Lidarr / Radarr / Sonarr which allow you to define the API Key, PORT etc.. in the container env vars. You can pre-set the API key this way and have exportarr and Lidarr pull from the same Kubernetes secret without having to provide the config.xml

However, I really like the configuration with config.xml. The reason for this is relatively simple. When the API key is generated in the GUI, it is also saved in the config.xml. This means that exportarr can react to this via fs.Notify and read in the API key again. The option to define the API key via environment variable does not offer this possibility.

rtrox commented 6 months ago

@volker-raschek - from what it sounds like, you have this set up in a pretty non-standard way -- what you're describing here (using one port mapping for access, and another for internal access) is the perfect usecase for an ingress. If it's helpful, you can see an example using exportarr, nginx-ingress, and external-dns here.

Given the non-standard setup, I think @onedr0p and I are agreed that we shouldn't make a breaking change to the config parsing which would impact other users. If you need help setting up ingresses, or troubleshooting the network issues @onedr0p mentioned, I think the folks in the k8s-at-home discord would be more than happy to help troubleshoot with you in the support channel, but I'm going to go ahead and close this issue for now!

onedr0p commented 6 months ago

Couple things I wanted to add are:

Exactly, I realize the port mapping via the service and exportarr does not seems to support this, because exportarr priorize the configured port of the config.xml instead of the passed CLI flag --url.

You should be using cluster DNS and not taking the full trip to the load balacer IP and back to access exportarr since both the arr and exportarr are deployed in the same cluster. If this doesn't work you're are doing things in a very very non standard way. I get you don't want to use an Ingress and want to access the service on a IP without a port but this doesn't prevent you from using cluster DNS in a standard way. In your case the container has a port of 8787 and the load balancer has an IP of 80.

However, I really like the configuration with config.xml. The reason for this is relatively simple. When the API key is generated in the GUI, it is also saved in the config.xml. This means that exportarr can react to this via fs.Notify and read in the API key again. The option to define the API key via environment variable does not offer this possibility.

Restarting the exportarr pod because of a API key change isn't bad nor inconvenient. I don't get what efficiency you are trying to gain here since exportarr doesn't even implement any sort of fs handlers on the config.xml.

volker-raschek commented 6 months ago

Hi @rtrox and @onedr0p, thanks for the answers.

@volker-raschek - from what it sounds like, you have this set up in a pretty non-standard way -- what you're describing here (using one port mapping for access, and another for internal access) is the perfect usecase for an ingress. If it's helpful, you can see an example using exportarr, nginx-ingress, and external-dns here.

Yes, I am very familiar with how ingress, external-dns and metallb work. All these applications also run in the cluster, but I don't want to use ingress for the deployment of arr applications.

The ingress is given a static IP address by metallb. All HTTP traffic is routed to this IP address via the router. This means that the lidarr application would be accessible via the nginx, when the internal DNS name is know to third party persons. I simply don't want that. After all, the application can also run without a reverse proxy.

You should be using cluster DNS...

I think I have now understood what you mean by this. You mean the DNS name assigned by external-dns to the service of type LoadBalancer of lidarr. Please correct me if I am wrong.

That is a valid point. But why does the HTTP request from exportarr to lidarr have to leave the virtual network of the kubernetes environment to reach lidarr? This makes absolutely no sense to me if I can use the internal DNS name of the LoadBalancer. Finally, kubernetes offers the functionality of direct communication between rolled out applications/services. This option is not supported here because the port of the config.xml overrides the port specification in the CLI flag --url and the port corresponds to the port of the Lidarr application and not the Liadarr service.

Restarting the exportarr pod because of an API key change isn't bad nor inconvenient.

This may be a matter of opinion, but I strongly assume that you would also dislike it if a restart of the cert-manager were necessary if you changed the issuer of an Ingress and the cert-manager needed a restart for this change.

What I'm actually getting at is that it should be up to the user how the exportarr provides the API key of an arr application. Whether by environment variable or config.xml is up to the user. Only latest would make it possible to react to changes in config.xml. This is a very common procedure in kubernetes/microservices, which you could also build on here. I am aware that exportarr does not have this option yet, but it would be easy to implement and automatically supported for each config.xml setup.

Examples of fs.Notify are not only changes to a config file, which usually occur less frequently. However, the scenario that new certificates are generated by the cert-manager occurs much more frequently in the microservice environment. These are mounted in the container file system. The application is able to inform itself via fs.Notify that the certificates have been renewed and restarts the web server process accordingly. This eliminates the need to restart the entire application and does not require any manual interaction.

Volker