search

kouprlabs / voltaserve

⚡️ Cloud Storage for Creators
https://voltaserve.com
Other
248 stars 13 forks source link

[Kubernetes] UI Proxy Error: socket hang up #73

Closed Wtfitsaduck closed 1 month ago

Wtfitsaduck commented 9 months ago

Intro

Hello! I want to first of all say thank you for this project. It looks great and I look forward to its future! I tried standing up this service in a Kubernetes pod and ran into a very weird issue that I'm not quite sure what the cause of it is.

The error

When trying to sign up through the UI, I get the following error in the UI container:

9:41:19 PM [vite] http proxy error at /v1/accounts:
Error: socket hang up
at connResetException (node:internal/errors:721:14)
at Socket.socketOnEnd (node:_http_client:519:23)
at Socket.emit (node:events:526:35)
at endReadableNT (node:internal/streams/readable:1408:12)
at process.processTicksAndRejections (node:internal/process/task_queues:82:21)

I have made absolute sure that the containers are able to communicate with each others and they're reachable through the pod internal network. I was able to curl the API container from the UI container.

voltaserve-79bfbbd857-qpgpl:/app # curl -vvv http://localhost:5000
*   Trying 127.0.0.1:5000...
* Connected to localhost (127.0.0.1) port 5000 (#0)
> GET / HTTP/1.1
> Host: localhost:5000
> User-Agent: curl/8.0.1
> Accept: */*
>
< HTTP/1.1 400 Bad Request
< Date: Thu, 28 Dec 2023 23:43:15 GMT
< Content-Type: text/plain; charset=utf-8
< Content-Length: 24
< Vary: Origin
< Access-Control-Allow-Origin:
<
* Connection #0 to host localhost left intact
Missing or malformed JWT

Here are the raw requests and responses being made (with redactions):

Request:

POST /proxy/idp/v1/accounts HTTP/2
Host: voltaserve.example.com
User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:121.0) Gecko/20100101 Firefox/121.0
Accept: */*
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate, br
Referer: https://voltaserve.example.com/sign-up
Content-Type: application/json
Content-Length: 78
Origin: https://voltaserve.example.com
Sec-Gpc: 1
Sec-Fetch-Dest: empty
Sec-Fetch-Mode: cors
Sec-Fetch-Site: same-origin
Te: trailers

{"fullName":"User","email":"email@example.com","password":"very-real-password"}

Response:

HTTP/2 500 Internal Server Error
Date: Thu, 28 Dec 2023 21:52:56 GMT
Content-Type: text/plain
Access-Control-Allow-Origin: *
Strict-Transport-Security: max-age=31536000; includeSubDomains

Context

I am running everything in kubernetes via ArgoCD. Other containers are able to communicate with each others correctly. This is my ArgoCD helm manifest:

Manifest 
apiVersion: argoproj.io/v1alpha1
kind: Application
metadata:
  name: voltaserve
  namespace: argocd
spec:
  project: default
  source:
    chart: app-template
    repoURL: https://bjw-s.github.io/helm-charts
    targetRevision: 2.4.0
    helm:
      values: |
        # https://github.com/bjw-s/helm-charts/blob/main/charts/library/common/values.yaml
        controllers:
          main:
            annotations:
              secret.reloader.stakater.com/reload: "voltaserve-api-secrets,voltaserve-conversion-secrets,voltaserve-idp-secrets"
            replicas: 1
            containers:
              main:
                nameOverride: ui
                image:
                  repository: gitea.example.com/wtfitsaduck/voltaserve-ui
                  tag: latest@sha256:ea4c36014dc34dce88dd2fa370a23e21da94789ef7f2691c6c442fd4e3ec1c52
                env:
                  API_URL: http://127.0.0.1:5000
                  IDP_URL: http://127.0.0.1:7000
              api:
                image:
                  repository: gitea.perfectra1n.com/wtfitsaduck/voltaserve-api
                  tag: latest
                envFrom:
                  - secretRef:
                      name: voltaserve-api-secrets
              meilisearch:
                image: 
                  repository: getmeili/meilisearch
                  tag: v1.2@sha256:5cfc6195599bdfd3c1cfeaeae97ae72dfc0f47ecc8c718839c0af788edb6c653
              redis:
                image:
                  repository: redis
                  tag: 7.0
                env:
                  REDIS_REPLICATION_MODE: master
              mailhog:
                image:
                  repository: mailhog/mailhog
                  tag: v1.0.1
              idp:
                image:
                  repository: gitea.example.com/wtfitsaduck/voltaserve-idp
                  tag: latest@sha256:4d3d8dbdbc1128a0cf12f001382d29c8b08aabeb5c8bb6e48ede0ef9b25468bf
                envFrom:
                  - secretRef:
                      name: voltaserve-idp-secrets
              webdav:
                image:
                  repository: gitea.example.com/wtfitsaduck/voltaserve-webdav
                  tag: latest
                env:
                  API_URL: http://127.0.0.1:5000
                  IDP_URL: http://127.0.0.1:7000
              conversion:
                image:
                  repository: gitea.example.com/wtfitsaduck/voltaserve-conversion
                  tag: latest
                envFrom:
                  - secretRef:
                      name: voltaserve-conversion-secrets
        service:
          main:
            enabled: true
            ports:
              http:
                # What cluster is listening on
                port: 3000
                # What port the container is listening on
                targetPort: 3000
        persistence:
          meilisearch:
            enabled: true
            existingClaim: voltaserve-meilisearch-pvc
            advancedMounts:
              main:
                meilisearch:
                  - path: /meili_data
  destination:
    server: "https://kubernetes.default.svc"
    namespace: voltaserve
  syncPolicy:
    syncOptions:
    - CreateNamespace=true
    automated:
      prune: true
      selfHeal: true

I used the following environment variables, with redactions of course:

Secrets 
apiVersion: v1
kind: Secret
metadata:
  name: voltaserve-api-secrets
  namespace: voltaserve
type: Opaque
stringData:
  PORT: "5000"
  VOLTASERVE_HOSTNAME: voltaserve.example.com
  VOLTASERVE_SMTP_HOST: smtppro.zoho.com
  VOLTASERVE_SMTP_PORT: "465"
  VOLTASERVE_SMTP_SECURE: "true"
  VOLTASERVE_SMTP_USERNAME: noreply@example.com
  VOLTASERVE_SMTP_PASSWORD: another-real-password
  VOLTASERVE_SMTP_SENDER_ADDRESS: noreply@example.com
  VOLTASERVE_SMTP_SENDER_NAME: Voltaserve
  CONVERSION_URL: http://localhost:5001"
  POSTGRES_URL: postgresql://voltaserve:epic-password@homelab-cluster-16-rw.databases.svc.cluster.local:5432/voltaserve
  S3_URL: minio.minio.svc.cluster.local:6379
  S3_ACCESS_KEY: voltaserve
  S3_SECRET_KEY: its-something
  S3_REGION: us-west-2
  SEARCH_URL: http://localhost:7700
  PUBLIC_UI_URL: https://voltaserve.example.com
  REDIS_ADDRESS: localhost:6379
---
apiVersion: v1
kind: Secret
metadata:
  name: voltaserve-idp-secrets
  namespace: voltaserve
type: Opaque
stringData:
  PORT: "7000"
  POSTGRES_URL: postgresql://voltaserve:epic-password@homelab-cluster-16-rw.databases.svc.cluster.local:5432/voltaserve
  VOLTASERVE_SMTP_HOST: smtppro.zoho.com
  VOLTASERVE_SMTP_PORT: "465"
  VOLTASERVE_SMTP_SECURE: "true"
  VOLTASERVE_SMTP_USERNAME: noreply@example.com
  VOLTASERVE_SMTP_PASSWORD: another-real-password
  VOLTASERVE_SMTP_SENDER_ADDRESS: noreply@example.com
  VOLTASERVE_SMTP_SENDER_NAME: voltaserve
  SEARCH_URL: http://localhost:7700
  PUBLIC_UI_URL: https://voltaserve.example.com
---
apiVersion: v1
kind: Secret
metadata:
  name: voltaserve-conversion-secrets
  namespace: voltaserve
type: Opaque
stringData:
  PORT: "5001"
  API_URL: http://127.0.0.1:7000
  S3_URL: minio.minio.svc.cluster.local:6379
  S3_ACCESS_KEY: voltaserve
  S3_SECRET_KEY: its-something
  S3_REGION: us-west-2

Please let me know if you need any more details or if I'm just doing something dumb (very likely lol!) I could just be missing something obvious.

bouassaba commented 7 months ago

@Wtfitsaduck Thank you for the detailed description, I will investigate this issue and come back to you shortly in the next few days.

bouassaba commented 7 months ago

@Wtfitsaduck I see that postgres and minio are missing in your Manifest. Are they available?

Wtfitsaduck commented 7 months ago

@Wtfitsaduck I see that postgres and minio are missing in your Manifest. Are they available?

Yup! Those are running in the cluster separate of this manifest.. I have Minio at minio.minio.svc.cluster.local:6379 and postgres at homelab-cluster-16-rw.databases.svc.cluster.local:5432. Both are known good and are in use. Minio is version RELEASE.2023-09-07T02-05-02Z and Postgres is on version 16.1 with a custom image ghcr.io/bo0tzz/cnpgvecto.rs:16.1-v0.1.11

bouassaba commented 2 months ago

@Wtfitsaduck we just released v2 with a lot of fixes and container improvements, could you please try and let me know if the issue still persists within your Kubernetes deployment?

Wtfitsaduck commented 2 months ago

You're not kidding you did change a lot! I went ahead and rewrote my deployment and was not able to recreate this same bug. I'm still having issues with the sign up process though, the sign up confirmation email is not being sent for some reason. I'm not seeing any logs in stdout/stderr. Are there logs any where else i could use to troubleshoot? I went ahead and double checked my SMTP settings and they all look correct. I've used the same SMTP set up in other applications as well:

SMTP_HOST: smtppro.zoho.com
SMTP_PORT: "465"
SMTP_SECURE: "true"
SMTP_USERNAME: noreply@domain.com
SMTP_PASSWORD: realpassword
SMTP_SENDER_ADDRESS: noreply@domain.com
SMTP_SENDER_NAME: Voltaserve

Would you like me to add more details in this issue or would you like me to start a new one?

bouassaba commented 2 months ago

@Wtfitsaduck I just did a fix for the email sending issue tonight, first do a git pull then either rebuild all your images from the source, or get the latest ones from Docker Hub with docker compose pull, then delete all existing containers and volumes, and start everything new.

Wtfitsaduck commented 2 months ago

I went ahead and pulled the latest images and I'm still seeing the same behavior. Is there a debug mode or any debug logs I can look at?

bouassaba commented 2 months ago

In this case I need to run the Voltaserve stack in minikube and debug it, let me work on this, I will add this to the next update v2.1 because I believe strong Kubernetes support is crucial at this point. Once I do some progress on this, I will get back to you so you could test one more time on your environment. We stay in touch!

Wtfitsaduck commented 2 months ago

Sounds good! let me know if you need any of my deployment manifests or any help

dsonck92 commented 2 months ago

You're not kidding you did change a lot! I went ahead and rewrote my deployment and was not able to recreate this same bug. I'm still having issues with the sign up process though, the sign up confirmation email is not being sent for some reason. I'm not seeing any logs in stdout/stderr. Are there logs any where else i could use to troubleshoot? I went ahead and double checked my SMTP settings and they all look correct. I've used the same SMTP set up in other applications as well:

SMTP_HOST: smtppro.zoho.com
SMTP_PORT: "465"
SMTP_SECURE: "true"
SMTP_USERNAME: noreply@domain.com
SMTP_PASSWORD: realpassword
SMTP_SENDER_ADDRESS: noreply@domain.com
SMTP_SENDER_NAME: Voltaserve

Would you like me to add more details in this issue or would you like me to start a new one?

I actually had a similar issue and you might want to check if your mail server is set-up to use TLS, or STARTTLS. If it is STARTTLS you want to have SMTP_SECURE: "false" because otherwise it will have an SSL handshake error. You can check the logs of the IDP component to see if this is the case.

Besides that, I would recommend not running all microservices in one pod, as I feel you can scale each microservice separately. I will probably donate my helm chart once I feel it is up to my quality standards, but for now you can look at my embedded voltaserve helm chart here.

Wtfitsaduck commented 2 months ago

You're not kidding you did change a lot! I went ahead and rewrote my deployment and was not able to recreate this same bug. I'm still having issues with the sign up process though, the sign up confirmation email is not being sent for some reason. I'm not seeing any logs in stdout/stderr. Are there logs any where else i could use to troubleshoot? I went ahead and double checked my SMTP settings and they all look correct. I've used the same SMTP set up in other applications as well:

SMTP_HOST: smtppro.zoho.com
SMTP_PORT: "465"
SMTP_SECURE: "true"
SMTP_USERNAME: noreply@domain.com
SMTP_PASSWORD: realpassword
SMTP_SENDER_ADDRESS: noreply@domain.com
SMTP_SENDER_NAME: Voltaserve

Would you like me to add more details in this issue or would you like me to start a new one?

I actually had a similar issue and you might want to check if your mail server is set-up to use TLS, or STARTTLS. If it is STARTTLS you want to have SMTP_SECURE: "false" because otherwise it will have an SSL handshake error. You can check the logs of the IDP component to see if this is the case.

Besides that, I would recommend not running all microservices in one pod, as I feel you can scale each microservice separately. I will probably donate my helm chart once I feel it is up to my quality standards, but for now you can look at my embedded voltaserve helm chart here.

Zoho does use TLS so that's not it :/ I did try without TLS just in case and got the same behavior. Not seeing logs or errors in any of the containers either way.

dsonck92 commented 2 months ago

Strange, yes, in my case the errors were really clear in IDP where it was complaining about SSL. So maybe double check if the call is indeed reaching the API server. I have noticed that the proxy just doesn't report errors with connectivity, which caused me to chase a little typo for over a day :sweat_smile:

bouassaba commented 2 months ago

Our team is working on an excellent Helm charts implementation: https://github.com/kouprlabs/voltaserve-helm Feel free to try it and give us some feedback ;)