novuhq / novu

Open-Source Notification Platform. Embeddable Notification Center, E-mail, Push and Slack Integrations.
https://novu.co
Other
34.01k stars 3.47k forks source link

šŸ› Bug Report: Novu Application getting Network Error while getting singup #3602

Open HanumanthaRAON opened 1 year ago

HanumanthaRAON commented 1 year ago

šŸ“œ Description

We have deployed novu in Kubernetes environment with provided kustomize deployments files in Novu Documentation.

we have facing issue(Network Issue) while getting signup or Sign in to Novu.

We have using ingress AWS ALB to expose novu application to public.

my env-dev file is šŸ‘

Secrets

YOU MUST CHANGE THESE BEFORE GOING INTO PRODUCTION

JWT_SECRET=your-secret STORE_ENCRYPTION_KEY=ekwUKf9yLjGPLOz939Y1GM0nJckVoVyF

General

NODE_ENV=production MONGO_URL=XXXXX/novu-db MONGO_MAX_POOL_SIZE=500 REDIS_HOST=novu-redis-dev

REDIS_CACHE_SERVICE_HOST=novu-redis-dev

REDIS_CACHE_SERVICE_PORT=6379

AWS

S3_LOCAL_STACK=http://localhost:4566

S3_BUCKET_NAME=XXXXX S3_REGION=XXXX AWS_ACCESS_KEY_ID=XXXX AWS_SECRET_ACCESS_KEY=XXXX

Ports

API_PORT=3000 REDIS_PORT=6379 WS_PORT=3002

Root URL

REACT_APP_API_URL=https://novu-api-dev:3000 REACT_APP_WS_URL=https://novu-ws-dev:3002 API_ROOT_URL=https://novu-api-dev:3000 DISABLE_USER_REGISTRATION=false FRONT_BASE_URL=https://novu-web-dev:4200 WIDGET_EMBED_PATH=https://novu-embed-dev:4701/embed.umd.min.js WIDGET_URL=https://novu-widget-dev:4500

Context Paths

Only needed for setups with reverse-proxies

GLOBAL_CONTEXT_PATH= WEB_CONTEXT_PATH= API_CONTEXT_PATH= WS_CONTEXT_PATH= WIDGET_CONTEXT_PATH=

We are applying the deployments with below command

kubectl apply -f kustomize/

all pods are running fine without any error.

NAME READY STATUS RESTARTS AGE novu-api-dev-77555bd5b5-tnrx8 1/1 Running 0 15m novu-embed-dev-55c5dd565f-lhws6 1/1 Running 0 15m novu-mongodb-dev-666b94cff7-xblfr 1/1 Running 0 15m novu-redis-dev-86697d5b54-97kch 1/1 Running 0 15m novu-web-dev-6b5848996d-4fnqp 1/1 Running 0 15m novu-widget-dev-6c9ccd79f-bhxw5 1/1 Running 0 15m novu-worker-dev-56b9cf7c4f-6scqs 1/1 Running 0 15m novu-ws-dev-5465fd96d7-q92tp 1/1 Running 0 15m

Below is AWS ingress LB config:

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations:

alb.ingress.kubernetes.io/certificate-arn: arn:aws:acm:us-east-1:XXXXXX
alb.ingress.kubernetes.io/group.name: XXXXXXXXXX
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/subnets: subnet-09d9c9ef0fe16b58c,subnet-0b83921c80fdf09dc,subnet-031e5a6e26a1a06e4
alb.ingress.kubernetes.io/target-type: ip
  #alb.ingress.kubernetes.io/enable-cors: "true"
  #alb.ingress.kubernetes.io/cors-allow-origin: "*"
kubernetes.io/ingress.class: alb

name: novu-alb namespace: novu spec: rules:

šŸ‘Ÿ Reproduction steps

my env-dev file is šŸ‘

Secrets

YOU MUST CHANGE THESE BEFORE GOING INTO PRODUCTION

JWT_SECRET=your-secret STORE_ENCRYPTION_KEY=ekwUKf9yLjGPLOz939Y1GM0nJckVoVyF

General

NODE_ENV=production MONGO_URL=XXXXX/novu-db MONGO_MAX_POOL_SIZE=500 REDIS_HOST=novu-redis-dev

REDIS_CACHE_SERVICE_HOST=novu-redis-dev

REDIS_CACHE_SERVICE_PORT=6379

AWS

S3_LOCAL_STACK=http://localhost:4566

S3_BUCKET_NAME=XXXXX S3_REGION=XXXX AWS_ACCESS_KEY_ID=XXXX AWS_SECRET_ACCESS_KEY=XXXX

Ports

API_PORT=3000 REDIS_PORT=6379 WS_PORT=3002

Root URL

REACT_APP_API_URL=https://novu-api-dev:3000 REACT_APP_WS_URL=https://novu-ws-dev:3002 API_ROOT_URL=https://novu-api-dev:3000 DISABLE_USER_REGISTRATION=false FRONT_BASE_URL=https://novu-web-dev:4200 WIDGET_EMBED_PATH=https://novu-embed-dev:4701/embed.umd.min.js WIDGET_URL=https://novu-widget-dev:4500

Context Paths

Only needed for setups with reverse-proxies

GLOBAL_CONTEXT_PATH= WEB_CONTEXT_PATH= API_CONTEXT_PATH= WS_CONTEXT_PATH= WIDGET_CONTEXT_PATH=

We are applying the deployments with below command

kubectl apply -f kustomize/

all pods are running fine without any error.

NAME READY STATUS RESTARTS AGE novu-api-dev-77555bd5b5-tnrx8 1/1 Running 0 15m novu-embed-dev-55c5dd565f-lhws6 1/1 Running 0 15m novu-mongodb-dev-666b94cff7-xblfr 1/1 Running 0 15m novu-redis-dev-86697d5b54-97kch 1/1 Running 0 15m novu-web-dev-6b5848996d-4fnqp 1/1 Running 0 15m novu-widget-dev-6c9ccd79f-bhxw5 1/1 Running 0 15m novu-worker-dev-56b9cf7c4f-6scqs 1/1 Running 0 15m novu-ws-dev-5465fd96d7-q92tp 1/1 Running 0 15m

šŸ‘ Expected behavior

We are seeing CORS issue with our env.

We are also seeing name not resolved issue sometime.

šŸ‘Ž Actual Behavior with Screenshots

MicrosoftTeams-image (1)MicrosoftTeams-image

šŸ“ƒ Provide any additional context for the Bug.

No response

šŸ‘€ Have you spent some time to check if this bug has been raised before?

šŸ¢ Have you read the Contributing Guidelines?

Are you willing to submit PR?

None

p-fernandez commented 1 year ago

@HanumanthaRAON šŸ‘‹šŸ» What version of Novu are you using? As a starting guess try commenting REACT_APP_API_URL as it is not being used in the deployments. Only for the local environment. Reference of other users with similar problem (though with a NGINX reverse proxy): https://discord.com/channels/895029566685462578/1088670290357264384/1088713207960571965

HanumanthaRAON commented 1 year ago

We are using novu 0.15.0 version. do we need rebuild the yaml files when we change values in env-dev in kustomize directory.

commented out REACT_APP_API_URL section but no luck still getting error.

when try to signup it is asking for details like username , email and password .

I am able to enter username and email and it is getting hung after when try to enter password filed not sure what is going with deployments.

HanumanthaRAON commented 1 year ago

Below are the logs for all deployments :

API

[ec2-user@ip- kubernetes]$ kubectl logs -n novu novu-api-dev-58db49cf88-r5rcg 2023-06-15T16:41:25: PM2 log: Launching in no daemon mode 2023-06-15T16:41:25: PM2 log: App [main:0] starting in -fork mode- 2023-06-15T16:41:25: PM2 log: App [main:0] online New Relic for Node.js was unable to bootstrap itself due to an error: Error: New Relic requires that you name this application! Set app_name in your newrelic.js or newrelic.cjs file or set environment variable NEW_RELIC_APP_NAME. Not starting! at createAgent (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:141:11) at initialize (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:82:15) at Object. (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:37:3) at Module._compile (node:internal/modules/cjs/loader:1196:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10) at Module.load (node:internal/modules/cjs/loader:1074:32) at Function.Module._load (node:internal/modules/cjs/loader:909:12) at Module.require (node:internal/modules/cjs/loader:1098:19) at Module.Hook._require.Module.require (/usr/local/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:101:39) at require (node:internal/modules/cjs/helpers:108:18) Environment: production Platform: Docker Tenant: OS

WEB

[ec2-user@ip- kubernetes]$ kubectl logs -n novu novu-web-dev-5777659685-8jvnm

@novu/web@0.15.0 start:static:build /app pnpm envsetup:docker && http-server build -p 4200 --proxy http://localhost:4200?

@novu/web@0.15.0 envsetup:docker /app chmod +x ./env.sh && ./env.sh && mv ./env-config.js ./build/env-config.js

Starting up http-server, serving build

http-server version: 14.1.1

http-server settings: CORS: disabled Cache: 3600 seconds Connection Timeout: 120 seconds Directory Listings: visible AutoIndex: visible Serve GZIP Files: false Serve Brotli Files: false Default File Extension: none

Available on: http://127.0.0.1:4200 http://X.X.X.X:4200 Unhandled requests will be served from: http://localhost:4200? Hit CTRL-C to stop the server

[2023-06-15T16:41:38.235Z] "GET /" "ELB-HealthChecker/2.0" (node:18) [DEP0066] DeprecationWarning: OutgoingMessage.prototype._headers is deprecated (Use node --trace-deprecation ... to show where the warning was created)

WS

[ec2-user@ip- kubernetes]$ kubectl logs -n novu novu-ws-dev-6889959666-6cftf 2023-06-15T16:41:25: PM2 log: Launching in no daemon mode 2023-06-15T16:41:26: PM2 log: App [main:0] starting in -fork mode- 2023-06-15T16:41:26: PM2 log: App [main:0] online New Relic for Node.js was unable to bootstrap itself due to an error: Error: New Relic requires that you name this application! Set app_name in your newrelic.js or newrelic.cjs file or set environment variable NEW_RELIC_APP_NAME. Not starting! at createAgent (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:141:11) at initialize (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:82:15) at Object. (/usr/src/app/node_modules/.pnpm/newrelic@9.10.2/node_modules/newrelic/index.js:37:3) at Module._compile (node:internal/modules/cjs/loader:1196:14) at Object.Module._extensions..js (node:internal/modules/cjs/loader:1250:10) at Module.load (node:internal/modules/cjs/loader:1074:32) at Function.Module._load (node:internal/modules/cjs/loader:909:12) at Module.require (node:internal/modules/cjs/loader:1098:19) at Module.Hook._require.Module.require (/usr/local/lib/node_modules/pm2/node_modules/require-in-the-middle/index.js:101:39) at require (node:internal/modules/cjs/helpers:108:18) Environment: production Platform: Docker Tenant: OS Selected Log Transport None [Nest] 17 - 06/15/2023, 4:41:35 PM LOG Creating queue ws_socket_queue bullmq pro is Disabled (node:17) [MONGOOSE] DeprecationWarning: Mongoose: the strictQuery option will be switched back to false by default in Mongoose 7. Use mongoose.set('strictQuery', false); if you want to prepare for this change. Or use mongoose.set('strictQuery', true); to suppress this warning. (Use node --trace-deprecation ... to show where the warning was created) {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"NestFactory","msg":"Starting Nest application..."} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"JwtModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"AppModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"LoggerModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"TerminusModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"SocketModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"SharedModule dependencies initialized"} {"level":30,"time":1686847298102,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"InstanceLoader","msg":"HealthModule dependencies initialized"} {"level":30,"time":1686847300297,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"RoutesResolver","msg":"AppController {/}:"} {"level":30,"time":1686847300301,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"RouterExplorer","msg":"Mapped {/, GET} route"} {"level":30,"time":1686847300302,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"RoutesResolver","msg":"HealthController {/v1/health-check}:"} {"level":30,"time":1686847300302,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"RouterExplorer","msg":"Mapped {/v1/health-check, GET} route"} {"level":30,"time":1686847300316,"pid":17,"serviceName":"@novu/ws","serviceVersion":"0.15.0","platform":"Docker","tenant":"OS","context":"NestApplication","msg":"Nest application successfully started"}

HanumanthaRAON commented 1 year ago

novu

We are also seeing CORS error and on the top of the browser showing some scritpt is running that is slowing page and getting hung

HanumanthaRAON commented 1 year ago

Updated env file

[ec2-user@ip-XXXX kustomize]$ cat env-dev

Secrets

YOU MUST CHANGE THESE BEFORE GOING INTO PRODUCTION

JWT_SECRET=your-secret STORE_ENCRYPTION_KEY=ekwUKf9yLjGPLOz939Y1GM0nJckVoVyF

General

NODE_ENV=production MONGO_URL=XXXXXXXXXXXXX MONGO_MAX_POOL_SIZE=500 REDIS_HOST=XXXXXXXXXXXXXXXX REDIS_CACHE_SERVICE_HOST=XXXXXXXXXXXX REDIS_CACHE_SERVICE_PORT=6379

AWS

S3_LOCAL_STACK=http://localhost:4566

S3_BUCKET_NAME=XXXXXXXX S3_REGION=us-east-1 AWS_ACCESS_KEY_ID=XXXXXXXX AWS_SECRET_ACCESS_KEY=XXXXXXXXXXX

Ports

API_PORT=3000 REDIS_PORT=6379 WS_PORT=3002

Root URL

REACT_APP_API_URL=https://novu-api-dev:3000

REACT_APP_WS_URL=http://novu-ws-dev:3002 API_ROOT_URL=http://novu-api-dev:3000 DISABLE_USER_REGISTRATION=false FRONT_BASE_URL=http://novu-app-dev:4200 WIDGET_EMBED_PATH=http://novu-widget-dev:4701/embed.umd.min.js WIDGET_URL=http://localhost:4500

Context Paths

Only needed for setups with reverse-proxies

GLOBAL_CONTEXT_PATH= WEB_CONTEXT_PATH= API_CONTEXT_PATH= WS_CONTEXT_PATH= WIDGET_CONTEXT_PATH=

HanumanthaRAON commented 1 year ago

Below is Ingress controller file

[ec2-user@ip- kustomize]$ kubectl describe ing -n novu Name: novu-alb Namespace: novu Address: XXXXXXXXXXXXXXXXXXXX Default backend: default-http-backend:80 (<error: endpoints "default-http-backend" not found>) Rules: Host Path Backends


novu-dev.com / novu-web-dev:4200 (172.20.0.240:4200) / novu-api-dev:3000 (172.20.1.115:3000) Annotations: alb.ingress.kubernetes.io/certificate-arn: XXXXXXXXXXXXXXX alb.ingress.kubernetes.io/group.name: XXXXXXXXXXXXX alb.ingress.kubernetes.io/listen-ports: [{"HTTP": 80}, {"HTTPS":443}] alb.ingress.kubernetes.io/scheme: internet-facing alb.ingress.kubernetes.io/ssl-redirect: 443 alb.ingress.kubernetes.io/subnets: subnet-09d9c9ef0fe16b58c,subnet-0b83921c80fdf09dc,subnet-031e5a6e26a1a06e4 alb.ingress.kubernetes.io/target-type: ip kubernetes.io/ingress.class: alb Events: Type Reason Age From Message


Normal SuccessfullyReconciled 41m (x32 over 21h) ingress Successfully reconciled [ec2-user@ip kustomize]$

wh1337 commented 1 year ago

Can confirm, getting the same issue.

edit: Changed the environment variables to see if a different NODE_ENV would allow it. It does not and you still get the CORS policy error.

edit 2: Tested this with v0.14.0 and it's doing the same thing. Here is a dump of the redacted task definition:

{
  "family": "novu",
  "containerDefinitions": [
    {
      "name": "novu-api",
      "image": "ghcr.io/novuhq/novu/api:0.14.0",
      "cpu": 0,
      "portMappings": [
        {
          "name": "novu-api-3000-tcp",
          "containerPort": 3000,
          "hostPort": 3000,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "REDIS_HOST",
          "value": "redacted"
        },
        {
          "name": "MONGO_URL",
          "value": "redacted"
        },
        {
          "name": "PORT",
          "value": "3002"
        },
        {
          "name": "REDIS_PORT",
          "value": "6379"
        },
        {
          "name": "S3_REGION",
          "value": "us-east-1"
        },
        {
          "name": "JWT_SECRET",
          "value": "redacted"
        },
        {
          "name": "AWS_ACCESS_KEY_ID",
          "value": "redacted"
        },
        {
          "name": "AWS_SECRET_ACCESS_KEY",
          "value": "redacted"
        },
        {
          "name": "STORE_ENCRYPTION_KEY",
          "value": "redacted"
        },
        {
          "name": "NODE_ENV",
          "value": "test"
        },
        {
          "name": "S3_BUCKET_NAME",
          "value": "novu"
        },
        {
          "name": "FRONT_BASE_URL",
          "value": "https://web-novu.internal-domain.com/"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/novu",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    },
    {
      "name": "novu-worker",
      "image": "ghcr.io/novuhq/novu/worker:0.14.0",
      "cpu": 0,
      "portMappings": [],
      "essential": true,
      "environment": [
        {
          "name": "REDIS_HOST",
          "value": "redacted"
        },
        {
          "name": "MONGO_URL",
          "value": "redacted"
        },
        {
          "name": "PORT",
          "value": "3002"
        },
        {
          "name": "REDIS_PORT",
          "value": "6379"
        },
        {
          "name": "S3_REGION",
          "value": "us-east-1"
        },
        {
          "name": "JWT_SECRET",
          "value": "redacted"
        },
        {
          "name": "AWS_ACCESS_KEY_ID",
          "value": "redacted"
        },
        {
          "name": "AWS_SECRET_ACCESS_KEY",
          "value": "redacted"
        },
        {
          "name": "STORE_ENCRYPTION_KEY",
          "value": "redacted"
        },
        {
          "name": "NODE_ENV",
          "value": "test"
        },
        {
          "name": "S3_BUCKET_NAME",
          "value": "novu"
        },
        {
          "name": "FRONT_BASE_URL",
          "value": "https://web-novu.internal-domain.com/"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/novu",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    },
    {
      "name": "novu-ws",
      "image": "ghcr.io/novuhq/novu/ws:0.14.0",
      "cpu": 0,
      "portMappings": [
        {
          "name": "novu-ws-3002-tcp",
          "containerPort": 3002,
          "hostPort": 3002,
          "protocol": "tcp",
          "appProtocol": "http"
        }
      ],
      "essential": true,
      "environment": [
        {
          "name": "REDIS_HOST",
          "value": "redacted"
        },
        {
          "name": "MONGO_URL",
          "value": "redacted"
        },
        {
          "name": "PORT",
          "value": "3002"
        },
        {
          "name": "REDIS_PORT",
          "value": "6379"
        },
        {
          "name": "S3_REGION",
          "value": "us-east-1"
        },
        {
          "name": "JWT_SECRET",
          "value": "redacted"
        },
        {
          "name": "AWS_ACCESS_KEY_ID",
          "value": "redacted"
        },
        {
          "name": "AWS_SECRET_ACCESS_KEY",
          "value": "redacted"
        },
        {
          "name": "STORE_ENCRYPTION_KEY",
          "value": "redacted"
        },
        {
          "name": "NODE_ENV",
          "value": "test"
        },
        {
          "name": "S3_BUCKET_NAME",
          "value": "novu"
        },
        {
          "name": "FRONT_BASE_URL",
          "value": "https://web-novu.internal-domain.com/"
        }
      ],
      "mountPoints": [],
      "volumesFrom": [],
      "logConfiguration": {
        "logDriver": "awslogs",
        "options": {
          "awslogs-create-group": "true",
          "awslogs-group": "/ecs/novu",
          "awslogs-region": "us-east-1",
          "awslogs-stream-prefix": "ecs"
        }
      }
    }
  ],
  "executionRoleArn": "redacted",
  "networkMode": "awsvpc",
  "requiresCompatibilities": [
    "FARGATE"
  ],
  "cpu": "2048",
  "memory": "4096",
  "ephemeralStorage": {
    "sizeInGiB": 21
  },
  "runtimePlatform": {
    "cpuArchitecture": "X86_64",
    "operatingSystemFamily": "LINUX"
  }
}
HanumanthaRAON commented 1 year ago

Can someone help us to resolve the issue, our testing is blocked due this

wh1337 commented 1 year ago

Echoing with what @HanumanthaRAON stated. Atleast point us in the correct direction for a fix. Thanks!

p-fernandez commented 1 year ago

@wh1337 in the task definition you are sharing all the services have this value set for them all as environment variables:

   {
          "name": "PORT",
          "value": "3002"
        },

Not sure who or what generates that task definition, but from its output the first thing I would suggest you to check is that every service is deployed and exposed in the port set in the Docker/k8s configuration files in case something has manually identified. If nothing has been modified manually we chosen the defaults for the following ports: API -> 3000 WEB -> 4200 Worker -> 3004 WS -> 3002

Then as following step to that, to double check the network is open from where the Web app is deployed to the API app and that any network infrastructure has any kind of generic disallowing list that avoids the access of the Web app to the API app.

HanumanthaRAON commented 1 year ago

Hi ,

Below are our services in Kubernetes env.

[ec2-user@ip-XXXXX ~]$ kubectl get svc -n novu NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE novu-api-dev ClusterIP 3000/TCP 4d novu-embed-dev ClusterIP 4701/TCP 4d novu-mongodb-dev ClusterIP 27017/TCP 4d novu-redis-dev ClusterIP 6379/TCP 4d novu-web-dev ClusterIP 4200/TCP 4d novu-widget-dev ClusterIP 4500/TCP 4d novu-worker-dev ClusterIP 3004/TCP 4d novu-ws-dev ClusterIP 3002/TCP 4d [ec2-user@ip-XXXX ~]$

p-fernandez commented 1 year ago

@HanumanthaRAON Your Web app seems to be located in novu-web-dev but in your env variables files you set the FRONT_BASE_URL to http://novu-app-dev:4200/. So at first sight that seems to be a mismatch.

ElieAbboudMsi commented 1 year ago

Any update on this? I am facing the same issue.

I checked on Discord and the team said it should work on Novu version 0.13.

However, when i do that. the api url is always set to localhost:3000.

I set my Root URLS to in the env-dev file:

# Root URL REACT_APP_API_URL=http://notificationapi.test.co.uk:3000 REACT_APP_WS_URL=http://notification.test.co.uk:3002 API_ROOT_URL=http://notificationapi.test.co.uk:3000 DISABLE_USER_REGISTRATION=false FRONT_BASE_URL=http://notification.test.co.uk:4200 WIDGET_EMBED_PATH=http://notification.test.co.uk:4701/embed.umd.min.js WIDGET_URL=http://notification.test.co.uk:4500

However it still calls localhost:3000/auth/login.

Any ideas please?

sushmasabbani commented 1 year ago

Hi Team, Even I am also facing the same issue. I am running novu version of 0.15.0.

Please find the required details below

Pods are running fine NAME READY STATUS RESTARTS AGE novu-api-dev-54b4f74846-r8zq2 1/1 Running 0 179m novu-embed-dev-86ccb78496-k69hf 1/1 Running 0 179m novu-mongodb-dev-854bbd686-nxfdv 1/1 Running 0 179m novu-redis-dev-86697d5b54-589r4 1/1 Running 0 179m novu-web-dev-668bc5479f-fc22g 1/1 Running 0 179m novu-widget-dev-f89679f6-wjhn6 1/1 Running 0 179m novu-worker-dev-58f4f9b9c8-8wkms 1/1 Running 0 179m novu-ws-dev-856cc7d55b-7m9tj 1/1 Running 0 179m

Below are the services:

novu-api-dev ClusterIP 10.100.195.134 3000/TCP 3h2m novu-embed-dev ClusterIP 10.100.87.43 4701/TCP 3h2m novu-mongodb-dev ClusterIP 10.100.66.201 27017/TCP 3h2m novu-redis-dev ClusterIP 10.100.236.214 6379/TCP 3h2m novu-web-dev ClusterIP 10.100.59.64 4200/TCP 3h2m novu-widget-dev ClusterIP 10.100.98.67 4500/TCP 3h2m novu-worker-dev ClusterIP 10.100.181.13 3004/TCP 3h2m novu-ws-dev ClusterIP 10.100.206.136 3002/TCP 3h2m

And service endpoints also resolving properly

[ec2-user@ip-172-20-3-216 kustomize]$ kubectl get ep -n novu NAME ENDPOINTS AGE novu-api-dev 172.20.1.241:3000 3h2m novu-embed-dev 172.20.1.85:4701 3h2m novu-mongodb-dev 172.20.1.5:27017 3h2m novu-redis-dev 172.20.0.48:6379 3h2m novu-web-dev 172.20.0.70:4200 3h2m novu-widget-dev 172.20.1.203:4500 3h2m novu-worker-dev 172.20.1.243:3004 3h2m novu-ws-dev 172.20.1.178:3002 3h2m

Below is my kubernetes ingress file. apiVersion: networking.k8s.io/v1 kind: Ingress metadata: annotations:

alb.ingress.kubernetes.io/certificate-arn: ***

alb.ingress.kubernetes.io/certificate-arn: ******
alb.ingress.kubernetes.io/group.name: ***************
alb.ingress.kubernetes.io/scheme: internet-facing
alb.ingress.kubernetes.io/listen-ports: '[{"HTTP": 80}, {"HTTPS":443}]'
alb.ingress.kubernetes.io/ssl-redirect: '443'
alb.ingress.kubernetes.io/subnets: ******************
alb.ingress.kubernetes.io/target-type: ip
kubernetes.io/ingress.class: alb

name: novu-alb namespace: novu spec: rules:

And in configuration file, found something like

Context Paths

Only needed for setups with reverse-proxies

GLOBAL_CONTEXT_PATH= WEB_CONTEXT_PATH= API_CONTEXT_PATH= WS_CONTEXT_PATH= WIDGET_CONTEXT_PATH=

What should be the values given here, as we are using application loadbalancer ?

However it is still calling localhost:3000/auth/login image

HanumanthaRAON commented 1 year ago

Can someone look into the issue and help us resolve the issue.

As we are using AWS ALB do we need update reverse-proxies setting , please let us know if we need to do so

we are waiting for inputs as our development team is waiting as we are unable to host the app properly

wh1337 commented 1 year ago

@HanumanthaRAON,

Are you hosting with Kubernetes on AWS or are you using ECS?

HanumanthaRAON commented 1 year ago

Hi @wh1337 ,

We are using AWS EKS to host the novu application.

HanumanthaRAON commented 1 year ago

HI @wh1337 , do you have any update for us , we are waiting for your update

wh1337 commented 1 year ago

HI @wh1337 , do you have any update for us , we are waiting for your update

Sadly I do not. We do not use EKS in our environment so I cannot speak in how to get Novu to function with EKS.

We do however, have ours running in ECS.

HanumanthaRAON commented 1 year ago

Hi @wh1337 ,

We have also deployed Novu application in AWS EC2 server with local deployment with ALB configured but still we are facing the issue with CORS

wh1337 commented 1 year ago

Hey @HanumanthaRAON,

Looking at your notes from earlier:

REACT_APP_API_URL=https://novu-api-dev:3000/ REACT_APP_WS_URL=http://novu-ws-dev:3002/ API_ROOT_URL=http://novu-api-dev:3000/ DISABLE_USER_REGISTRATION=false FRONT_BASE_URL=http://novu-app-dev:4200/ WIDGET_EMBED_PATH=http://novu-widget-dev:4701/embed.umd.min.js WIDGET_URL=http://localhost:4500/

This may be your issue. Your FRONT_BASE_URL should be set to your FQDN. For example, if you access Novu from https://web.novu.mydomain.com, you would want to set that as your FRONT_BASE_URL.

HanumanthaRAON commented 1 year ago

@wh1337 , we have already tried with that option as well but no luck.

wh1337 commented 1 year ago

@HanumanthaRAON I'm not too familiar with how EKS works. Are you getting any errors when your containers are starting?

@p-fernandez I think we can move this to discussions vs being an issue

HanumanthaRAON commented 1 year ago

@wh1337 all containers are running fine don't see any errors .

HanumanthaRAON commented 1 year ago

We have deployed novu on EC2 using docker-compose with local deployment but still facing issues.

While getting signup web browser hangs and showing browser trying to some script want stop or debug .

sometimes it works fine with details and when we submit it says CORS error.

Please let me know if we can connect and resolve the issue

hilmia commented 1 year ago

Any suggestions to solve this? I'm also getting "Network Error" after deploying to cloud.. Works fine in local

HanumanthaRAON commented 1 year ago

Hi Team,

Can someone help us here as we are blocked using Novu platform.

I have taken AWS EC2 instance and deployed using Docker-compose and still getting CORS error.

scopsy commented 1 year ago

To try moving this forward, @HanumanthaRAON do you have some availability for a short video sharing call? So we can review it together and post the findings here later? Feel free to use my cal to schedule: https://calendly.com/dima-grossman/product-ideation

kaptinlin commented 1 year ago

i am getting the errors with the steps below

  1. create a new folder named novu

  2. copy https://github.com/novuhq/novu/blob/next/docker/local/deployment/docker-compose.yml to novu/docker-compose.yml

  3. copy https://github.com/novuhq/novu/blob/next/docker/.env.example to novu/.env

  4. docker-compose up

Then we'll get "Network Error" in the signup process.

HanumanthaRAON commented 1 year ago

@scopsy We are available in IST timezone can you please let me know if you are available at tomorrow 3PM IST 24th Aug .

Please let me your available time or share us meeting invite.

scopsy commented 1 year ago

@HanumanthaRAON feel free to schedule something next week here: https://calendly.com/dima-grossman/15min

billyurs commented 11 months ago

I am also facing similar issue, no help

badan-cloud commented 11 months ago

I managed to fix it.

It looks like it writes the REACT_APP_API_URL (REACT_APP_API_URL: ${API_ROOT_URL}) to the mongodb database during the first start. Subsequent modification of the environment variable API_ROOT_URL doesn't have an impact. And even if you do docker-compose down to remove the environment, the mongodb volume still stays there and is picked up during the next start, so you'll still face the problem.

My solution was to terminate the installation:

docker-compose down

Then remove the mongodb volume:

docker volume rm deployment_mongodb

Then start the application again with correct .env file.

aayushave commented 9 months ago

hi @HanumanthaRAON i would like to connect with you reagrding same if issue is resolved? can you help with ingress and kustomize i am not able to resolve api paths IST time works for me

RShivathanu commented 5 months ago

This network problem is because of api services routing, to correct this error strip the api-services prefix using kubernetes rewrite i.e instead of notifications.app.com/api/v1/auth you have to strip the /api and capture all the remaining coming after the api, below I mentioned the ingress manifest file, go into the file for your better understanding, so the api-services ingress will be notifications.app.com/v1/auth after striping /api..

apiVersion: networking.k8s.io/v1 kind: Ingress metadata: name: api-ingress namespace: novu annotations: nginx.ingress.kubernetes.io/rewrite-target: /$2 spec: ingressClassName: nginx rules:

let me know if still any one struggles with this is issue or any other issue.. thankyou..