search

n8n-io / n8n

Free and source-available fair-code licensed workflow automation tool. Easily automate tasks across different services.
https://n8n.io
Other
48.34k stars 7.52k forks source link

n8n workers wont connect to main node. #8767

Closed shankarmn94 closed 8 months ago

shankarmn94 commented 8 months ago

Bug Description

server 1

WEBHOOK_URL=http://10.102.0.5:5678 DB_TYPE=postgresdb DB_POSTGRESDB_DATABASE=workflow DB_POSTGRESDB_HOST=10.102.0.5 DB_POSTGRESDB_PORT=5432 DB_POSTGRESDB_USER=workflow DB_POSTGRESDB_PASSWORD=workflow N8N_PAYLOAD_SIZE_MAX=56MB EXECUTIONS_DATA_PRUNE=true EXECUTIONS_DATA_PRUNE_MAX_COUNT=50000 EXECUTIONS_DATA_SAVE_ON_ERROR=all EXECUTIONS_DATA_SAVE_ON_SUCCESS=none NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js" N8N_ENCRYPTION_KEY=DAWJBDAWIGBCAIUCNALA

EXECUTIONS_MODE=queue

QUEUE_BULL_REDIS_HOST=10.102.0.3

server 2 - same env

server 1 i started n8n using of npm as in

cd ~/.n8n here the .env will be there.

/usr/bin/n8n start

User settings loaded from: /root/.n8n/config Initializing n8n process n8n ready on 0.0.0.0, port 5678 Version: 1.27.3

Start Active Workflows:

Editor is now accessible via: http://localhost:5678/

in server 2 with same folder and env cd ~/.n8n /usr/bin/n8n worker.

it will start as

n8n worker User settings loaded from: /root/.n8n/config

n8n worker is now ready

when i start a workflow it will will run in only server (server 1) and wont come to other server.

if i enable quemode it will give error like

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

how can i handle and resolve this

To Reproduce

same settings as above

Expected behavior

workers should get running or how can i start multiple servers as n8n start for same db..

Operating System

ubuntu 22.04

n8n Version

1.27.3

Node.js Version

v20.10.0

Database

postgresql

Execution mode

main (default) and queue mode tried

Joffcom commented 8 months ago

@shankarmn94 this doesn't appear to be a bug and looks to probably be around your configuration.

For general support I would recommend posting on the community support forum.

To start with this make sure you have followed our documentation on setting up queue mode (both all instances need to be set to run in queue) it would also be handy to know how you are triggering your workflow, if you are doing it from the UI then I would expect the main n8n instance to process the job but if it is a webhook or a schedule the worker should take it.

shankarmn94 commented 8 months ago

could you just share that please i got confused after trying these que mode with different configs.

https://docs.n8n.io/hosting/scaling/queue-mode/#set-encryption-key

this is what i tried.

shankarmn94 commented 8 months ago

@Joffcom

shankarmn94 commented 8 months ago

sometimes we are getting context deadline exceeded.

Joffcom commented 8 months ago

@shankarmn94 Can you share the full output from the worker nodes when you have queue mode configured as documented?

shankarmn94 commented 8 months ago

WEBHOOK_URL=http://10.51.112.8:5678/ N8N_BASIC_AUTH_USER=cloudquen8n N8N_BASIC_AUTH_PASSWORD=n8ncloudque DB_TYPE=postgresdb DB_POSTGRESDB_DATABASE=workflow DB_POSTGRESDB_HOST=10.51.112.8 DB_POSTGRESDB_PORT=5432 DB_POSTGRESDB_USER=workflow DB_POSTGRESDB_PASSWORD=workflow

N8N_PAYLOAD_SIZE_MAX=56MB

EXECUTIONS_DATA_PRUNE=true EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000 EXECUTIONS_DATA_SAVE_ON_ERROR=all EXECUTIONS_DATA_SAVE_ON_SUCCESS=all NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js" EXECUTIONS_MODE=queue N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru QUEUE_BULL_REDIS_HOST=10.51.112.9 QUEUE_BULL_PREFIX=workflows QUEUE_BULL_REDIS_DB=0 QUEUE_BULL_REDIS_PORT=6379

i kept this in two node.. one i started as main like n8n start and in other server n8n worker..

services will start..

worker wont get any request or take workflows...

if i trigger workflow in main nodes, below errror is mentioned

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

shankarmn94 commented 8 months ago

what is wrong with this.

Joffcom commented 8 months ago

Hey @shankarmn94,

Can you share the full output from the worker node when it is starting.

shankarmn94 commented 8 months ago

root@cs-dev-5:~/.n8n# /usr/bin/n8n start User settings loaded from: /root/.n8n/config Initializing n8n process n8n ready on 0.0.0.0, port 5678 Version: 1.27.3

Start Active Workflows:

Editor is now accessible via: http://localhost:5678/

Press "o" to open in Browser. ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key ReplyError: ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key ERR Error running script (call to 8f55ae4a3be429c6d38c5d5db3e80edf89197b64): @user_script:56: script tried accessing undeclared key

/usr/bin/n8n worker User settings loaded from: /root/.n8n/config

n8n worker is now ready

Joffcom commented 8 months ago

Hey @shankarmn94,

I don't see any issue from the worker not connecting to the redis queue to pick up jobs, When you are testing to show it isn't working are you running the workflow on a schedule or using the test button in the n8n interface?

shankarmn94 commented 8 months ago

@Joffcom schedule / trigger

shankarmn94 commented 8 months ago

now i made some changes;

1) "n8n:cache:variables" 2) "bull:jobs:2" 3) "bull:jobs:id" 4) "bull:jobs:4" 5) "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78533" 6) "workflows:jobs:stalled-check" 7) "bull:jobs:priority" 8) "bull:jobs:wait" 9) "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78526" 10) "n8n:cache:webhook:POST-573da17f-bd25-4604-a542-1d1cc455b6a4" 11) "bull:jobs:5" 12) "n8n:cache:webhook:POST-2960855a-ff6d-48cb-b2d9-77faf5c78523" 13) "bull:jobs:3" 14) "bull:jobs:1"

its creating the keys in redis but the worker is not picking this nor main server not runing this

Joffcom commented 8 months ago

Hey @shankarmn94

I am not sure what those settings are that you have there but if they are in Redis I would recommend not playing with them.

Can you share the current configuration for your main instance and your worker as they currently are so I can check them again I suspect you still don't have it configured correctly.

shankarmn94 commented 8 months ago

as i mentioned earlier:

WEBHOOK_URL=http://10.51.112.8:5678/ N8N_BASIC_AUTH_USER=cloudquen8n N8N_BASIC_AUTH_PASSWORD=n8ncloudque DB_TYPE=postgresdb DB_POSTGRESDB_DATABASE=workflow DB_POSTGRESDB_HOST=10.51.112.8 DB_POSTGRESDB_PORT=5432 DB_POSTGRESDB_USER=workflow DB_POSTGRESDB_PASSWORD=workflow

N8N_PAYLOAD_SIZE_MAX=56MB

EXECUTIONS_DATA_PRUNE=true EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000 EXECUTIONS_DATA_SAVE_ON_ERROR=all EXECUTIONS_DATA_SAVE_ON_SUCCESS=all NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js" EXECUTIONS_MODE=queue N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru QUEUE_BULL_REDIS_HOST=10.51.112.9 QUEUE_BULL_PREFIX=workflows QUEUE_BULL_REDIS_DB=0 QUEUE_BULL_REDIS_PORT=6379

Joffcom commented 8 months ago

Hey @shankarmn94,

I just wanted to check again as the first post you had queue mode disabled which would have been an issue.

So assuming you have those settings on workers and the main instance it will work although there are some tweaks you can make as 2 of those keys don't exist anymore so in theory the below would work.

WEBHOOK_URL=http://10.51.112.8:5678/
DB_TYPE=postgresdb
DB_POSTGRESDB_DATABASE=workflow
DB_POSTGRESDB_HOST=10.51.112.8
DB_POSTGRESDB_PORT=5432
DB_POSTGRESDB_USER=workflow
DB_POSTGRESDB_PASSWORD=workflow
EXECUTIONS_DATA_PRUNE=true
EXECUTIONS_DATA_PRUNE_MAX_COUNT=5000000
EXECUTIONS_DATA_SAVE_ON_ERROR=all
EXECUTIONS_DATA_SAVE_ON_SUCCESS=all
NODE_OPTIONS="--max-old-space-size=16048 --require /root/.n8n/set-max-listeners.js"
EXECUTIONS_MODE=queue
N8N_ENCRYPTION_KEY=Zwyi2FIVHTJEYb5+KY74esTPlKn4Vjru
QUEUE_BULL_REDIS_HOST=10.51.112.9
QUEUE_BULL_PREFIX=workflows

How are you actually setting the env options?

shankarmn94 commented 8 months ago

am able to figure out the issue i had:

We were using dragonflydb which is alternative of redis.. while workers fails to get keys from dragonflydb later on i checked with redis where all my workers started take the triggered workflows and finish those..

i donno is that issue with n8n worker or dragonflydb.

Joffcom commented 8 months ago

Hey @shankarmn94,

We only officially support Redis so I would recommend sticking with that and not something that is Redis compatible, For now I am going to mark this as closed as it doesn't look like we have an issue here.

Thanks for the update 👍🏻

shankarmn94 commented 8 months ago

But Dragonlfydb is very big platform i donno why you dont support that. Anyway thanks for response. Have a great day...