taskforcesh / issues

Taskforce.sh issues
6 stars 0 forks source link

Excessive logging from ioredis after deploying BullMQ and Taskforce #72

Closed rob-grio closed 2 years ago

rob-grio commented 2 years ago

We just deployed BullMQ and Taskforce to our staging environment. The queue and worker are functioning well (we see completed jobs in the Taskforce UI), but we've got this showing up in our logs every 20 seconds now:

Oct 11 16:49:26: [ioredis] Unhandled error event: Error: connect ECONNREFUSED 127.0.0.1:6379
Oct 11 16:49:26:     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1146:16)
Oct 11 16:49:26:     at TCPConnectWrap.callbackTrampoline (internal/async_hooks.js:129:14)

Our Redis instance is not running on 127.0.0.1, so we don't know why anything is trying to make that connection.

Any ideas?

manast commented 2 years ago

Could you please clarify if this log comes from Taskforce.sh on premises or does it come from your BullMQ based application?

rob-grio commented 2 years ago

Unfortunately I can't tell whether it's coming from Taskforce.sh or BullMQ. When our API server starts, it connects to Taskforce.sh like this...

import { Connect } from 'taskforce-connector'

const {
  TASKFORCE_CONNECTION_NAME: name,
  TASKFORCE_CONNECTION_TOKEN: token,
  WORKER_REDIS_HOST: host
} = process.env

if (name && token && host) {
  Connect(name, token, { db: WORKER_REDIS_DB, host })
}

...and to BullMQ like this:

import { Queue, QueueScheduler } from 'bullmq'
import Redis from 'ioredis'

const BACKOFF_DELAY = 1000 * 60
const BACKOFF_TYPE = 'exponential'
const MAX_ATTEMPTS = 7
const MAX_COMPLETED_JOBS = 500
const WORKER_QUEUE_NAME = 'Pyvott Worker'

new QueueScheduler(WORKER_QUEUE_NAME)

const queue: Queue = new Queue(WORKER_QUEUE_NAME, {
  connection: new Redis({ db: 0, host: process.env.WORKER_REDIS_HOST }),
  defaultJobOptions: {
    attempts: MAX_ATTEMPTS,
    backoff: {
      delay: BACKOFF_DELAY,
      type: BACKOFF_TYPE
    },
    removeOnComplete: MAX_COMPLETED_JOBS
  }
})

We have triple-checked that our environment variables are all correctly defined.

manast commented 2 years ago

You are not specifying the redis host for the QueueScheduler:

new QueueScheduler(WORKER_QUEUE_NAME)
rob-grio commented 2 years ago

Thanks! That resolved the problem.