[BUG] redis (sentinel): ERR unknown command `eval`, with args beginning with:

[uhthomas]

The bug

Hi, I'm using redis and sentinel and Immich won't start.

/usr/src/app/node_modules/redis-parser/lib/parser.js:179
    return new ReplyError(string)
           ^

ReplyError: ERR unknown command `eval`, with args beginning with: `--[[   Adds a job to the queue by doing the following:     - Increases the job counter if needed.     - Creates a new job key wi`,
    at parseError (/usr/src/app/node_modules/redis-parser/lib/parser.js:179:12)
    at parseType (/usr/src/app/node_modules/redis-parser/lib/parser.js:302:14) {
  command: {
    name: 'eval',
    args: [
      '--[[\n' +
        '  Adds a job to the queue by doing the following:\n' +
        '    - Increases the job counter if needed.\n' +
        '    - Creates a new job key with the job data.\n' +
        '\n' +
        '    - if delayed:\n' +
        '      - computes timestamp.\n' +
        '      - adds to delayed zset.\n' +
        "      - Emits a global event 'delayed' if the job is delayed.\n" +
        '    - if not delayed\n' +
        '      - Adds the jobId to the wait/paused list in one of three ways:\n' +
        '         - LIFO\n' +
        '         - FIFO\n' +
        '         - prioritized.\n' +
        '      - Adds the job to the "added" list so that workers gets notified.\n' +
        '\n' +
        '    Input:\n' +
        "      KEYS[1] 'wait',\n" +
        "      KEYS[2] 'paused'\n" +
        "      KEYS[3] 'meta-paused'\n" +
        "      KEYS[4] 'id'\n" +
        "      KEYS[5] 'delayed'\n" +
        "      KEYS[6] 'priority'\n" +
        '\n' +
        '      ARGV[1]  key prefix,\n' +
        '      ARGV[2]  custom id (will not generate one automatically)\n' +
        '      ARGV[3]  name\n' +
        '      ARGV[4]  data (json stringified job data)\n' +
        '      ARGV[5]  opts (json stringified job opts)\n' +
        '      ARGV[6]  timestamp\n' +
        '      ARGV[7]  delay\n' +
        '      ARGV[8]  delayedTimestamp\n' +
        '      ARGV[9]  priority\n' +
        '      ARGV[10] LIFO\n' +
        '      ARGV[11] token\n' +
        ']]\n' +
        'local jobId\n' +
        'local jobIdKey\n' +
        'local rcall = redis.call\n' +
        '\n' +
        'local jobCounter = rcall("INCR", KEYS[4])\n' +
        '\n' +
        'if ARGV[2] == "" then\n' +
        '  jobId = jobCounter\n' +
        '  jobIdKey = ARGV[1] .. jobId\n' +
        'else\n' +
        '  jobId = ARGV[2]\n' +
        '  jobIdKey = ARGV[1] .. jobId\n' +
        '  if rcall("EXISTS", jobIdKey) == 1 then\n' +
        '    return jobId .. "" -- convert to string\n' +
        '  end\n' +
        'end\n' +
        '\n' +
        '-- Store the job.\n' +
        'rcall("HMSET", jobIdKey, "name", ARGV[3], "data", ARGV[4], "opts", ARGV[5], "timestamp", ARGV[6], "delay", ARGV[7], "priority", ARGV[9])\n' +
        '\n' +
        '-- Check if job is delayed\n' +
        'local delayedTimestamp = tonumber(ARGV[8])\n' +
        'if(delayedTimestamp ~= 0) then\n' +
        '  local timestamp = delayedTimestamp * 0x1000 + bit.band(jobCounter, 0xfff)\n' +
        '  rcall("ZADD", KEYS[5], timestamp, jobId)\n' +
        '  rcall("PUBLISH", KEYS[5], delayedTimestamp)\n' +
        'else\n' +
        '  local target\n' +
        '\n' +
        '  -- Whe check for the meta-paused key to decide if we are paused or not\n' +
        '  -- (since an empty list and !EXISTS are not really the same)\n' +
        '  local paused\n' +
        '  if rcall("EXISTS", KEYS[3]) ~= 1 then\n' +
        '    target = KEYS[1]\n' +
        '    paused = false\n' +
        '  else\n' +
        '    target = KEYS[2]\n' +
        '    paused = true\n' +
        '  end\n' +
        '\n' +
        '  -- Standard or priority add\n' +
        '  local priority = tonumber(ARGV[9])\n' +
        '  if priority == 0 then\n' +
        '      -- LIFO or FIFO\n' +
        '    rcall(ARGV[10], target, jobId)\n' +
        '  else\n' +
        '    -- Priority add\n' +
        '    rcall("ZADD", KEYS[6], priority, jobId)\n' +
        '    local count = rcall("ZCOUNT", KEYS[6], 0, priority)\n' +
        '\n' +
        '    local len = rcall("LLEN", target)\n' +
        '    local id = rcall("LINDEX", target, len - (count-1))\n' +
        '    if id then\n' +
        '      rcall("LINSERT", target, "BEFORE", id, jobId)\n' +
        '    else\n' +
        '      rcall("RPUSH", target, jobId)\n' +
        '    end\n' +
        '\n' +
        '  end\n' +
        '\n' +
        '  -- Emit waiting event (wait..ing@token)\n' +
        '  rcall("PUBLISH", KEYS[1] .. "ing@" .. ARGV[11], jobId)\n' +
        'end\n' +
        '\n' +
        'return jobId .. "" -- convert to string\n',
      '6',
      'immich_bull:search-queue:wait',
      'immich_bull:search-queue:paused',
      'immich_bull:search-queue:meta-paused',
      'immich_bull:search-queue:id',
      'immich_bull:search-queue:delayed',
      'immich_bull:search-queue:priority',
      'immich_bull:search-queue:',
      '',
      'search-index-assets',
      '{}',
      '{"attempts":3,"removeOnComplete":true,"removeOnFail":false,"delay":0,"timestamp":1684872218171}',
      '1684872218171',
      '0',
      '0',
      '0',
      'LPUSH',
      'd1178744-332e-4a2f-bd8b-266684bc2fa7'
    ]
  }
}

Node.js v18.16.0

The OS that Immich Server is running on

Kubernetes

Version of Immich Server

1.57.1

Version of Immich Mobile App

N/A

Platform with the issue

• [X] Server
• [ ] Web
• [ ] Mobile

Your docker-compose.yml content

N/A

Your .env content

N/A

Reproduction steps

Use Redis Sentinel.

Additional information

https://github.com/uhthomas/automata/tree/898f231a69cffdbff6d6e1d74bc4d3d0ed0441ba/k8s/unwind/immich

[alextran1502]

@samip5 Do you have much experience in this?

[uhthomas]

I was thinking maybe a bad Redis version, but this is the image I'm using redis:6.2.6-alpine. Maybe I should try redis 7?

[jrasm91]

Not sure what I'm looking at here, but none of this looks familiar. The error is most likely not an immich bug, but related to however you are deploying it, which looks custom.

[alextran1502]

We are using Redis 6.2 for the official compose file.

[uhthomas]

We are using Redis 6.2 for the official compose file.

ah :(

[uhthomas]

Not sure what I'm looking at here, but none of this looks familiar. The error is most likely not an immich bug, but related to however you are deploying it, which looks custom.

It should just be standard redis/sentinel.

[samip5]

@samip5 Do you have much experience in this?

I don't sorry, but that sounds like an issue with how the config is encoded.

@uhthomas Could you please show us the base64 encoded redis sentinel config you're passing to immich in env? (I'm sorry, but I cannot read the cue things in your repo)

[uhthomas]

@samip5

I'm not quite sure exactly what you're looking for, but I hope this helps.

❯ k -n immich get deploy immich-server -ojson | jq '.spec.template.spec.containers[].env[] | select(.name | contains("REDIS"))'
{
  "name": "REDIS_HOSTNAME",
  "value": "rfs-redis"
}
{
  "name": "REDIS_PORT",
  "value": "26379"
}

❯ k -n immich get cm rfr-redis -ojson | jq '.data["redis.conf"]' -r
slaveof 127.0.0.1 6379
port 6379
tcp-keepalive 60
save 900 1
save 300 10
user pinger -@all +ping on >pingpass

❯ k -n immich get cm rfs-redis -ojson | jq '.data["sentinel.conf"]' -r
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 1000
sentinel failover-timeout mymaster 3000
sentinel parallel-syncs mymaster 2

[uhthomas]

Strange https://github.com/OptimalBits/bull/issues/1026, I guess I should try to figure out why "eval" is missing.

[samip5]

@samip5

I'm not quite sure exactly what you're looking for, but I hope this helps.

❯ k -n immich get deploy immich-server -ojson | jq '.spec.template.spec.containers[].env[] | select(.name | contains("REDIS"))'
{
  "name": "REDIS_HOSTNAME",
  "value": "rfs-redis"
}
{
  "name": "REDIS_PORT",
  "value": "26379"
}

❯ k -n immich get cm rfr-redis -ojson | jq '.data["redis.conf"]' -r
slaveof 127.0.0.1 6379
port 6379
tcp-keepalive 60
save 900 1
save 300 10
user pinger -@all +ping on >pingpass

❯ k -n immich get cm rfs-redis -ojson | jq '.data["sentinel.conf"]' -r
sentinel monitor mymaster 127.0.0.1 6379 2
sentinel down-after-milliseconds mymaster 1000
sentinel failover-timeout mymaster 3000
sentinel parallel-syncs mymaster 2

That seems like it's not correct sentinel config for immich, as it's expecting a base64 encoded url for it in REDIS_URL env variable.

Please see https://github.com/immich-app/immich/blob/main/docker/example.env#L21L24

[uhthomas]

I don't think that's it.

https://github.com/immich-app/immich/blob/e2bd7e1e085e7d8828fe3f9e201afd99e1eaf170/server/libs/infra/src/infra.config.ts#L18-L19

[samip5]

I don't think that's it.

https://github.com/immich-app/immich/blob/e2bd7e1e085e7d8828fe3f9e201afd99e1eaf170/server/libs/infra/src/infra.config.ts#L18-L19

https://github.com/immich-app/immich/blob/e2bd7e1e085e7d8828fe3f9e201afd99e1eaf170/server/libs/infra/src/infra.config.ts#L9

It's expecting it to be normal redis.

[bo0tzz]

To configure redis sentinel please use the REDIS_URL env var.

[samip5]

Hi, @uhthomas you might be interested our hobbyist communify of people running Kubernetes at home, which you don't seem to be apart of yet.

https://k8s-at-home.com

[uhthomas]

Just following up, thank you so much for the help and apologies for my misunderstanding / lack of understanding with respect to redis and sentinels. The URL worked as expected.

https://github.com/uhthomas/automata/blob/0549cc9fbdf656e969b6ba963a34cc6e94402bd7/k8s/unwind/immich/immich_server/deployment_list.cue#L64-L73

[uhthomas]

Hi, @uhthomas you might be interested our hobbyist communify of people running Kubernetes at home, which you don't seem to be apart of yet.

https://k8s-at-home.com

Thanks! I'll take a look :)