search

mendableai / firecrawl

🔥 Turn entire websites into LLM-ready markdown or structured data. Scrape, crawl and extract with a single API.
https://firecrawl.dev
GNU Affero General Public License v3.0
13.69k stars 990 forks source link

[BUG] ERR wrong number of arguments for 'sadd' command #567

Closed SalamanderXing closed 1 week ago

SalamanderXing commented 3 weeks ago

Describe the Bug I did a fresh clone of the repo yesterday and followed the self-hosting instructions with docker compose (default .env) but got ERR wrong number of arguments for 'sadd' command.

To Reproduce Steps to reproduce the issue:

  1. Follow instructions on self-hosting. The docker compose up looks like it runs successfully.
  2. 
curl -X POST http://localhost:3002/v0/crawl \
                          -H 'Content-Type: application/json' \
                          -d '{
                        "url": "https://mendable.ai"
                      }'

{"error":"ERR wrong number of arguments for 'sadd' command"}⏎ 


**Environment (please complete the following information):**
- OS: macOS
- Firecrawl Version: 1.0.0
- Node.js Version: default set in docker compose

**Logs**

WARN[0000] The "SELF_HOSTED_WEBHOOK_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "LOGGING_LEVEL" variable is not set. Defaulting to a blank string. WARN[0000] The "PROXY_SERVER" variable is not set. Defaulting to a blank string. WARN[0000] The "PROXY_USERNAME" variable is not set. Defaulting to a blank string. WARN[0000] The "PROXY_PASSWORD" variable is not set. Defaulting to a blank string. WARN[0000] The "BLOCK_MEDIA" variable is not set. Defaulting to a blank string. WARN[0000] The "OPENAI_BASE_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "SELF_HOSTED_WEBHOOK_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "LOGGING_LEVEL" variable is not set. Defaulting to a blank string. WARN[0000] The "OPENAI_BASE_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "SELF_HOSTED_WEBHOOK_URL" variable is not set. Defaulting to a blank string. WARN[0000] The "LOGGING_LEVEL" variable is not set. Defaulting to a blank string. [+] Running 4/4 ✔ Container firecrawl-redis-1 Created 0.0s ✔ Container firecrawl-playwright-service-1 Created 0.0s ✔ Container firecrawl-api-1 Recreated 0.2s ✔ Container firecrawl-worker-1 Recreated 0.1s Attaching to api-1, playwright-service-1, redis-1, worker-1 redis-1 | 1:C 22 Aug 2024 07:53:39.782 oO0OoO0OoO0Oo Redis is starting oO0OoO0OoO0Oo redis-1 | 1:C 22 Aug 2024 07:53:39.782 Redis version=7.4.0, bits=64, commit=00000000, modified=0, pid=1, just started redis-1 | 1:C 22 Aug 2024 07:53:39.782 Configuration loaded redis-1 | 1:M 22 Aug 2024 07:53:39.782 monotonic clock: POSIX clock_gettime redis-1 | 1:M 22 Aug 2024 07:53:39.782 Running mode=standalone, port=6379. redis-1 | 1:M 22 Aug 2024 07:53:39.783 Server initialized redis-1 | 1:M 22 Aug 2024 07:53:39.783 Loading RDB produced by version 7.4.0 redis-1 | 1:M 22 Aug 2024 07:53:39.783 RDB age 3050 seconds redis-1 | 1:M 22 Aug 2024 07:53:39.783 RDB memory usage when created 1.73 Mb redis-1 | 1:M 22 Aug 2024 07:53:39.783 Done loading RDB, keys loaded: 2, keys expired: 0. redis-1 | 1:M 22 Aug 2024 07:53:39.783 DB loaded from disk: 0.000 seconds redis-1 | 1:M 22 Aug 2024 07:53:39.783 Ready to accept connections tcp playwright-service-1 | playwright-service-1 | > playwright-scraper-api@1.0.0 start playwright-service-1 | > node dist/api.js playwright-service-1 | api-1 | ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.7.1.tgz worker-1 | ! Corepack is about to download https://registry.npmjs.org/pnpm/-/pnpm-9.7.1.tgz playwright-service-1 | Server is running on port 3000 api-1 | api-1 | > firecrawl-scraper-js@1.0.0 start:production /app api-1 | > tsc && node dist/src/index.js api-1 | worker-1 | worker-1 | > firecrawl-scraper-js@1.0.0 workers /app worker-1 | > nodemon --exec ts-node src/services/queue-worker.ts worker-1 | worker-1 | [nodemon] 2.0.22 worker-1 | [nodemon] to restart at any time, enter rs worker-1 | [nodemon] watching path(s): . worker-1 | [nodemon] watching extensions: ts,json worker-1 | [nodemon] starting ts-node src/services/queue-worker.ts api-1 | [2024-08-22T07:53:45.025Z]WARN - Authentication is disabled. Supabase client will not be initialized. worker-1 | [2024-08-22T07:53:45.115Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.352Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:45.353Z]INFO - Master 31 is running api-1 | [2024-08-22T07:53:45.401Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:45.801Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.814Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.818Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.821Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.826Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.832Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.851Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:45.968Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:46.030Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:46.077Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:46.086Z]WARN - Authentication is disabled. Supabase client will not be initialized. api-1 | [2024-08-22T07:53:46.086Z]WARN - Authentication is disabled. Supabase client will not be initialized. worker-1 | [2024-08-22T07:53:46.118Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.436Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.442Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.445Z]INFO - Worker 41 started api-1 | [2024-08-22T07:53:46.465Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.468Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.470Z]INFO - Worker 40 started api-1 | [2024-08-22T07:53:46.473Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.475Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.478Z]INFO - Worker 41 listening on port 3002 api-1 | [2024-08-22T07:53:46.478Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.480Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.481Z]INFO - Worker 76 started api-1 | [2024-08-22T07:53:46.492Z]INFO - Worker 40 listening on port 3002 api-1 | [2024-08-22T07:53:46.492Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.512Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.518Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.518Z]INFO - Worker 76 listening on port 3002 api-1 | [2024-08-22T07:53:46.518Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.523Z]INFO - Worker 106 started api-1 | [2024-08-22T07:53:46.527Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.528Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.529Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.532Z]INFO - Worker 69 started api-1 | [2024-08-22T07:53:46.537Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.541Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.549Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.552Z]INFO - Worker 90 started api-1 | [2024-08-22T07:53:46.552Z]INFO - Worker 106 listening on port 3002 api-1 | [2024-08-22T07:53:46.552Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.557Z]INFO - Worker 69 listening on port 3002 api-1 | [2024-08-22T07:53:46.557Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.563Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.566Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.572Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.573Z]INFO - Worker 90 listening on port 3002 api-1 | [2024-08-22T07:53:46.573Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.575Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.577Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.578Z]INFO - Worker 38 started api-1 | [2024-08-22T07:53:46.590Z]INFO - Worker 38 listening on port 3002 api-1 | [2024-08-22T07:53:46.590Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.597Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.652Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.653Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.654Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.654Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.655Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.656Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.656Z]INFO - Worker 59 started api-1 | [2024-08-22T07:53:46.657Z]INFO - Worker 56 started api-1 | [2024-08-22T07:53:46.659Z]INFO - Worker 39 started api-1 | [2024-08-22T07:53:46.675Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.679Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.681Z]INFO - Worker 59 listening on port 3002 api-1 | [2024-08-22T07:53:46.681Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.681Z]INFO - Number of CPUs: 12 available api-1 | [2024-08-22T07:53:46.686Z]INFO - Web scraper queue created api-1 | [2024-08-22T07:53:46.686Z]INFO - Worker 99 started api-1 | [2024-08-22T07:53:46.687Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.689Z]INFO - Worker 56 listening on port 3002 api-1 | [2024-08-22T07:53:46.689Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.691Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.694Z]INFO - Worker 39 listening on port 3002 api-1 | [2024-08-22T07:53:46.694Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.696Z]INFO - Worker 83 started api-1 | [2024-08-22T07:53:46.697Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.709Z]INFO - Worker 99 listening on port 3002 api-1 | [2024-08-22T07:53:46.709Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.711Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:53:46.714Z]INFO - Worker 83 listening on port 3002 api-1 | [2024-08-22T07:53:46.714Z]INFO - For the Queue UI, open: http://0.0.0.0:3002/admin/@/queues api-1 | [2024-08-22T07:53:46.716Z]INFO - Connected to Redis Session Rate Limit Store! api-1 | [2024-08-22T07:56:17.152Z]WARN - You're bypassing authentication api-1 | [2024-08-22T07:56:17.153Z]WARN - You're bypassing authentication api-1 | [2024-08-22T07:56:17.917Z]ERROR - ReplyError: ERR wrong number of arguments for 'sadd' command

mogery commented 3 weeks ago

Hey, looking into this.

AbdurrahmanTalha commented 2 weeks ago

I got a similar error but not the same. 

[2024-08-24T11:23:52.385Z]ERROR - ReplyError: ERR wrong number of arguments for 'expire' command
L1aoxw commented 1 week ago

I got a similar error but not the same.

[2024-08-24T11:23:52.385Z]ERROR - ReplyError: ERR wrong number of arguments for 'expire' command

This bug may be due to the too low version of Redis. I solved this error after updating redis to 7.0.11.

mogery commented 1 week ago

I got a similar error but not the same.

[2024-08-24T11:23:52.385Z]ERROR - ReplyError: ERR wrong number of arguments for 'expire' command

This bug may be due to the too low version of Redis. I solved this error after updating redis to 7.0.11.

Yes, we need a fairly new version of Redis.

Closing this since we've resolved the original issue.