cloudflare / workers-sdk

⛅️ Home to Wrangler, the CLI for Cloudflare Workers®
https://developers.cloudflare.com/workers/
Apache License 2.0
2.4k stars 589 forks source link

🐛 BUG: Producer sending message to queue throwing Error: Network connection lost in Miniflare #5908

Open Soviut opened 1 month ago

Soviut commented 1 month ago

Which Cloudflare product(s) does this pertain to?

Miniflare

What version(s) of the tool(s) are you using?

3.20240512.0 [miniflare]

What version of Node are you using?

20.9.0

What operating system and version are you using?

Unbutu 22 under WSL2 on Windows 11

Describe the Bug

My producer can't send messages to my queue in miniflare.

Observed behavior

I'm testing a basic miniflare configuration with two workers and a queue. One worker is the producer, the other is the consumer.

Trying to send any messages to the queue result in the following error.

Error: Network connection lost.

It happens when you call the producer's route (which sends a message to the queue)

const res = await mf.dispatchFetch('http://localhost:5000/producer')

And it also happens when you get the producer directly from the miniflare instance and try to send a message.

const queue = await mf.getQueueProducer('CANDIDATES_QUEUE', 'producer')
queue.send({ ding: 'Hello from Miniflare!' })

Expected behavior

I expect the producer to be able to send a message to the queue and return a response. I expect the consumer to consume the message from the queue and log out what it found.

Steps to reproduce

run the following code with npx tsx .

import { Miniflare } from 'miniflare'

const mf = new Miniflare({
  // host: '0.0.0.0',
  port: 5000,

  workers: [
    {
      name: 'producer',
      modules: true,
      routes: ['http://localhost/producer*'],
      script: `
        export default {
          async fetch(req, env, ctx) {
            console.log('fetch')
            await env.CANDIDATES_QUEUE.send({
              testing: 'Hello from Miniflare!',
            })
            return new Response('Producer sent message to queue');
          }
        }
      `,

      queueProducers: {
        CANDIDATES_QUEUE: 'candidates-queue',
      },
    },
    {
      name: 'consumer',
      modules: true,
      routes: ['http://localhost/queue*'],
      script: `
        export default {
          async queue(batch, env) {
            console.log('received ' + batch.messages.length)
            for (let message of batch.messages) {
              console.log(
                'message ' + message.id + ' processed: ' + JSON.stringify(message.body)
              )
            }
          }
        }
      `,

      queueConsumers: ['candidates-queue'],
    },
  ],
})

// this throws the network error
const res = await mf.dispatchFetch('http://localhost:5000/producer')
console.log(res.text())

// this throws the network error
const queue = await mf.getQueueProducer('CANDIDATES_QUEUE', 'producer')
queue.send({ ding: 'Hello from Miniflare!' })

Please provide a link to a minimal reproduction

No response

Please provide any relevant error logs

node:internal/process/task_queues:95
    runMicrotasks();
    ^

Error: Network connection lost.
    at process.processTicksAndRejections (node:internal/process/task_queues:95:5) {
  [cause]: undefined
}

Node.js v20.9.0
TimoWilhelm commented 1 week ago

Seeing the same issues on Windows 11 Node v21.7.3 with miniflare 3.20240610.1. All queue producers fail with Error: Network connection lost.

TimoWilhelm commented 6 days ago

I assume this issue is caused by https://github.com/cloudflare/workers-sdk/commit/66bdad08834b403100d1e4d6cd507978cc50eaba.

The zod schema for queueProducers is wrong and the Record<string, string> format is no longer valid. Using Record<string, QueueProducerOptions> works.

queueProducers: {
  CANDIDATES_QUEUE: {
    queueName: 'candidates-queue',
  },
},

I created a PR to fix the zod schema here https://github.com/cloudflare/workers-sdk/pull/6128.