Payload return from handler?

[justinclift]

Looking to replace RabbitMQ with something a bit simpler, so now looking at Neoq.

It seems fairly straight forward for the "add jobs to queue, have handlers process the jobs" type of thing.

There doesn't seem to be a clear way for handlers to return data (eg resized image, etc) back to the job submitter though.

Does that need to be done separately? eg have a db table for return data, probably indexed on job id or similar

[acaloiaro]

Hi @justinclift, you're right that there's no clear way for handlers to return data.

Returning data (in the sense of a function return type) would be a synchronous workflow, which is not in scope for neoq.

While that's the case, I'd simply encourage you to think about your problem differently. Let's take your case of resizing images.

In a synchronous workflow for resizing images, you ask a function to perform a resize, and the caller of that function does something with the resized image bytes after they're returned. Typically, they're going to place the image in permanent storage.

In an asynchronous workflow like neoq, this can be achieved as follows:

Instead of the caller placing the returned bytes in permanent storage, the caller declares where it wants the handler to store the bytes, leaving the permanent storage responsibility up to the handler code. Doing so might look like a job on a resize_images queue with a payload like

{
  "original": "https://files.example.com/path/to/original.png",
  "original_base64": "",
  "resize": "64x64", 
  "storage": {
     "bucket": "my-resized-images", "path": "/path/to/64x64.jpeg"
  }
}

With this payload structure, the code enqueuing the job declares where it wants resized images stored. It either downloads the original from the original URL, or grabs the base64-encoded bytes from original_base64, performs conversion, and places the result where storage specifies it should. On success, the resized image is guaranteed to be in the declared storage bucket and at the given path.

[justinclift]

Thanks. My example was just using image resizing as it's one of the code examples given.

My actual use cases though are substantially different from that, mostly around doing SQL operations or SQL adjacent things (database creation/deletion, running SQL queries and returning results, etc).

Can probably get away with returning SQL query results via a database table indexed on the job id. Will experiment a bit and see how it goes. :smile:

[acaloiaro]

Gotcha. In that case, I recommend inserting records in your tables directly from your Handler code.

[justinclift]

Yeah, that's the initial idea I'm working with. :smile: