search

mikestefanello / pagoda

Rapid, easy full-stack web development starter kit in Go
MIT License
2.21k stars 115 forks source link

Presenting Worker Tasks results #81

Closed humpalum closed 1 month ago

humpalum commented 2 months ago

Hey there,

i was wondering if there's already some logic implemented that allows me to present the results or even the progress of a task to the user.

My use case is something along these lines: User Requests endpoint "search" "search" is something that takes quite some time and i would probably rather do it with a worker -> Open Task "search" Results of search will be displayed async because of its duration.

As far as i understand it so far there would currently be no way of requesting the state as well as the results of the task for the frontend?

I was thinking of doing something like this:

"search" endpoint: Create Task with "search string" and "task id" Inserts into Tasks Table with "task id", "user(so users cant see other users searches)", "task Payload", "status: running", "results: nil" Forward user to /search/:taskid or something similar This page could poll the results

Then the Task can update the table entry with its progress and when it finishes.

User will get the result eventually when polling ( Could also be done via messaging? Haven't looked into that part yet)

While this would work, its sounds kind of stupid to be having two states of the task to manage. I just don`t see how to "store" the results of tasks, but maybe i'm just getting the concept wrong x)

I'm still using asynq for the tasks as i didn't switch yet, but if backlite has a proper solution i'm gladly switching. With asynq i would actually be a little further as i could probably use the ResultWriter as well as Inspector.GetTaskInfo() in order to get the results but i would still need to do the extra work of securing it via an extra table to stop users of accessing other users queries and such...

I hope i could make my question somewhat clear. Thanks for this Project and for any response :)

mikestefanello commented 1 month ago

You're very welcome and I'm glad to hear that you're finding it useful.

The approach that you proposed is exactly what I would probably do in this situation. While it feels like there's some repetition, I think it's best to separate the underlying tasks (that workers interact with) with the end result that your users will interact with. That allows for the most flexibility and adaptability going forward (like if you later switched from asynq to backlite; or something else). It would also allow you to perhaps have this search task be operated on by multiple queued tasks, etc (ie, if there are different steps - like a task that triggers another task, and so on, so it's more of a workflow). If you're using Ent for an ORM, you'd maintain easy access to this entity type (whereas Ent cannot access queued tasks). And since you control the schema, you can store things like the progress or other attributes to present to the user, whereas by only relying on the underlying task, you'd have to stuff that data in to the task itself.