Closed kennsippell closed 1 month ago
Discussion here to also move households and not just CHP areas https://github.com/moh-kenya/config-echis-2.0/issues/1662 + using forms in the CHT to trigger the move (not a UI interface like cht-user-management).
Notes from design conversation today with @paulpascal.
What states will jobs go through:
What is needed from a Cloud-Based Queue
Design questions:
Next Steps:
-- End of MVP --
Any thoughts of doing this through a long lived task via CHT Core API and Sentinel? While it would mean needing to upgrade Core where you want to use the feature, there's already a system in place for long running tasks and queues - eg bulk upload. The user man. tool should also be able to query the status of the job as well.
@mrjones-plip Who can we talk with to learn about how this is implemented in sentinel and how these long running tasks/queues work today? I'm only familiar with our homebrew couch-based queuing system used for outbound push. Is this the same system?
@latin-panda @m5r and @njuguna-n did the original work on the CSV bulk upload according the PR I found!
cc @jkuester
Thanks. Quick scan and this PR doesn't appear to touch sentinel at all. Perhaps bulk uploads aren't processed by sentinel at all?
Oh! Well, that would be good to know if it wasn't in Sentinel - sorry if I've led you astray.
I dug up some test steps (private slack link) that I originally used to performance testing of bulk upload. Until some the engineers chime in on this ticket, maybe this will better expose how it works? From what I can tll there's a parseCsv()
function which writes to the medic_log
database. The medic_log
entry is queried via AJAX to update the job progress on screen as its rows of the CSV are processed. When it's done a CSV of errors is available for download.
Yes, as @mrjones-plip explained, that bulk upload tool is a bunch of promises and waits until all are resolved - ongoing in the server (not a scheduler and not sentinel) - and writes the upload status in the medic_log (how many users are pending, failed, or successful). At the moment, I don't have more experience in Sentinel than what's documented, perhaps if this feature is expected to be heavy, it might need to create some sort of transition (entry point) then Sentinel will start listening for db changes and queueing those transitions to apply the changes
OK good to know! Thanks to both of you for the background info.
If we consider the long-term plan to move contacts without dataloss via something like https://github.com/medic/cht-core/issues/8860, then this will become a very async operation (multiple days). Therefore, I don't think the pattern used for bulk upload is right for this problem.
I think it's quite a bit more complex to build in sentinel. It would probably look a bit like how outbound push is written now; but I personally think we shouldn't be investing bespoke queuing technologies which clutter our already strained CouchDB. If a Core Dev has time to take this on, I do think it would be a more reusable approach across projects ... but I also think this is too much to ask of @paulpascal who has time to make progress on this now, and is doing great via a public reusable queue.
Thanks for the suggestion and please do let me know if you think we're missing anything, if our plan to move contacts via multi-day async queuing isn't correct, or if anything else here isn't in the best interest of users.
Thanks @kennsippell - that sounds like a fair assessment! I appreciate the consideration on what it would look like inside CHT Core vs outside.
Currently this tool outputs a
cht
console line which can be executed to move contacts. Let's take this to the next step and execute that command in the cloud.