Scale to all nodes

[cohsk]

find a way to port rclone / rclone browser to a multi-node / multi-computer / distributed system

[cohsk]

Thought -- scanner container, queue container, process containers. Scanner runs "rclone dry run" and generates a list of items to be processed. List is sent to the queue - status "to be processed", item name, item operation. Process nodes work on items in the queue to completion.

Need to figure out queue contention / arbitration to avoid process node conflicts.

[cohsk]

rough design for a distribution and queueing system

[cohsk]

Updated thought -- leverage the rclone client server architecture. put the work queue producer and consumers on the main node. Only put rclone on the minion nodes.

[cohsk]

[cohsk]

Continued work to simplify the distributed design

• Use EasyScript as the scheduler

• Scale rclone browser all nodes so that we have rcd listening on a port (5572??) internally. Need to figure out how that works when we deploy on athena. If I recall properly from the notes, on the internal 10. network, the native ports, not the translated ports are available.

• Write a python script to be a controller for this. Codename Gru.

• See attached whiteboard captures for psedo code for Gru

[cohsk]

Done as part of release 1.1.2

Note -- distributing rclone jobs is part of project Gru