We will use an Azure Service Bus queue to distribute tasks to each VM in the pool. Script should provide the following capabilities:
[x] Create a queue
[x] View the status of a queue
[x] Push tasks to a queue from a file
[x] Pull the next task from a queue to a file
[x] Empty a queue
[x] Delete a queue
Notes
What should the status check show? We could list the contents of each message, but messages can be (i) structured and (ii) up to 1MB in size. Perhaps just show the number of messages on the queue (and potentially some metadata like size or date)?
How should the file containing tasks to be pushed to the queue be structured? We're pushing parameters to be passed to a Julia script initially, but we should make the structure generic enough to be used easily by others. [We chose to pass a simple string in the end, with each task being a line in the source file. We experimented with JSON briefly, but had some issues getting JSON produced in one Julia to be parsed correctly by Python].
martintoreilly
commented
7 years ago
We will use an Azure Service Bus queue to distribute tasks to each VM in the pool. Script should provide the following capabilities:
Notes