Automating analysis workflow

[constantinpape]

I have been thinking a bit about how to automate running the analysis pipeline. I think the ideal setup would be setting up a cronjob that does the following:

• rsync some directory on vibors microscopy set-up where he puts the data that should be processed.
• return if rsync didn't copy anything
• pull the current batchlib master
• run analysis on the new data
• have a slackbot that posts summary to our channel (Was there a failure running the pipeline? How many plates / images were copied and processed? etc.) (this one is kind of optional, but it would be cool to have :D)

Obviously, this will only work if we can set up rsync to work with windows at EMBL; I will try to do this with EMBL IT next week. Some other things we would need:

• agree on which analysis workflow(s) to run "in production"
• move those into the batchlib library, for easy integration testing (see below)
• add CI that runs an integration test for the production workflow (see also #21)
• keep track of the batchlib commit that was used for analysis, see #48

[wolny]

Hey @constantinpape, the setup you described should definitely be executable, however I'd rather choose a different trigger for starting the pipeline than the cronjob/rsync. I'd rather rely on something sending the data to EMBL premises, where we have a watchdog process (it can be a cronjob), which spawns the pipeline process (point 3 onward) when a new data shows up. It's not a big change to the setup you described, but would allow us to make the processing on EMBL side bulletproof without worrying about the cronjob/rsync connection not working properly. At the beginning the data can be sent manually and we would see this nice slack messages right away.

[constantinpape]

I'd rather rely on something sending the data to EMBL premises, where we have a watchdog process (it can be a cronjob), which spawns the pipeline process (point 3 onward) when a new data shows up. It's not a big change to the setup you described, but would allow us to make the processing on EMBL side bulletproof without worrying about the cronjob/rsync connection not working properly. At the beginning the data can be sent manually and we would see this nice slack messages right away.

Yeah that's a good point and doesn't change much in terms of setting up the system. @imagirom and me are working on getting the whole code more deployable now and then we can sync on what to do here exactly in the next few days; also with the database solution we eventually want to have in mind.

[constantinpape]

Working heavily on cleaning up the code and making it run smoother in #51. Figuring out the automatic deployment is not so high priority, because the number of new plates in the next 1 or 2 weeks will still be manageable with manually taking care of running jobs.