Feature request: Uploading data, how to move away from the silly drag-and-drop

[lpelkmans]

As far as I know, we still do not have a good implementation of a functionality we had in iBrain, namely that one could point at a location somewhere on a server and have every image coming into that location automatically processed and analysed with some pre-defined pipeline.

This functionality will be essential for automated high-throughput screens, and in general for every experiment in which total imaging time may be long, and time can be saved by not having to wait before processing and analysis starts until all images are acquired.

[alecrimi]

This features is instead been implemented, it is the icon on the right which allows to select files files from a folder instead of the drag&drop feature

[lpelkmans]

That's good, but that does still require all images to be present right? We will need something (a daemon function) that actively monitors a location for new incoming images and submits them automatically for processing and analysis.

Lucas Pelkmans PhD Professor & Ernst Hadorn Chair Department of Molecular Life Sciences University of Zurich, Irchel Campus Winterthurerstrasse 190 8057 Zurich Switzerland

Building/Room: Y55-K-04 Phone (direct): +41 44 63 53 123 Phone (secretary): +41 44 63 53 193 Fax: +41 44 63 56 811 Email: lucas.pelkmans@imls.uzh.ch URL: www.pelkmanslab.org

On 23 Jan 2017, at 19:37, Alessandro Crimi notifications@github.com wrote:

This features is instead been implemented, it is the icon on the right which allows to select files files from a folder instead of the drag&drop feature

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or mute the thread.

[hackermd]

Upload of files has the following steps:

• check if files are valid (tmserver.api.upload.file_validity_check()): ensures that files have the correct format for the specified microscope type (see for example tmlib.workflow.metaconfig.cellvoyager for the CV7000 microscope)
• register files for upload (tmserver.api.upload.register_upload()): creates a database entry for each file that is valid and should be upload (see tmlib.models.file.MicroscopeImageFile)
• upload files (tmserver.api.upload.upload_file()): saves each uploaded file server side and updates the respective entry in the database

At the moment, this is implemented client side (in the user interface and in tmclient.api. upload_microscope_files()) such that all images in the provided folder are registered at once and then uploaded one after another.

To achieve upload of newly added images on the fly, we would need a server (or daemon process) that runs client side (on the microscope computer) and

• screens the folder of interest and keeps track of images
• registers new images and uploads them via tm_client

To this end, I would implement

• a new method tmclient.api.upload_image_file(), which takes filename instead of directory as input argument and registers and uploads an individual file
• a small bash or python script ("tm_robocopy"), which monitors folder(s) and directly uploads new files via tmclient.api.upload_image_file()

The script can then be run in the background, i.e. daemonized.