This makes it possible to transfer files over HTTP to the web server. To not expose the storage over the web, storage will run an rsync task to fetch newly arrived files. This avoids using SCP and spreading write-access SSH keys, or handling temporary token SSH keys.
MD5 sums will be calculated in the upload view (transfer_file), unless the uploaded file is zipped, then it will be unzipped on storage and MD5 will be calculated on the relevant inside file.
In a future update we'll have to make sure we can also upload pre-zipped files to storage/archive, where the MD5 should happen on the zip file. Usecase: e.g. thousands of microscopy files.
An issue to look at would be upload times, HTTP will time out unlike SCP which we used before.
This makes it possible to transfer files over HTTP to the web server. To not expose the storage over the web, storage will run an rsync task to fetch newly arrived files. This avoids using SCP and spreading write-access SSH keys, or handling temporary token SSH keys.
MD5 sums will be calculated in the upload view (transfer_file), unless the uploaded file is zipped, then it will be unzipped on storage and MD5 will be calculated on the relevant inside file. In a future update we'll have to make sure we can also upload pre-zipped files to storage/archive, where the MD5 should happen on the zip file. Usecase: e.g. thousands of microscopy files. An issue to look at would be upload times, HTTP will time out unlike SCP which we used before.