Atomic uploads

[jamesmoss]

We have a problem where large files (that can take a few seconds to upload) get access by visitors whilst they are still being transferred to the server. e.g they get served a half uploaded CSS file and the site looks wrong.

Transmit on OSX gets around this by uploading the file using a temporary file name, then performs a rename it once it's fully uploaded.

Would it be possible to do something like this with dandelion? My Ruby knowledge isnt that strong but I can work on this feature if people want it. I imagine there'd need to be a config option to toggle it but should it be on or off by default?

[joeholdcroft]

+1

[martinheidegger]

With another (more cumbersome) deploy tool I implemented this feature by uploading the new version in a temporary directory like:

/www_20131023232013300/

then I rename the original directory to a temporary folder

/www/ → /www_old/

then I rename the new folder to be used as main folder

/www_20131023232013300/ → /www/

this way all javascript/css/whatever is consistent to the user of the site and the site is down for the duration of 1 rename call to ftp (<1second) no matter how big the changes were. (of course the old /www/ directory gets deleted)

It is a safe approach but also a slow one since I have to upload all files on every deploy. It would be cooler if the system would try to copy the files on the ftp server:

/www/ → (copy) → /www_20131023232013300/

and then (like dandelion) upload just the difference to the temporary directory and then apply the switch.

[jamesmoss]

This is a similar approach to the one capistrano takes and I personally think it's probably overkill for a little tool like dandelion. Even for a one line change you'd end up duplicating the entire site. For large sites this would be unfeasible; it also doesn't consider untracked files e.g file uploads. Capistrano gets around this by storing assets outside of the deploy directory and then symlinks them in. We can't do that with dandelion as it's not possible via FTP, I can also foresee issues with shared hosting.

I had imagined it working on a file by file level rather than a sitewide one. e.g /assets/main.css needs updating. It gets uploaded as /assets/main.css.tmp (or something similar, with checks ensuring it doesn't clash with any other files). Once the upload has finished the file gets renamed and overwrites the older version.

[martinheidegger]

The larger the site becomes the more important atomic updates become imho. With a small site all files are updated in a very short time, but the bigger a site gets the higher the chance for cross-file-dependencies (specially within js/html logic) to create inconsistent states(broken homepages). Which in turn leads to 4am updates to not avoid problems for the enduser (the sort of thing I wish to avoid). Anyways: I got your message: Use Capistrano. ;)

[scttnlsn]

I think this is a bit beyond the scope of Dandelion. Is there anything keeping you from scripting the atomic upload yourself while still utilizing Dandelion for the file transfer? Something like:

• duplicate remote directory structure to new location (cp www www_new)
• dandelion deploy to that new location (path: www_new), only changes are uploaded
• rename the old directory (mv www www_old)
• rename the new directory (mv www_new www)