When working with the local backend, the output is copied over from the tmp folder to the desired output location using file.rename, which is brittle (does not work sometimes, when the tmp folder is on a different file system, for example).
Should probably be replaced with a combination of file.copy and then deleting the temp file using file.remove (similar to how it's done for the hadoop backed).
When working with the local backend, the output is copied over from the tmp folder to the desired output location using file.rename, which is brittle (does not work sometimes, when the tmp folder is on a different file system, for example).
Should probably be replaced with a combination of file.copy and then deleting the temp file using file.remove (similar to how it's done for the hadoop backed).