Delete temp and result directories in local mode

[GoogleCodeExporter]

What steps will reproduce the problem?
1. Run a job with the engine in local mode
2. Delete the job

The result files and temp files are persisted. There should be an option (in 
system.properties?) to enable handling of result directories even in local mode.

- temp files should be deleted right after the job has finished executing
- result files should be deleted when the job is deleted

Original issue reported on code.google.com by josteinaj@gmail.com on 26 Apr 2013 at 7:41

[GoogleCodeExporter]

See also discussion on issue 303

Original comment by josteinaj@gmail.com on 26 Apr 2013 at 7:41

[GoogleCodeExporter]

Hum, what if I usually use as my output-dir / ? or ~javi? there is no way, from 
the fwk point of view, to know which files where created by the underlying 
script or where already there. 

I guess something similar happens to temp, If this derives from issues with the 
webui, what was the reason again to start the pipeline in local mode instead of 
remote mode? There are more features in remote mode that would make sense for 
the webui such as deletion and fine granular result retrieval. 

Original comment by capitan....@gmail.com on 26 Apr 2013 at 8:59

[GoogleCodeExporter]

The reason for using local mode is that files can be passed by reference (file: 
URIs) instead of zipping them and including them in POST requests.

If result and temp files were enabled in local mode, I think you would not be 
able to specify them as options in the jobRequest; they would be set 
automatically by the framework just like in remote mode. So then you would be 
unable to use / or ~javi as output-dirs.

This would also enable (and require?) the framework to provide access to the 
result files through the Web API, like it's done in remote mode.

Original comment by josteinaj@gmail.com on 26 Apr 2013 at 9:10

[GoogleCodeExporter]

I suppose another way to look at what I'm requesting is "remote mode with the 
ability to provide file: URIs"

Original comment by josteinaj@gmail.com on 26 Apr 2013 at 9:13

[GoogleCodeExporter]

1. But the user is already providing you the zip file, right?

2. I guess we could do that, although the user loses the control over that
parts of the local mode.

Original comment by capitan....@gmail.com on 26 Apr 2013 at 9:15

[GoogleCodeExporter]

1. The user can provide both zip files and unzipped files, but either way they 
are uploaded to the Web UI and not the framework. When the job is created, the 
files that were uploaded are unzipped to a context directory (system temp 
directory), and the context directory are then either passed by reference to 
the Web API if it is running in local mode, or if the Web API is running in 
remote mode, it is zipped and POSTed, and the engine has to unzip it again. In 
cases where only one file were uploaded and that file were a zip file, then 
that file can be POSTed directly, but it will still take time to POST the ZIP 
file even though the webui and fwk are running on the same computer, and it 
probably takes a bit of time for the fwk to unzip it as well. In addition, it 
requires that yet another copy of the fileset is stored on the filesystem.

2. Alternatively, we could say that if the client specifies a temp/result 
directory, then the client has to manage the directory itself, but if it's not 
specified then the fwk will handle it for you?

Original comment by josteinaj@gmail.com on 26 Apr 2013 at 9:23

[GoogleCodeExporter]

Yep it looks like that :P hehehe. I personally would prefer to leave local
mode and remote mode as they are, so if you want full control in a local
environment go for local, if you want the fwk take care of your outputs go
for remote. Although I can see why what you ask is handy... I'm not sure if
adding an extra mode is a perfect scenario, that would complicate how
clients interact with the fwk. As romain is not watching I'd say "I dont
have a strong opinion" let me think a little bit about it.

Original comment by capitan....@gmail.com on 26 Apr 2013 at 9:25

[GoogleCodeExporter]

imo. there is less and less reason to distinguish between remote and local mode.

Original comment by josteinaj@gmail.com on 26 Apr 2013 at 9:30

[GoogleCodeExporter]

In Copenhagen we decided to merge local and remote modes, which makes this a 
non-issue. I'm closing it.

Original comment by josteinaj@gmail.com on 26 Jun 2013 at 8:52

• Changed state: WontFix