Clean up output on raijin

[dfalster]

Running simulation via PBS script on raijin leaves 3 files in place where experimentr was launched:

• pbs launch script
• file with output from stdout
• file with summary stats of resources used

Could make a function to move these into output directory for that experiment? e.g. For experiment called "seed_size_initial_exploration", need to run

mkdir experiments/output/seed_size_initial_exploration/jobs
mv seed_size_init*  experiments/output/seed_size_initial_exploration/jobs/ 

Also need to figure out how we can link files back to task, as name of job gets shortened and task if gets dropped

[richfitz]

From what you copied on richfitz/tree#85, we could use this information:

Experiment: seed_size_initial_exploration
Task:       simulations
Id:         2
StdOut stored in seed_size_initial_exploration/jobs/seed_size_initi.e6624580

It's a bit of a pain that things get trimmed in l length. But it will be easy enough to go through and wire these up together I'm sure.

[dfalster]

yep, is possible. Although I added the last line manually.

What we'd need to do is one of following:

1. Look in some random file named something like: compare_assembl.e6616315
2. Parse first few lines which give experiment and task
3. Move to right place

Alternatively,

1. when calling qsub, could catch job number that it returns and associate this with task
2. Use saved job number to get output files

On 12 September 2014 07:53, Rich FitzJohn notifications@github.com wrote:

From what you copied on richfitz/tree#85 https://github.com/richfitz/tree/issues/85, we could use this information:

Experiment: seed_size_initial_exploration Task: simulations Id: 2 StdOut stored in seed_size_initial_exploration/jobs/seed_size_initi.e6624580

It's a bit of a pain that things get trimmed in l length. But it will be easy enough to go through and wire these up together I'm sure.

— Reply to this email directly or view it on GitHub https://github.com/richfitz/experimentr/issues/16#issuecomment-55334650.

[richfitz]

Waiting for dfalster/successional_diversity#14, so that we have fresh output to work with.

[richfitz]

Next batch we set off, let's try the stuff I put into place in f75405d0f33ad8791992eb3b757e2f1001dd4283.

[dfalster]

Ok. I've also been writing a function to move output files