search

gvegayon / parallel

PARALLEL: Stata module for parallel computing
https://rawgit.com/gvegayon/parallel/master/ado/parallel.html
MIT License
118 stars 26 forks source link

Removing log messages in tmpdir #11

Closed bquistorff closed 8 years ago

bquistorff commented 9 years ago

In parallel_recursively_rm.mata, it does not "remove logfiles in tmpdir". I understand wanting to keep them around so the user can scrutinize them. But we should delete them at some point because (a) that's correct usage of temp files (otherwise they should be normal files in one of the user's main directories), and (b) they can, in aggregate, take up lots of space (especially if trace is on).

I definitely think -parallel clean, all- should delete them by default. Normal usage shouldn't shouldn't create clutter. One option then is to consider them like the other files during normal operation (delete, keep, or keeplast) unless there is an error. If so we could (a) copy them out to pwd before treating them like the other files, or (b) leave in tmpdir (they could inspect them, copy them out, or disregard) and require the user to -parallel clean-.

We could have an option in setclusters to enable the current behavior and then provide another option to -parallel clean- to delete the normally kept logs.

What do you think?

bquistorff commented 9 years ago

I've put in a partial/most fix at bd14449f4cc8d3078fea68cedef918438678b683 so that recursive_rm takes another option of rmlogs. When called from -parallel clean- calls recursive_rm it is enabled. So the logs are kept after a normal run, but deleted upon clean. Hope that works.

gvegayon commented 9 years ago

In what OS + Stata versions did you test this?? I'll try it in RedHat

bquistorff commented 9 years ago

I tried this on Windows Stata 13 (built on Stata 12.1).