Temp files are left on disk

[thongsav-usgs]

Hi, I have not done a thorough review of the code, but we are using rmarkdown as part of a rendering engine for a live web application and are finding that the disk on the servers fill up with rmarkdown files. I have stepped through the code enough just to know that there are instances where tempfile (EG: https://github.com/rstudio/rmarkdown/blob/master/R/html_resources.R#L186) is called, but have not gone further to see if rmarkdown already attempts to clean them up.

If rmarkdown does not clean up, I would be happy if the temp files were completely hidden from the user and cleaned up in the background, but would also be OK with the option of passing in the directory to use for all tempfile calls.

[yihui]

Typically temp files are cleaned up in on.exit(), but there might be a few places where we forgot to clean them up. If you can tell us the patterns of these temp files (filenames/content), we may be able to find out where they were created.

[thongsav-usgs]

Ah, I see the on exit line below. The temp files come from the MD file creation 5 lines below the line I linked. I think that explains it. We are running rmarkdown via Rserve, and I don't believe the rserve process would ever exit and trigger that event.

I imagine this may become a more common pattern as Rserve becomes more common, but it is not an inherent problem with the cleanup strategy you've implemented.

Let me know if you have other questions about our use case, but I hesitate to ask for specific changes without knowing the full workflow rmarkdown takes.

[yihui]

on.exit() is executed when the function call is finished, and it does not require the R process to exit.

[thongsav-usgs]

Hmm, in that case here are some files left over: rmarkdown-str[hash].html file[hash] directories called file[hash] with various javascript/css lib files used by our reports

I can confirm that they are created inside the method referenced above.

(Sorry, I'm new to R and did not look at the documentation on on.exit until now)

[yihui]

I just submitted a fix in #820 that will clean up the rmarkdown-str[hash].html files. I'm not sure what file[hash] files/dirs are. Can you install the development version of knitr and the patched version of rmarkdown via

devtools::install_github(c('yihui/knitr', 'yihui/rmarkdown@bugfix/clean-tempfiles'))

and see which temp files are still left over? Thanks!

[thongsav-usgs]

That did take care of the "-str" files, but I still have one file[hash] file and one file[hash]_files folder, with bootstrap, highlight, jquery, and navigation libs.

[yihui]

Okay, could you provide a minimal, self-contained, and reproducible example that shows the problem after you call rmarkdown::render()?

[thongsav-usgs]

Here's a simple one to leave a file[hash] file. Use an Rmd file with the following and call render(the_rmd_file):

```{r echo=FALSE, warning=FALSE, message=FALSE, results='asis'}
```{r echo=FALSE, warning=FALSE, message=FALSE, dev='svg', fig.height=10, fig.width=13, results='hide'}

[thongsav-usgs]

Here's how to reproduce the folder containing javascript/css files, we have a default.yaml file with the following contents:

xaxs: "i"
yaxs: "i"
tcl: 0.3
mgp: [1.5, 0.3, 0.0]
yaxt: "s"
xaxt: "s"
points:
  pch: 6
  col: "red"
lines: 
  col: "black"
  lty: 1
  lwd: 1
abline:
  col: "grey"
  lty: 1
  lwd: 1
axis:
  at: NULL
  labels: TRUE
  tick: TRUE 
  line: NULL 
  pos: NULL
  outer: FALSE
  las: 1
title: NULL
text: NULL
mtext: 
  las: 0
grid:
  col: "grey"
  lwd: 1
  lty: 2
segments: NULL
error_bar:
  y.high: 0 
  y.low: 0 
  x.high: 0
  x.low: 0
  epsilon: 0.1
arrows: NULL
callouts:
  col: 'black'
bgCol: 
  col: 'lightgrey'

run render(your_rmd_file, intermediates_dir=dir_containing_default_yaml)

[jjallaire]

It may be that the intermediates_dir isn't cleaned up by design (I'm not familiar enough w/ the runtime: shiny features it supports to say for sure)

In some cases we might consider it risky to do this file cleanup without a full inspection of all code paths which might be affected. It could be a while before we do this.

I'd suggest this workaround: from your long running Rserve process just call system("R -e ...") for your rmarkdown renders rather than doing them in process. All temp files will be cleaned up after each render.

On Fri, Sep 30, 2016 at 3:57 PM, thongsav-usgs notifications@github.com wrote:

Here's how to reproduce the folder containing javascript/css files, we have a default.yaml file with the following contents:

xaxs: "i" yaxs: "i" tcl: 0.3 mgp: [1.5, 0.3, 0.0] yaxt: "s" xaxt: "s" points: pch: 6 col: "red" lines: col: "black" lty: 1 lwd: 1 abline: col: "grey" lty: 1 lwd: 1 axis: at: NULL labels: TRUE tick: TRUE line: NULL pos: NULL outer: FALSE las: 1 title: NULL text: NULL mtext: las: 0 grid: col: "grey" lwd: 1 lty: 2 segments: NULL error_bar: y.high: 0 y.low: 0 x.high: 0 x.low: 0 epsilon: 0.1 arrows: NULL callouts: col: 'black' bgCol: col: 'lightgrey'

run render(your_rmd_file, intermediates_dir=dir_containing_default_yaml)

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/rstudio/rmarkdown/issues/819#issuecomment-250837705, or mute the thread https://github.com/notifications/unsubscribe-auth/AAGXxwwJ9sZs85gqW2NhM5Rx53Lv_qTJks5qvWmMgaJpZM4KKBBU .

[yihui]

@thongsav-usgs Could you simply upload the reproducible example as a zip archive here on Github? I don't really understand what these two examples mean. The first one is two empty R code chunks, and I don't know what default.yaml in the second example means.

[thongsav-usgs]

The Rmd file does next to nothing and produces a blank HTML file. I'm also not really sure how the default.yaml works or if it's even the cause. It could simply be the use of the intermediates_dir above. In either case, that is the full example.

We do have a work around in place so we are not blocked by this.

[yihui]

I need to see the full Rmd file (e.g. whether you have runtime: shiny here), so the best way is to upload your reproducible example here instead of verbally describing what it contains.

[thongsav-usgs]

Let me consult some of my colleagues that know more about what's going on in our code so I can give you an answer that probably makes more sense.

I did not originally create/upload files because those code blocks are literally the entire contents of the Rmd/yaml files I'm using to reproduce the problem.

As far as shiney, I don not think we are using that, but I'd have to verify. Will likely not have an answer until Monday.

[github-actions[bot]]

This old thread has been automatically locked. If you think you have found something related to this, please open a new issue by following the issue guide (https://yihui.org/issue/), and link to this old issue if necessary.