Temporary files pile-up

[emillykkejensen]

I have a repeat loop, where I continuously gather, analyze and index data on one server (using R-studio server) and then store it on another elasticsearch server.

The problem is, that the temp dir on my R-Studio server fills up and after a weeks’ time all of my disk space is used up – resulting in all hell breaking loose.

So is there a way to fix this? Found a possible fix in the raster package with removeTmpFiles() (https://whalendillon.wordpress.com/2014/09/11/temporary-files-pile-up-while-using-the-raster-package-in-r/) but haven’t tried it out yet, as I would think it might be a problem this packages should address out of the box?

[sckott]

What version do you have? I think should be fixed via https://github.com/ropensci/elastic/issues/188

[emillykkejensen]

I'm using the CRAN version 0.8

[sckott]

Try installing latest here from github devtools::install_github("ropensci/elastic") let me know if that fixes it

[emillykkejensen]

Okay - I'll try the github package and will get back to you in 24 hours or so to see if it is fixed :-)

[emillykkejensen]

Just had a look at my disk usage and unfortunately there still is a steady increase due to a file pile-up in the ‘/tmp/RtmpBOjw9w’ temp folder.

[sckott]

can you paste in the output of sessionInfo() so I can see what version of elastic you are running

[emillykkejensen]

The output of sessionInfo() on my R-studio Server:

R version 3.4.3 (2017-11-30)
Platform: x86_64-pc-linux-gnu (64-bit)
Running under: Ubuntu 16.04.3 LTS

Matrix products: default
BLAS: /usr/lib/libblas/libblas.so.3.6.0
LAPACK: /usr/lib/lapack/liblapack.so.3.6.0

locale:
 [1] LC_CTYPE=en_US.UTF-8       LC_NUMERIC=C               LC_TIME=en_US.UTF-8        LC_COLLATE=en_US.UTF-8    
 [5] LC_MONETARY=en_US.UTF-8    LC_MESSAGES=en_US.UTF-8    LC_PAPER=en_US.UTF-8       LC_NAME=C                 
 [9] LC_ADDRESS=C               LC_TELEPHONE=C             LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C       

attached base packages:
[1] stats     graphics  grDevices utils     datasets  methods   base     

other attached packages:
 [1] textreuse_0.1.4     rlist_0.4.6.1       xml2_1.1.1          httr_1.3.1          lubridate_1.7.1     elastic_0.8.0.9100 
 [7] TextFunc_0.1.0      data.table_1.10.4-3 pool_0.1.3          RMySQL_0.10.13      DBI_0.7            

loaded via a namespace (and not attached):
 [1] Rcpp_0.12.14     bindr_0.1        magrittr_1.5     devtools_1.13.4  R6_2.2.2         rlang_0.1.6      stringr_1.2.0   
 [8] dplyr_0.7.4      RcppProgress_0.4 tools_3.4.3      withr_2.1.1      dbplyr_1.2.0     digest_0.6.14    assertthat_0.2.0
[15] tibble_1.4.1     NLP_0.1-11       bindrcpp_0.2     later_0.6        curl_3.1         memoise_1.1.0    glue_1.2.0      
[22] stringi_1.1.6    compiler_3.4.3   pillar_1.1.0     jsonlite_1.5     pkgconfig_2.0.1 

Version info running elastic::info():

$version
$version$number
[1] "5.6.6"

and

$version$lucene_version
[1] "6.6.1"

[sckott]

okay, i'll look into it.

I realize that I don't know what functions you're using - are you using docs_bulk by chance?

[emillykkejensen]

Yes I'm using docs_bulk :)

[sckott]

thanks

[sckott]

seems like you must not be restarting the R session much since temp files are cleaned up on the session ending?

[emillykkejensen]

Nop - no restart. It runs in a repeat loop continuously.

[sckott]

@emillykkejensen try reinstalling devtools::install_github("ropensci/elastic") and trying again, i guess though if you don't restart your session those temp files currently there will still be there - unless you clean them up manually

[emillykkejensen]

Okay - I'll try reinstalling and see if it fixes it.

[emillykkejensen]

Perfect - that did it - it no longer builds up the temp dir.

Problem solved!

[sckott]

great, thx