search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.83k stars 2k forks source link

h2o.cor memory leak #7175

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

I have found that repeated calls to h2o.cor permanently reduces available memory, even when trying force garbage collection.

Use-case: I was trying to implement the base R stats::cor(..., use = "pairwise.complete.obs") option, which requires numerous calls to h2o.cor, and I was finding that I was running out of memory.

{code:r}

library(data.table) data.table 1.14.2 using 4 threads (see ?getDTthreads). Latest news: r-datatable.com library(h2o)

Your next step is to start H2O:

h2o.init()

For H2O package documentation, ask for help:

??h2o

After starting H2O, you can use the Web UI at http://localhost:54321 For more information visit https://docs.h2o.ai

Attaching package: ‘h2o’

The following objects are masked from ‘package:data.table’:

hour, month, week, year

The following objects are masked from ‘package:stats’:

cor, sd, var

The following objects are masked from ‘package:base’:

%*%, %in%, &&, ||, apply, as.factor, as.numeric, colnames, colnames<-, ifelse, is.character, is.factor, is.numeric, log, log10, log1p, log2, round,
signif, trunc

devtools::session_info()

  • Session info ---------------------------------------------------------------------------------------------------------------------------------------------------------- setting value version R version 4.1.2 (2021-11-01) os Windows 10 x64 (build 22000) system x86_64, mingw32 ui RStudio language (EN) collate English_United States.1252 ctype English_United States.1252 tz America/Chicago date 2022-01-05 rstudio 2021.09.0+351 Ghost Orchid (desktop) pandoc NA

h2o.init()

H2O is not running yet, starting it now...

Note: In case of errors look at the following log files: C:\Users\donne\AppData\Local\Temp\Rtmp8IrKXv\file39cccda1a1d/h2o_donne_started_from_r.out C:\Users\donne\AppData\Local\Temp\Rtmp8IrKXv\file39cc10247f5a/h2o_donne_started_from_r.err

java version "11.0.10" 2021-01-19 LTS Java(TM) SE Runtime Environment 18.9 (build 11.0.10+8-LTS-162) Java HotSpot(TM) 64-Bit Server VM 18.9 (build 11.0.10+8-LTS-162, mixed mode)

Starting H2O JVM and connecting: . Connection successful!

R is connected to the H2O cluster: H2O cluster uptime: 5 seconds 610 milliseconds H2O cluster timezone: America/Chicago H2O data parsing timezone: UTC H2O cluster version: 3.36.0.1 H2O cluster version age: 7 days, 2 hours and 58 minutes
H2O cluster name: H2O_started_from_R_donne_aln969 H2O cluster total nodes: 1 H2O cluster total memory: 1.97 GB H2O cluster total cores: 8 H2O cluster allowed cores: 8 H2O cluster healthy: TRUE H2O Connection ip: localhost H2O Connection port: 54321 H2O Connection proxy: NA H2O Internal Security: FALSE H2O API Extensions: Amazon S3, Algos, Infogram, AutoML, Core V3, TargetEncoder, Core V4 R Version: R version 4.1.2 (2021-11-01)

songs <- h2o.importFile("https://s3.amazonaws.com/h2o-public-test-data/bigdata/laptop/testng/milsongs-train.csv.gz") |=================================================================================================================================================================| 100% gc(); gc(); h2o:::.h2o.garbageCollect() used (Mb) gc trigger (Mb) max used (Mb) Ncells 976846 52.2 2007050 107.2 2007050 107.2 Vcells 1558518 11.9 8388608 64.0 2419104 18.5 used (Mb) gc trigger (Mb) max used (Mb) Ncells 976847 52.2 2007050 107.2 2007050 107.2 Vcells 1558537 11.9 8388608 64.0 2419104 18.5 print(unlist(h2o.clusterStatus()$free_mem)/1024^3) Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.791814

for(ii in 1:10){

  • invisible(h2o.cor(x = songs[, paste0("C", 1:10)], na.rm = TRUE, use = "complete.obs"))
  • gc(); gc(); h2o:::.h2o.garbageCollect()
  • print(unlist(h2o.clusterStatus()$free_mem)/1024^3)
  • } Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.791814 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.727581 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.664097 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.600744 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.537389 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.284952 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.410875 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.347645 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.220971 Version: 3.36.0.1 Cluster name: H2O_started_from_R_donne_aln969 Cluster size: 1 Cluster is locked

free_mem 1.220971 {code}

h2o-ops-ro commented 1 year ago

JIRA Issue Details

Jira Issue: PUBDEV-8487 Assignee: Michal Kurka Reporter: N/A State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A