search

h2oai / h2o-3

H2O is an Open Source, Distributed, Fast & Scalable Machine Learning Platform: Deep Learning, Gradient Boosting (GBM) & XGBoost, Random Forest, Generalized Linear Modeling (GLM with Elastic Net), K-Means, PCA, Generalized Additive Models (GAM), RuleFit, Support Vector Machine (SVM), Stacked Ensembles, Automatic Machine Learning (AutoML), etc.
http://h2o.ai
Apache License 2.0
6.94k stars 2k forks source link

Parse should not accept files larger than available memory #14045

Open exalate-issue-sync[bot] opened 1 year ago

exalate-issue-sync[bot] commented 1 year ago

Using latest h2o-dev as of today.

mdowle@MattsMac:~$ R

R version 3.2.0 (2015-04-16) -- "Full of Ingredients" Copyright (C) 2015 The R Foundation for Statistical Computing Platform: x86_64-pc-linux-gnu (64-bit)

R is free software and comes with ABSOLUTELY NO WARRANTY. You are welcome to redistribute it under certain conditions. Type 'license()' or 'licence()' for distribution details.

Natural language support but running in an English locale

R is a collaborative project with many contributors. Type 'contributors()' for more information and 'citation()' on how to cite R or R packages in publications.

Type 'demo()' for some demos, 'help()' for on-line help, or 'help.start()' for an HTML browser interface to help. Type 'q()' to quit R.

require(h2o) Loading required package: h2o Loading required package: statmod

Your next step is to start H2O and get a connection object (named 'localH2O', for example):

localH2O = h2o.init()

For H2O package documentation, ask for help:

??h2o

After starting H2O, you can use the Web UI at http://localhost:54321 For more information visit http://docs.0xdata.com

h2oServer <- h2o.init(nthreads=-1)

H2O is not running yet, starting it now...

Note: In case of errors look at the following log files: /tmp/Rtmpig7DSz/h2o_mdowle_started_from_r.out /tmp/Rtmpig7DSz/h2o_mdowle_started_from_r.err

Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar java version "1.7.0_80" Java(TM) SE Runtime Environment (build 1.7.0_80-b15) Java HotSpot(TM) 64-Bit Server VM (build 24.80-b11, mixed mode)

Successfully connected to http://127.0.0.1:54321/

R is connected to H2O cluster: H2O cluster uptime: 966 milliseconds H2O cluster version: 0.3.0.99999 H2O cluster name: H2O_started_from_R H2O cluster total nodes: 1 H2O cluster total memory: 3.46 GB H2O cluster total cores: 8 H2O cluster allowed cores: 8 H2O cluster healthy: TRUE

system.time(DF <- h2o.importFile(h2oServer, path = "/tmp/test.csv")) |==================== | 29%Error in .h2o.doSafeREST(conn = conn, h2oRestApiVersion = h2oRestApiVersion, : Unexpected CURL error: Failed to connect to 127.0.0.1 port 54321: Connection refused Timing stopped at: 0.12 0.004 25.877 system("ls -lh /tmp/test.csv") -rw-rw-r-- 1 mdowle mdowle 9.0G May 5 21:17 /tmp/test.csv system("wc -l /tmp/test.csv") 201000001 /tmp/test.csv system("head /tmp/test.csv") id1,id2,id3,id4,id5,id6,v1,v2,v3 id092,id072,id0000003994,62,72,7044,5,4,63.4882 id097,id016,id0000008706,70,51,3016,1,2,33.1901 id020,id047,id0000008414,22,78,1703,3,5,80.5984 id031,id019,id0000005250,32,6,1101,5,2,34.6695 id085,id015,id0000008528,64,29,4414,4,2,94.5902 id032,id042,id0000001955,67,81,1239,3,5,12.4556 id008,id075,id0000000694,46,79,4291,1,1,98.5869 id001,id055,id0000009733,5,65,8768,5,5,83.1582 id082,id045,id0000008240,38,61,9249,3,3,93.2447

exalate-issue-sync[bot] commented 1 year ago

Matt Dowle commented: I retried and the same problem occurred at 35% in to reading the file. I have 16GB RAM of which about 5GB was being used at the time of failing according to htop. Seems like I had plenty of spare RAM. I could see 8 cores busy according to htop.

exalate-issue-sync[bot] commented 1 year ago

Spencer Aiello commented: heyo matt, is it possible for you to scp test.csv to one of the machines?

if you don't have credentials yet, grab some from Kevin :)

or can just scp it to my home dir on 181 e.g.

exalate-issue-sync[bot] commented 1 year ago

Tom Kraljevic commented: Matt, this is part user-error, part not getting a good message back. Java won't use all the mem on your system. It will use as much as you tell it to when you start up h2o.

In this case:

H2O cluster total memory: 3.46 GB
exalate-issue-sync[bot] commented 1 year ago

Spencer Aiello commented: I think we should have this one piggy-back onto PUBDEV-905

exalate-issue-sync[bot] commented 1 year ago

Brandon Hill commented: Agreed. The short term solution should at least reject the parse job. The long term solution should be PUBDEV-905. The difference being that PUBDEV-905 can try reading and compressing, and only give up when death is certain. Memory performance of the parse will be improving, so this will be feasible in time.

DinukaH2O commented 1 year ago

JIRA Issue Migration Info

Jira Issue: PUBDEV-1065 Assignee: Brandon Hill Reporter: Matt Dowle State: Open Fix Version: N/A Attachments: N/A Development PRs: N/A