search

h2oai / h2o3-sagemaker

Integrating H2O-3 AutoML with Amazon Sagemaker
Apache License 2.0
14 stars 13 forks source link

Sagemaker hosting instance runs out of diskspace #6

Open bchalamayya opened 5 years ago

bchalamayya commented 5 years ago

Hi Team,

We used this docker file to build and train model using Amazon Sagemaker. After hosting the model and endpoint using the docker image, We get issues as the hosting instance disk is full. Not sure about the files it generates to fill the disk space. AWS support team said that they cant do anything for a custom image. Any pointers on this issue would be appreciated Thank you

nkpng2k commented 5 years ago

@bchalamayya, typically you can customize the amount of disk space you have on the instance. Which image are you using? can you share some more information?

Typically speaking, everything on the H2O-3 platform is performed in memory, so there should not be anything being written to disk unless it is specifically told to do so.

nkpng2k commented 5 years ago

if you are talking about the algorithms hosted on sagemaker marketplace, it is likely that aws is copying the datasets provided into the instance which is where this issue is arising. We read in the data from disk to memory and then do all our work in memory, and you would need to provide more disk space to the sagemaker instance.

In the future we might be able to ingest the dataset directly from s3 eliminating the need to have sagemaker copy the data to disk first.