search

spring-attic / spring-hadoop-samples

Spring Hadoop Samples
Apache License 2.0
492 stars 466 forks source link

mapreduce on CDH5: not enough memory for Yarn #13

Closed renaud closed 10 years ago

renaud commented 10 years ago

I had the following error when trying to run the mapreduce example on CDH5.

InvalidResourceRequestException Invalid resource request, 
requested memory < 0, or  requested memory > max configured,
requestedMemory=1536, maxMemory=1024

I had to do the following fix to make it work:

Hue > Yarn > Configuration > Edit Search for "valve", set property YARN Service Advanced Configuration Snippet (Safety Valve) for yarn-site.xml to:

<property>
    <name>yarn.scheduler.maximum-allocation-mb</name>
    <value>2024</value>
 </property>

Save changes and redeploy.

trisberg commented 10 years ago

Hi, thanks for trying this. I know I tested the sample on the CDH5 Quickstart VM and don't think I changed that setting. Did you use the Quickstart VM or did you use your own install?

renaud commented 10 years ago

Thanks Thomas for writing back. I used the Quickstart VM.

trisberg commented 10 years ago

Strange. Looking at my config -

Container Memory Minimum 1 GiB default value yarn.scheduler.minimum-allocation-mb

So my mapreduce example runs in less than 1GB.

I assume you have the latest with the Spring for Apache Hadoop 2.0 and build using the cdh5 profile:

mvn clean package -Pcdh5 sh target/appassembler/bin/wordcount

renaud commented 10 years ago

Yes, I did build with the cdh5 profile. So if you can't reproduce let's close this issue. It's working on my end, so all good. Thanks, Renaud

trisberg commented 10 years ago

I just tried a new install of the Quickstart VM and I'm able to reproduce this - must have made some config change on the other VM I tried - I'll add a note in the instructions.

trisberg commented 10 years ago

Thanks for reporting this.

I have updated the README

renaud commented 10 years ago

Ok. Thanks too for the prompt reply!