search

Shopify / camus

Kafka->HDFS pipeline from LInkedIn. It is a mapreduce job that does distributed data loads out of Kafka.
7 stars 4 forks source link

Bump heap to 2gb #37

Closed yagnik closed 9 years ago

yagnik commented 9 years ago

Camus is failing due to heap space, giving it more memory. @datwright @kmtaylor-github

datwright commented 9 years ago

Because of failed camus run here that ran out of heap space

datwright commented 9 years ago

:ship:

datwright commented 9 years ago

@yagnik the following run finished up successfully. Maybe this isn't necessary? https://azkaban.data.shopify.com/manager?project=Camus&flow=Camus#executions

yagnik commented 9 years ago

Yes it only happens once ina while when a lot of data is being loaded in one go, it's better to bump it for future.

kmtaylor-github commented 9 years ago

Leaving the asymmetric initial 1g vs. 2g max should help to keep the heap size down unless needed (sort of).

Go for it.

drdee commented 9 years ago

Why do you think it's on the driver? Couldn't it be a mapper?