search

hpcugent / hanythingondemand

hanythingondemand provides a set of scripts to easily set up an ad-hoc Hadoop cluster through PBS jobs
https://hod.readthedocs.org
GNU General Public License v2.0
12 stars 6 forks source link

Make spark use /tmp for local dir. #141

Closed ehiggs closed 8 years ago

ehiggs commented 8 years ago

Spark normally takes the yarn local dir. However, it does this only from the local dir on the master node. This means all the spark executors on all the nodes want to talk to the one hod localworkdir used by the master. This commit changes this to use /tmp

Another solution would have been to set HOD_LOCALWORKDIR in the environments and use {{HOD_LOCALWORKDIR}} and let spark's templating resolve it later.

hpcugentbot commented 8 years ago

Test FAILed.

See https://jenkins1.ugent.be/job/hanythingondemand-pr-builder/335/console for more details.

hpcugentbot commented 8 years ago

Test PASSed (test).

See https://jenkins1.ugent.be/job/hanythingondemand-pr-builder/336/console for more details.

boegel commented 8 years ago

I suggest you test this first at length before we merge it?

hpcugentbot commented 8 years ago

Test PASSed (test).

See https://jenkins1.ugent.be/job/hanythingondemand-pr-builder/337/console for more details.

ehiggs commented 8 years ago

This has been tested now. It seems to work v. well. :rocket:

boegel commented 8 years ago

Thanks @ehiggs.