search

gateway-experiments / hadoop-yarn-api-python-client

Python client for Hadoop® YARN API
https://yarn-api-client-python.readthedocs.io/en/latest/
BSD 3-Clause "New" or "Revised" License
109 stars 48 forks source link

Automate YARN Integration Tests #18

Open lresende opened 5 years ago

lresende commented 5 years ago

The existing integration tests require a running YARN server to execute. We could integrate with a YARN docker image to run these tests as part of the CI build.

dimon222 commented 4 years ago

I've done some research after recent accidental findings of #75 since it does seem to feel that we suffer from not being able to run tests on actual real YARN environment.

One of implementations of docker that worked for me seems to be https://github.com/flokkr/docker-hadoop (after small tuning in https://github.com/flokkr/docker-hadoop/pull/7)

It includes full-pledged stack of hadoop (so that means Hadoop & Yarn). I was able to run compose and bring up the working Yarn within 1 minute without much extra actions on Windows (perhaps that means that on Linux/OSX its even much easier). Its full stack, so we thinking about quite plenty of RAM. I'm not sure how feasible its for usage in integration tests on free public CI platforms unless we're considering $$$.