Python client for Apache Hadoop® YARN API
Package documentation: yarn-api-client-python.readthedocs.org
REST API documentation: hadoop.apache.org
Warning: CLI is outdated & broken. Please don't use CLI. This will be resolved in future releases.
yarn-api-client-python | Apache Hadoop |
---|---|
1.0.2 | 3.2.1 |
1.0.3 | 3.3.0, 3.3.1 |
If u have version other than mentioned (or vendored variant like Hortonworks), certain APIs might be not working or have differences in implementation. If u plan to use certain API long-term, you might want to make sure its not in Alpha stage in documentation.
From PyPI
pip install yarn-api-client
From Anaconda (conda forge)
conda install -c conda-forge yarn-api-client
From source code
pip install git+https://github.com/CODAIT/hadoop-yarn-api-python-client.git
See example below:
from yarn_api_client.auth import SimpleAuth
from yarn_api_client.history_server import HistoryServer
auth = SimpleAuth('impersonated_account_name')
history_server = HistoryServer('https://127.0.0.2:5678', auth=auth)
requests_kerberos
package To avoid deployment issues on a non Kerberized environment, the requests_kerberos
dependency is optional and needs to be explicit installed in order to enable access
to YARN console protected by Kerberos/SPNEGO.
pip install requests_kerberos
From python code
from yarn_api_client.history_server import HistoryServer
from requests_kerberos import HTTPKerberosAuth
history_server = HistoryServer('https://127.0.0.2:5678', auth=HTTPKerberosAuth())
PS: You need to get valid kerberos ticket in systemwide kerberos cache before running your code, otherwise calls to kerberized environment won't go through (run kinit before proceeding to run code)
gssapi
package If you want to avoid using terminal calls, you have to perform SPNEGO handshake to retrieve ticket yourself. Full API documentation: https://pythongssapi.github.io/python-gssapi/latest/
Warning: CLI is outdated & broken. Please don't use CLI. This will be resolved in future releases.
First way
bin/yarn_client --help
Alternative way
python -m yarn_api_client --help
from yarn_api_client import ApplicationMaster, HistoryServer, NodeManager, ResourceManager
am = ApplicationMaster('https://127.0.0.2:5678')
app_information = am.application_information('application_id')
1.0.3 Release
1.0.2 Release
1.0.1 Release
1.0.0 Release
Documentation has been updated.
NOTE: Applications using APIs relative to releases prior to 1.0 should pin their dependency on yarn-api-client to less than 1.0 and are encouraged to update to 1.0 as soon as possible.
0.3.7 Release
0.3.6 Release
0.3.5 Release
0.3.4 Release
0.3.3 Release
0.3.2 Release
0.3.1 Release
0.3.0 Release
requests
package for REST API invocationhttp_con
property, as connections are now managed by requests
package0.2.5 Release
0.2.4 Release
YARN API client is developed by an open community, and the current maintainers are listed below in alphabetical order: