search

amundsen-io / amundsen

Amundsen is a metadata driven application for improving the productivity of data analysts, data scientists and engineers when interacting with data.
https://www.amundsen.io/amundsen/
Apache License 2.0
4.44k stars 961 forks source link

Hive Metadata Extractor Configuration #832

Closed vthorwat closed 3 years ago

vthorwat commented 3 years ago

Hello Guys,

I have just configured Amundsen via Docker with Apache Atlas backend by using below Blog:

Link: https://github.com/amundsen-io/amundsen/blob/master/docs/installation.md

Now I want to use it with hive service of our HDP (Hortonworks) cluster. But not able to locate any configurations for that.

After looking some issues I found that there is a SLACK Channel for Amundsen, is it an invite based channel?. If yes then please add me. (vinodthorwat@gmail.com)

HDP Stack : HDP-2.6.5.115 HIve Version: 1.2.1000

hive-site.xml hive.server2.authentication = NONE hive.metastore.uris = thrift://10.x.x.x:9083 hive.server2.transport.mode = binary

Current logs appearing on the terminal When we visit http://127.0.0.1:5000 : 


amundsenfrontend    | 2020-12-03T10:22:48+0000.106 [INFO] _internal._log:113 (1:Thread-10) - 10.20.220.37 - - [03/Dec/2020 10:22:48] "GET /api/metadata/v0/get_last_indexed HTTP/1.1" 500 -
amundsenfrontend    | 2020-12-03T10:22:48+0000.107 [DEBUG] connectionpool._make_request:442 (1:Thread-7) - http://amundsenmetadata:5002 "GET /tags/ HTTP/1.1" 500 None
amundsenfrontend    | 2020-12-03T10:22:48+0000.109 [DEBUG] connectionpool._make_request:442 (1:Thread-8) - http://amundsenmetadata:5002 "GET /popular_tables/?limit=4 HTTP/1.1" 500 None
amundsenfrontend    | 2020-12-03T10:22:48+0000.110 [ERROR] v0.popular_tables:72 (1:Thread-8) - Encountered error: Request to metadata service failed with status code 500
amundsenfrontend    | 2020-12-03T10:22:48+0000.110 [INFO] _internal._log:113 (1:Thread-8) - 10.20.220.37 - - [03/Dec/2020 10:22:48] "GET /api/metadata/v0/popular_tables HTTP/1.1" 500 -
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 187, in connect
amundsenmetadata    |     conn = self._new_conn()
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connection.py", line 172, in _new_conn
amundsenmetadata    |     self, "Failed to establish a new connection: %s" % e
amundsenmetadata    | urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f1d50247a90>: Failed to establish a new connection: [Errno -2] Name or service not known
amundsenmetadata    | 
amundsenmetadata    | During handling of the above exception, another exception occurred:
amundsenmetadata    | 
amundsenmetadata    | Traceback (most recent call last):
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 449, in send
amundsenmetadata    |     timeout=timeout
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
amundsenmetadata    |     **response_kw
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
amundsenmetadata    |     **response_kw
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
amundsenmetadata    |     **response_kw
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
amundsenmetadata    |     **response_kw
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 767, in urlopen
amundsenmetadata    |     **response_kw
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/connectionpool.py", line 727, in urlopen
amundsenmetadata    |     method, url, error=e, _pool=self, _stacktrace=sys.exc_info()[2]
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/urllib3/util/retry.py", line 446, in increment
amundsenmetadata    |     raise MaxRetryError(_pool, url, error or ResponseError(cause))
amundsenmetadata    | urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='jbdlcanary.data01.jio.com', port=21000): Max retries exceeded with url: /api/atlas/admin/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d50247a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
amundsenmetadata    | 
amundsenmetadata    | During handling of the above exception, another exception occurred:
amundsenmetadata    | 
amundsenmetadata    | Traceback (most recent call last):
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2309, in __call__
amundsenmetadata    |     return self.wsgi_app(environ, start_response)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2295, in wsgi_app
amundsenmetadata    |     response = self.handle_exception(e)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 273, in error_router
amundsenmetadata    |     return original_handler(e)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1741, in handle_exception
amundsenmetadata    |     reraise(exc_type, exc_value, tb)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 34, in reraise
amundsenmetadata    |     raise value.with_traceback(tb)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 2292, in wsgi_app
amundsenmetadata    |     response = self.full_dispatch_request()
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1815, in full_dispatch_request
amundsenmetadata    |     rv = self.handle_user_exception(e)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 273, in error_router
amundsenmetadata    |     return original_handler(e)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1718, in handle_user_exception
amundsenmetadata    |     reraise(exc_type, exc_value, tb)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/_compat.py", line 34, in reraise
amundsenmetadata    |     raise value.with_traceback(tb)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1813, in full_dispatch_request
amundsenmetadata    |     rv = self.dispatch_request()
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/app.py", line 1799, in dispatch_request
amundsenmetadata    |     return self.view_functions[rule.endpoint](**req.view_args)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 480, in wrapper
amundsenmetadata    |     resp = resource(*args, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask/views.py", line 88, in view
amundsenmetadata    |     return self.dispatch_request(*args, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flask_restful/__init__.py", line 595, in dispatch_request
amundsenmetadata    |     resp = meth(*args, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/flasgger/utils.py", line 249, in wrapper
amundsenmetadata    |     return function(*args, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/amundsen_metadata-3.0.0-py3.7.egg/metadata_service/api/tag.py", line 39, in get
amundsenmetadata    |     tag_usages = self.client.get_tags()
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/amundsen_metadata-3.0.0-py3.7.egg/metadata_service/proxy/atlas_proxy.py", line 792, in get_tags
amundsenmetadata    |     for metrics in self._driver.admin_metrics:
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/atlasclient/base.py", line 133, in __iter__
amundsenmetadata    |     self.inflate()
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/atlasclient/base.py", line 262, in inflate
amundsenmetadata    |     self.load(self.client.get(self.url, params=self._filter))
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/atlasclient/client.py", line 159, in request
amundsenmetadata    |     response = requests_method(url, **params)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 555, in get
amundsenmetadata    |     return self.request('GET', url, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
amundsenmetadata    |     resp = self.send(prep, **send_kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
amundsenmetadata    |     r = adapter.send(request, **kwargs)
amundsenmetadata    |   File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 516, in send
amundsenmetadata    |     raise ConnectionError(e, request=request)
amundsenmetadata    | requests.exceptions.ConnectionError: HTTPConnectionPool(host='jbdlcanary.data01.jio.com', port=21000): Max retries exceeded with url: /api/atlas/admin/metrics (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f1d50247a90>: Failed to establish a new connection: [Errno -2] Name or service not known'))
amundsenmetadata    | 2020-12-03T10:22:48+0000.107 [INFO] _internal._log:122 (9:Thread-3) - 172.18.0.4 - - [03/Dec/2020 10:22:48] "GET /popular_tables/?limit=4 HTTP/1.1" 500 -
amundsenfrontend    | 2020-12-03T10:22:48+0000.111 [ERROR] v0.get_tags:355 (1:Thread-7) - Encountered error: Tags Unavailable

Please let me know what i am missing here?

Thanks, Vinod Thorwat

jornh commented 3 years ago

Link icon for joining the Slack channel is in the top of README.md: https://github.com/amundsen-io/amundsen#readme

The slack workspace has a #troubleshoot channel among other things.

stale[bot] commented 3 years ago

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

stale[bot] commented 3 years ago

This issue has been automatically closed for inactivity. If you still wish to make these changes, please open a new pull request or reopen this one.