Opensearch IAM auth

[kizmanj]

Already posted about this in another ticket, but that issue has been closed since.

Tried using IAM auth to connect to OpenSearch from a container running in EKS, with https://docs.aws.amazon.com/eks/latest/userguide/iam-roles-for-service-accounts.html

Yente 4.1.0 seems to have updated components that helped, but did not fully resolve the problem.

Using these settings: export set YENTE_INDEX_TYPE="opensearch" export set YENTE_INDEX_URL="https://redacted/" export set YENTE_OPENSEARCH_REGION="eu-central-1" export set YENTE_OPENSEARCH_SERVICE="es"

The error currently looks like this:

Traceback (most recent call last): File "/venv/bin/yente", line 33, in sys.exit(load_entry_point('yente', 'console_scripts', 'yente')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/cli.py", line 44, in reindex asyncio.run(update_index(force=force)) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/app/yente/search/indexer.py", line 188, in update_index async with with_provider() as provider: File "/usr/lib/python3.12/contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/init.py", line 47, in with_provider provider = await _create_provider() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/init.py", line 25, in _create_provider return await OpenSearchProvider.create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/opensearch.py", line 54, in create await es.cluster.health(wait_for_status="yellow", timeout=5) File "/venv/lib/python3.12/site-packages/opensearchpy/_async/client/cluster.py", line 131, in health return await self.transport.perform_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 375, in perform_request await self._async_call() File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 198, in _async_call await self._async_init() File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 163, in _async_init self.set_connections(self.hosts) File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 255, in set_connections connections = list(zip(map(_create_connection, hosts), hosts)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 253, in _create_connection return self.connection_class(metrics=self.metrics, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/_async/http_aiohttp.py", line 149, in init self.headers.update(urllib3.make_headers(basic_auth=http_auth)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/urllib3/util/request.py", line 121, in make_headers ] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}" ^^^^^^^^^^^^^^^^^ AttributeError: 'AWSV4SignerAuth' object has no attribute 'encode'

According to this discussion, adding these args to the opensearch provider here should help:

connection_class = RequestsHttpConnection

[pudo]

Thanks for doing all this research, Josef! The only challenge I have with the discussion in the ticket is that they're talking sync Python, and we're using async. There's a similar ticket here talking about async: https://github.com/opensearch-project/opensearch-py/issues/698 - which introduces yet another class, AWSV4SignerAsyncAuth.

Maybe I can ask for your help on this: if I make that change in main, would you be able to run a specific docker hash to check if it works?

[pudo]

It's in this commit, building docker now: https://github.com/opensanctions/yente/commit/1c26099a8a043baf35aa2fbe7640d36daf89100d

[pudo]

OK, tool a few rounds - docker pull ghcr.io/opensanctions/yente:sha-b18443a contains the fix

[kizmanj]

Thanks a lot, will try it later today!

[kizmanj]

tried it, but still getting the same error.

Will try a manual build with this later: connection_class = RequestsHttpConnection

[pudo]

I'd be really surprised to see that work, since requests is a sync library. Can you share with me the updated stack trace from running this version, and I'll file a report on opensearch-py?

[kizmanj]

You are right, it did not help.

Here's the stack trace coming out of ghcr.io/opensanctions/yente:sha-b18443a image

Traceback (most recent call last): File "/venv/bin/yente", line 33, in sys.exit(load_entry_point('yente', 'console_scripts', 'yente')()) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1157, in call return self.main(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1078, in main rv = self.invoke(ctx) ^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1688, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 1434, in invoke return ctx.invoke(self.callback, ctx.params) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/click/core.py", line 783, in invoke return __callback(args, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/cli.py", line 44, in reindex asyncio.run(update_index(force=force)) File "/usr/lib/python3.12/asyncio/runners.py", line 194, in run return runner.run(main) ^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/runners.py", line 118, in run return self._loop.run_until_complete(task) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/usr/lib/python3.12/asyncio/base_events.py", line 687, in run_until_complete return future.result() ^^^^^^^^^^^^^^^ File "/app/yente/search/indexer.py", line 188, in update_index async with with_provider() as provider: File "/usr/lib/python3.12/contextlib.py", line 210, in aenter return await anext(self.gen) ^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/init.py", line 47, in with_provider provider = await _create_provider() ^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/init.py", line 25, in _create_provider return await OpenSearchProvider.create() ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/app/yente/provider/opensearch.py", line 54, in create await es.cluster.health(wait_for_status="yellow", timeout=5) File "/venv/lib/python3.12/site-packages/opensearchpy/_async/client/cluster.py", line 131, in health return await self.transport.perform_request( ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 375, in perform_request await self._async_call() File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 198, in _async_call await self._async_init() File "/venv/lib/python3.12/site-packages/opensearchpy/_async/transport.py", line 163, in _async_init self.set_connections(self.hosts) File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 255, in set_connections connections = list(zip(map(_create_connection, hosts), hosts)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/transport.py", line 253, in _create_connection return self.connection_class(metrics=self.metrics, kwargs) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/opensearchpy/_async/http_aiohttp.py", line 149, in init self.headers.update(urllib3.make_headers(basic_auth=http_auth)) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ File "/venv/lib/python3.12/site-packages/urllib3/util/request.py", line 121, in make_headers ] = f"Basic {b64encode(basic_auth.encode('latin-1')).decode()}" ^^^^^^^^^^^^^^^^^ AttributeError: 'AWSV4SignerAsyncAuth' object has no attribute 'encode'

[pudo]

So I made this experiment but that won't even pass our tests (breaks opensearch): https://github.com/opensanctions/yente/commit/196691dc11df73c251a19711d4a9a2dd92b67d75

I think it's just broken. Will need to file a ticket upstream.

[kizmanj]

I kept experimenting and got it working!

Added this line before the AWSV4SignerAsyncAuth line 44. (and it's import too) kwargs["connection_class"] = AsyncHttpConnection

Idea came from here

[pudo]

I tried the same thing yesterday, but AsyncHttpConnection seems to have another bug: it doesn't properly translate HTTP error codes. If you look at the build log, it looks like all server-side errors (400 etc.) are being translated into HTTP 500 errors, which messes up all of our error handling downstream in the app.

[kizmanj]

I have seen this on my test deployment too. It seems like it's trying to create a new temporary index with the same name as a previous one. In my case i thought it was related to a previous interrupted reindex run. Removing the index from opensearch solved it.

[kizmanj]

Unfortunately can't say much about the 400->500 thing, as i'm not familiar with python or the codebase.