Updating Traction to 1.0.0 can cause issues with older Innkeeper setups on single database mode

Earlier Traction deployments on OCP (pre 0.12.0 of ACA-Py it seems) have been running in single wallet db mode (so multitenant sub wallets go in one DB) but it seems that due to Innkeeper plugin running through a different multitenancy provider path (plugin config, order of loading plugins) that the Innkeeper tenant was still creating in it's own database. And that was still working in that bifurcated mode. Tenants to a single DB, Innkeeper to a standalone.

See example DB setup:

After fixing up some multitenancy and innkeeper plugin things this setup stopped working once upgrading Traction to 1.0.0 using the newer ACA-Py releases and plugin changes.

If manifests in the following way:

You can get a token for the innkeeper
Using that token to try an API call errors as below

2024-11-12 17:58:13,266 TENANT: None ERROR /home/aries/.venv/lib/python3.12/site-packages/aiohttp/web_protocol.py:448 Error handling request
Traceback (most recent call last):
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/askar/profile.py", line 244, in _setup
self._handle = await asyncio.wait_for(self._opener, 10)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/tasks.py", line 520, in wait_for
return await fut
^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/aries_askar/store.py", line 733, in _open
await bindings.session_start(self._store, self._profile, self._is_txn),
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/aries_askar/bindings/__init__.py", line 262, in session_start
handle = await invoke_async(
^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/aries_askar/bindings/lib.py", line 362, in invoke_async
return await self.loaded.invoke_async(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/futures.py", line 291, in __await__
yield self # This tells Task to wait for completion.
^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
future.result()
File "/usr/local/lib/python3.12/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
aries_askar.error.AskarError: Profile not found
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/home/aries/.venv/lib/python3.12/site-packages/aiohttp/web_protocol.py", line 477, in _handle_request
resp = await request_handler(request)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/aiohttp/web_app.py", line 559, in _handle
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/aiohttp/web_middlewares.py", line 117, in impl
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/admin/server.py", line 141, in ready_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/admin/server.py", line 212, in debug_middleware
return await handler(request)
^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/admin/server.py", line 357, in setup_context
return await task
^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/futures.py", line 291, in __await__
yield self # This tells Task to wait for completion.
^^^^^^^^^^
File "/usr/local/lib/python3.12/asyncio/tasks.py", line 385, in __wakeup
future.result()
File "/usr/local/lib/python3.12/asyncio/futures.py", line 203, in result
raise self._exception.with_traceback(self._exception_tb)
File "/usr/local/lib/python3.12/asyncio/tasks.py", line 316, in __step_run_and_handle_result
result = coro.throw(exc)
^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/admin/server.py", line 183, in upgrade_middleware
async with context.profile.session() as session:
^^^^^^^^^^^^^^^^^^^^^^^^^
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/core/profile.py", line 197, in __aenter__
await self._setup()
File "/home/aries/.venv/lib/python3.12/site-packages/acapy_agent/askar/profile.py", line 246, in _setup
raise ProfileError("Error opening store session") from err
acapy_agent.core.error.ProfileError: Error opening store session

Note this is a pretty generic error that can happen in other cases where storage can't find the profile (DB inaccessible, etc). So this error isn't specific to the Innkeeper case here.

Other existing tenants work just fine, can access them, use webhooks, use API Keys, etc. It's just the Innkeeper tenant that breaks this way. This means you can't look at other tenants, delete them, manage API keys at the Innkeeper level, or create new tenants.

Easiest fix for this is to just create a new Innkeeper, there's no historical data kept in the old Innkeeper or anything. The Innkeeper is just another Tenant, it just has specific permissions to call specific routes in the Innkeeper plugin is all.

To do that can leverage the fact that Traction will create an innkeeper on startup if one is not found. So can do the following steps.

Alter the Innkeeper credentials in the environment. Can just change it so there's a new Tenant ID that will not be found on startup. For the ACA-PY process this is the INNKEEPER_WALLET_TENANT_ID environment variable. For our OCP deployments can
- Go alter the traction-acapy-plugin-innkeeper secret and change the Tenant ID to something new.
- Restart an ACA-Py pod
- On startup you'll see logging about the new Innkeeper TENANT: None INFO /home/aries/traction_plugins/traction_innkeeper/v1_0/innkeeper/tenant_manager.py:219 ...created 'traction_innkeeper' tenant and wallet.
- Restart the Tenant UI so the OIDC integration (if using) picks up the new secret values as well
After verifying you can use this new Innkeeper you can delete the old one
- Log in with new Innkeeper and get token
- Use the swagger for the env
- Go check /innkeepers/tenants and find the old one
- Run the hard delete route on the tenant ID of the old Innkeeper

bcgov / traction

Updating Traction to 1.0.0 can cause issues with older Innkeeper setups on single database mode #1423