Closed hongbo-miao closed 2 years ago
Sorry, I missed https setup in the step.
After I added https support for config server, and pointed to https://localhost:26660/config?token=xxx
.
Now my error becomes the one below.
As I am testing with self-signed certificate, I think I need find a way to test it. Is there a flag allowing me to use self-signed certificate for testing purpose? Thanks!
Also, I am wondering if my original issue is related https. If related, then the original error might be misleading.
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fa0bac986d0>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7fa0ba523310>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7fa0ba523430>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7fa0ba5233a0>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7fa0b9908ee0>
└ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7fa0b98ec0d0>
└ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 84, in run
return asyncio.run(self._serve())
│ │ │ └ <function UvicornWorker._serve at 0x7fa0b98ec040>
│ │ └ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
│ └ <function run at 0x7fa0ba04d8b0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
│ │ └ <coroutine object UvicornWorker._serve at 0x7fa0b874ddc0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 168, in on_connect
await self.get_base_policy_data()
│ └ <function DataUpdater.get_base_policy_data at 0x7fa0b8b47c10>
└ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 152, in get_base_policy_data
sources_config = await self.get_policy_data_config(url=config_url)
│ │ └ None
│ └ <function DataUpdater.get_policy_data_config at 0x7fa0b8b47b80>
└ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
> File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 133, in get_policy_data_config
response = await session.get(url, **self._ssl_context_kwargs)
│ │ │ │ └ {}
│ │ │ └ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
│ │ └ 'http://opal-server-service.hm-opa:7002/data/config'
│ └ <function ClientSession.get at 0x7fa0b904ec10>
└ <aiohttp.client.ClientSession object at 0x7fa0b6415d30>
File "/root/.local/lib/python3.8/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
│ └ None
└ <aiohttp.client.ClientSession object at 0x7fa0b6415d30>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
│ │ │ │ └ ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=None)
│ │ │ └ []
│ │ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b641a700>
│ └ <function TCPConnector._create_connection at 0x7fa0b91035e0>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b641a220>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
│ │ │ │ └ ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=None)
│ │ │ └ []
│ │ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b641a700>
│ └ <function TCPConnector._create_direct_connection at 0x7fa0b9103940>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b641a220>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
└ ClientConnectorCertificateError(ConnectionKey(host='config-server-service.hm', port=26660, is_ssl=True, ssl=None, proxy=None,...
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
│ └ <function TCPConnector._wrap_create_connection at 0x7fa0b91038b0>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b641a220>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection
raise ClientConnectorCertificateError(req.connection_key, exc) from exc
│ │ └ <property object at 0x7fa0b90d9450>
│ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b641a700>
└ <class 'aiohttp.client_exceptions.ClientConnectorCertificateError'>
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host config-server-service.hm:26660 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125)')]
2021-10-27T23:01:17.439201+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - Cannot connect to host config-server-service.hm:26660 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125)')]
2021-10-27T23:01:20.520629+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-27T23:01:20.555255+0000 | opal_client.data.updater | INFO | Connected to server
2021-10-27T23:01:20.556196+0000 | opal_client.data.updater | INFO | Performing data configuration, reason: Initial load
2021-10-27T23:01:20.557603+0000 | opal_client.data.updater | INFO | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-27T23:01:20.597011+0000 | inspect |ERROR | Task was destroyed but it is pending!
task: <Task pending name='Task-80' coro=<Event.wait() done, defined at /usr/local/lib/python3.8/asyncio/locks.py:296> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fa0b6415cd0>()]> cb=[as_completed.<locals>._on_completion() at /usr/local/lib/python3.8/asyncio/tasks.py:606]>
2021-10-27T23:01:20.597557+0000 | inspect |ERROR | Task was destroyed but it is pending!
task: <Task pending name='Task-83' coro=<Event.wait() done, defined at /usr/local/lib/python3.8/asyncio/locks.py:296> wait_for=<Future pending cb=[<TaskWakeupMethWrapper object at 0x7fa0b641a040>()]> cb=[as_completed.<locals>._on_completion() at /usr/local/lib/python3.8/asyncio/tasks.py:606]>
2021-10-27T23:01:20.594183+0000 | opal_client.data.updater |ERROR | Failed to load data sources config
Traceback (most recent call last):
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 969, in _wrap_create_connection
return await self._loop.create_connection(*args, **kwargs) # type: ignore # noqa
│ │ │ │ └ {'ssl': <ssl.SSLContext object at 0x7fa0b859cc40>, 'family': 2, 'proto': 6, 'flags': <AddressInfo.AI_NUMERICSERV|AI_NUMERICHO...
│ │ │ └ (functools.partial(<class 'aiohttp.client_proto.ResponseHandler'>, loop=<uvloop.Loop running=True closed=False debug=False>),...
│ │ └ <method 'create_connection' of 'uvloop.loop.Loop' objects>
│ └ <uvloop.Loop running=True closed=False debug=False>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b6425310>
File "uvloop/loop.pyx", line 2069, in create_connection
raise
File "uvloop/loop.pyx", line 2064, in uvloop.loop.Loop.create_connection
await ssl_waiter
File "uvloop/sslproto.pyx", line 517, in uvloop.loop.SSLProtocol._on_handshake_complete
raise handshake_exc
File "uvloop/sslproto.pyx", line 499, in uvloop.loop.SSLProtocol._do_handshake
self._sslobj.do_handshake()
File "/usr/local/lib/python3.8/ssl.py", line 944, in do_handshake
self._sslobj.do_handshake()
│ │ └ <method 'do_handshake' of '_ssl._SSLSocket' objects>
│ └ <_ssl._SSLSocket object at 0x7fa0b641bf30>
└ <ssl.SSLObject object at 0x7fa0b64259a0>
ssl.SSLCertVerificationError: [SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125)
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/root/.local/bin/gunicorn", line 8, in <module>
sys.exit(run())
│ │ └ <function run at 0x7fa0baba3b80>
│ └ <built-in function exit>
└ <module 'sys' (built-in)>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/wsgiapp.py", line 67, in run
WSGIApplication("%(prog)s [OPTIONS] [APP_MODULE]").run()
└ <class 'gunicorn.app.wsgiapp.WSGIApplication'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 231, in run
super().run()
File "/root/.local/lib/python3.8/site-packages/gunicorn/app/base.py", line 72, in run
Arbiter(self).run()
│ └ <gunicorn.app.wsgiapp.WSGIApplication object at 0x7fa0bac986d0>
└ <class 'gunicorn.arbiter.Arbiter'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 202, in run
self.manage_workers()
│ └ <function Arbiter.manage_workers at 0x7fa0ba523310>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 551, in manage_workers
self.spawn_workers()
│ └ <function Arbiter.spawn_workers at 0x7fa0ba523430>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 622, in spawn_workers
self.spawn_worker()
│ └ <function Arbiter.spawn_worker at 0x7fa0ba5233a0>
└ <gunicorn.arbiter.Arbiter object at 0x7fa0ba09a5b0>
File "/root/.local/lib/python3.8/site-packages/gunicorn/arbiter.py", line 589, in spawn_worker
worker.init_process()
│ └ <function UvicornWorker.init_process at 0x7fa0b9908ee0>
└ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 66, in init_process
super(UvicornWorker, self).init_process()
│ └ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
└ <class 'uvicorn.workers.UvicornWorker'>
File "/root/.local/lib/python3.8/site-packages/gunicorn/workers/base.py", line 142, in init_process
self.run()
│ └ <function UvicornWorker.run at 0x7fa0b98ec0d0>
└ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
File "/root/.local/lib/python3.8/site-packages/uvicorn/workers.py", line 84, in run
return asyncio.run(self._serve())
│ │ │ └ <function UvicornWorker._serve at 0x7fa0b98ec040>
│ │ └ <uvicorn.workers.UvicornWorker object at 0x7fa0b9744c40>
│ └ <function run at 0x7fa0ba04d8b0>
└ <module 'asyncio' from '/usr/local/lib/python3.8/asyncio/__init__.py'>
File "/usr/local/lib/python3.8/asyncio/runners.py", line 44, in run
return loop.run_until_complete(main)
│ │ └ <coroutine object UvicornWorker._serve at 0x7fa0b874ddc0>
│ └ <method 'run_until_complete' of 'uvloop.loop.Loop' objects>
└ <uvloop.Loop running=True closed=False debug=False>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 168, in on_connect
await self.get_base_policy_data()
│ └ <function DataUpdater.get_base_policy_data at 0x7fa0b8b47c10>
└ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 152, in get_base_policy_data
sources_config = await self.get_policy_data_config(url=config_url)
│ │ └ None
│ └ <function DataUpdater.get_policy_data_config at 0x7fa0b8b47b80>
└ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
> File "/usr/local/lib/python3.8/site-packages/opal_client-0.1.16-py3.8.egg/opal_client/data/updater.py", line 133, in get_policy_data_config
response = await session.get(url, **self._ssl_context_kwargs)
│ │ │ │ └ {}
│ │ │ └ <opal_client.data.updater.DataUpdater object at 0x7fa0b873dbb0>
│ │ └ 'http://opal-server-service.hm-opa:7002/data/config'
│ └ <function ClientSession.get at 0x7fa0b904ec10>
└ <aiohttp.client.ClientSession object at 0x7fa0b64250a0>
File "/root/.local/lib/python3.8/site-packages/aiohttp/client.py", line 520, in _request
conn = await self._connector.connect(
│ └ None
└ <aiohttp.client.ClientSession object at 0x7fa0b64250a0>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 535, in connect
proto = await self._create_connection(req, traces, timeout)
│ │ │ │ └ ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=None)
│ │ │ └ []
│ │ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b64257f0>
│ └ <function TCPConnector._create_connection at 0x7fa0b91035e0>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b6425310>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 892, in _create_connection
_, proto = await self._create_direct_connection(req, traces, timeout)
│ │ │ │ └ ClientTimeout(total=300, connect=None, sock_read=None, sock_connect=None)
│ │ │ └ []
│ │ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b64257f0>
│ └ <function TCPConnector._create_direct_connection at 0x7fa0b9103940>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b6425310>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 1051, in _create_direct_connection
raise last_exc
└ ClientConnectorCertificateError(ConnectionKey(host='config-server-service.hm', port=26660, is_ssl=True, ssl=None, proxy=None,...
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 1020, in _create_direct_connection
transp, proto = await self._wrap_create_connection(
│ └ <function TCPConnector._wrap_create_connection at 0x7fa0b91038b0>
└ <aiohttp.connector.TCPConnector object at 0x7fa0b6425310>
File "/root/.local/lib/python3.8/site-packages/aiohttp/connector.py", line 971, in _wrap_create_connection
raise ClientConnectorCertificateError(req.connection_key, exc) from exc
│ │ └ <property object at 0x7fa0b90d9450>
│ └ <aiohttp.client_reqrep.ClientRequest object at 0x7fa0b64257f0>
└ <class 'aiohttp.client_exceptions.ClientConnectorCertificateError'>
aiohttp.client_exceptions.ClientConnectorCertificateError: Cannot connect to host config-server-service.hm:26660 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125)')]
2021-10-27T23:01:20.602078+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO | RPC Connection failed - Cannot connect to host config-server-service.hm:26660 ssl:True [SSLCertVerificationError: (1, '[SSL: CERTIFICATE_VERIFY_FAILED] certificate verify failed: self signed certificate (_ssl.c:1125)')]
Hey @Hongbo-Miao , this is probably a config issue.
I would first: 1) make sure in requests and responses that you return - you must pass the content-type http header with "application/json". this is required by fastapi to process the json correctly 2) regarding SSL - are you using self signed certificates?
The best thing to do is to jump on zoom and debug your config together.
When using c.JSON
in a Go (Gin) server, it returns content-type: application/json; charset=utf-8
by default.
Yeah, I am trying to use a self-signed certificate in my local Kubernetes to test.
As I cannot find a quick solution not using self-signed certificate, I tried to change to content-type: application/json
without TLS enabled. I still get the original issue
1 validation error for DataSourceConfig entries field required (type=value_error.missing)
Thanks @asafc the help offline!
My real issue is because my config server exposed the endpoint /config
.
However, my opal-server is using
OPAL_DATA_CONFIG_SOURCES: {"external_source_url":"http://config-server-service.hm:26660"}
The correct one is
OPAL_DATA_CONFIG_SOURCES: {"external_source_url":"http://config-server-service.hm:26660/config"}
The error logs shows
1 validation error for DataSourceConfig entries field required (type=value_error.missing)
Because it hits my config server healthcheck endpoint /
whose result does not include entries
field.
I can confirm both content-type: application/json; charset=utf-8
and content-type: application/json
work.
Regarding how to use self-signed certificate for local development,
https://github.com/authorizon/opal/pull/104
has some info, I guess setting these two to opal client will help (didn't try myself)
OPAL_CLIENT_SELF_SIGNED_CERTIFICATES_ALLOWED=true
OPAL_CLIENT_SSL_CONTEXT_TRUSTED_CA_FILE=/path/to/ca-public.crt
Based on the How to configure external data sources, I created my config server.
When I hit my config server by
I got response:
However, my opal client log shows
entries
field is missing:Any idea? Thanks!