search

permitio / opal

Policy and data administration, distribution, and real-time updates on top of Policy Agents (OPA, Cedar, ...)
https://opal.ac
Apache License 2.0
4.14k stars 153 forks source link

OPAL_DEFAULT_UPDATE_CALLBACKS is not parsed correctly when provided with config #166

Open hongbo-miao opened 2 years ago

hongbo-miao commented 2 years ago

I have successfully set up the JWT token. This is my opal-client log which shows it works well.

[2021-10-21 20:30:06 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-21 20:30:06 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-21 20:30:06 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-21 20:30:06 +0000] [10] [INFO] Booting worker with pid: 10
2021-10-21T20:30:06.997648+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-21T20:30:06.998161+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-21T20:30:07.028551+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-21T20:30:07.028762+0000 | opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
2021-10-21T20:30:07.029683+0000 | opal_client.callbacks.register          | INFO  | Callbacks register loaded
2021-10-21T20:30:07.030053+0000 | opal_client.client                      | INFO  | API authentication disabled (public encryption key was not provided)
2021-10-21T20:30:07.097858+0000 | uvicorn.server                          | INFO  | Started server process [10]
2021-10-21T20:30:07.098212+0000 | uvicorn.lifespan.on                     | INFO  | Waiting for application startup.
2021-10-21T20:30:07.098566+0000 | opal_client.opa.runner                  | INFO  | Launching opa runner
2021-10-21T20:30:07.098808+0000 | uvicorn.lifespan.on                     | INFO  | Application startup complete.
2021-10-21T20:30:07.101344+0000 | opal_client.opa.runner                  | INFO  | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-21T20:30:07.130883+0000 | opal_client.opa.logger                  | INFO  | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-21T20:30:07Z'}
2021-10-21T20:30:07.132577+0000 | opal_client.opa.logger                  | INFO  | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-21T20:30:07Z'}
2021-10-21T20:30:08.114086+0000 | opal_client.opa.runner                  | INFO  | Running OPA initial start callbacks
2021-10-21T20:30:08.114794+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-21T20:30:08.118031+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60792', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.119414+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60792', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 1.4131, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.120468+0000 | opal_client.data.updater                | INFO  | Launching data updater
2021-10-21T20:30:08.120688+0000 | opal_client.policy.updater              | INFO  | Launching policy updater
2021-10-21T20:30:08.120808+0000 | opal_client.data.updater                | INFO  | Subscribing to topics: ['policy_data']
2021-10-21T20:30:08.121188+0000 | opal_client.policy.updater              | INFO  | Subscribing to topics: ['policy:.']
2021-10-21T20:30:08.121432+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.121708+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.124114+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.124399+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.157712+0000 | opal_client.data.updater                | INFO  | Connected to server
2021-10-21T20:30:08.157862+0000 | opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
2021-10-21T20:30:08.158014+0000 | opal_client.data.updater                | INFO  | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-21T20:30:08.161874+0000 | opal_client.policy.updater              | INFO  | Connected to server
2021-10-21T20:30:08.162482+0000 | opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
2021-10-21T20:30:08.169424+0000 | opal_client.data.updater                | INFO  | Triggering data update with id: 5bd1936ae65146648788dd6081c79b2b
2021-10-21T20:30:08.170344+0000 | opal_client.data.updater                | INFO  | Fetching policy data
2021-10-21T20:30:08.170627+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: postgresql://admin@opa-db-service.hm-opa:40072/opa_db
2021-10-21T20:30:08.193387+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:30:08.225411+0000 | opal_client.policy.updater              | INFO  | Got policy bundle with 1 rego files, 0 data files, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-21T20:30:08.228156+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='postgresql://admin@opa-db-service.hm-opa:40072/opa_db', destination path='/roles'
2021-10-21T20:30:08.234055+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60808', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.234757+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60810', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.239404+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60808', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 4603, 'resp_duration': 1.7899, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.239731+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60810', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 1.3754, 'resp_status': 204, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.240629+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '5bd1936ae65146648788dd6081c79b2b', 'actions': ['set_policy_data'], 'success': True, 'error': ''}
2021-10-21T20:30:08.240791+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-21T20:30:08.243687+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60812', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.244179+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60814', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.250523+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60812', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 6.3852, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.251354+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-21T20:30:08.251502+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=true, healthy=true
2021-10-21T20:30:08.256960+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:60816', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.268133+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60814', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 23.318, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.273519+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:60816', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 14.152, 'resp_status': 200, 'time': '2021-10-21T20:30:08Z'}
2021-10-21T20:30:08.276124+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws

Now I added

OPAL_SHOULD_REPORT_ON_DATA_UPDATES: "True"
OPAL_DEFAULT_UPDATE_CALLBACKS: '{"callbacks":["http://opal-server-service.hm-opa:7002/data/callback_report"]}'

However, my opal-client failed to report the update to opal-server.

[2021-10-21 20:25:33 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-21 20:25:33 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-21 20:25:33 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-21 20:25:33 +0000] [9] [INFO] Booting worker with pid: 9
2021-10-21T20:25:34.312137+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-21T20:25:34.312482+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-21T20:25:34.346065+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-21T20:25:34.346267+0000 | opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
2021-10-21T20:25:34.347387+0000 | opal_client.callbacks.register          | INFO  | Callbacks register loaded
2021-10-21T20:25:34.347677+0000 | opal_client.client                      | INFO  | API authentication disabled (public encryption key was not provided)
2021-10-21T20:25:34.416326+0000 | uvicorn.server                          | INFO  | Started server process [9]
2021-10-21T20:25:34.416639+0000 | uvicorn.lifespan.on                     | INFO  | Waiting for application startup.
2021-10-21T20:25:34.417208+0000 | opal_client.opa.runner                  | INFO  | Launching opa runner
2021-10-21T20:25:34.417500+0000 | uvicorn.lifespan.on                     | INFO  | Application startup complete.
2021-10-21T20:25:34.419152+0000 | opal_client.opa.runner                  | INFO  | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-21T20:25:34.472966+0000 | opal_client.opa.logger                  | INFO  | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-21T20:25:34Z'}
2021-10-21T20:25:34.474321+0000 | opal_client.opa.logger                  | INFO  | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-21T20:25:34Z'}
2021-10-21T20:25:35.425433+0000 | opal_client.opa.runner                  | INFO  | Running OPA initial start callbacks
2021-10-21T20:25:35.426772+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-21T20:25:35.432836+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57464', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.435393+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57464', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 2.7916, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.437263+0000 | opal_client.data.updater                | INFO  | Launching data updater
2021-10-21T20:25:35.437548+0000 | opal_client.policy.updater              | INFO  | Launching policy updater
2021-10-21T20:25:35.437701+0000 | opal_client.data.updater                | INFO  | Subscribing to topics: ['policy_data']
2021-10-21T20:25:35.438353+0000 | opal_client.policy.updater              | INFO  | Subscribing to topics: ['policy:.']
2021-10-21T20:25:35.438681+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.439513+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.443346+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.445246+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.513493+0000 | opal_client.data.updater                | INFO  | Connected to server
2021-10-21T20:25:35.513641+0000 | opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
2021-10-21T20:25:35.513738+0000 | opal_client.data.updater                | INFO  | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-21T20:25:35.517492+0000 | opal_client.policy.updater              | INFO  | Connected to server
2021-10-21T20:25:35.517724+0000 | opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
2021-10-21T20:25:35.521039+0000 | opal_client.data.updater                | INFO  | Triggering data update with id: b950fa27e74743c39c8dd01d7cc53999
2021-10-21T20:25:35.521225+0000 | opal_client.data.updater                | INFO  | Fetching policy data
2021-10-21T20:25:35.521581+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: postgresql://admin@opa-db-service.hm-opa:40072/opa_db
2021-10-21T20:25:35.538200+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.548979+0000 | opal_client.policy.updater              | INFO  | Got policy bundle with 1 rego files, 0 data files, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-21T20:25:35.555292+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57478', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.555595+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57478', 'req_id': 2, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 4603, 'resp_duration': 0.6619, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.560291+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57480', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.563343+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57480', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 4.317, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.566660+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-21T20:25:35.567034+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-21T20:25:35.569385+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks
2021-10-21T20:25:35.570242+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-21T20:25:35.575745+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='postgresql://admin@opa-db-service.hm-opa:40072/opa_db', destination path='/roles'
2021-10-21T20:25:35.578385+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57482', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.580662+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57482', 'req_id': 4, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 3.8314, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.584730+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-21T20:25:35.585993+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57486', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.586574+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57486', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 1.4706, 'resp_status': 204, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.588882+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': 'b950fa27e74743c39c8dd01d7cc53999', 'actions': ['set_policy_data'], 'success': True, 'error': ''}
2021-10-21T20:25:35.589174+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=true, healthy=true
2021-10-21T20:25:35.592062+0000 | opal_client.callbacks.reporter          |ERROR  | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
2021-10-21T20:25:35.592760+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:57488', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.595426+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:57488', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 2.4091, 'resp_status': 200, 'time': '2021-10-21T20:25:35Z'}
2021-10-21T20:25:35.597263+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks
2021-10-21T20:25:35.597523+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-21T20:25:35.607322+0000 | opal_client.callbacks.reporter          |ERROR  | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}

Both my opal-client and opal-server versions are 0.1.16.

asafc commented 2 years ago

Hey @Hongbo-Miao, you configured the callbacks to be sent to the OPAL server (you can actually send them to whatever server you choose).

Since OPAL server is running in secure mode, it requires that requests to its API will be provided with a bearer token (the client JWT token), and you did not provide a bearer token in your callback configuration.

The solution is very simple.

Step 1: Obtain a client token

Step 2: Provide the token in your callback configuration

Assuming your client jwt token is MY_JWT:

Your configuration should look like this:

OPAL_DEFAULT_UPDATE_CALLBACKS: '{"callbacks":[("http://opal-server-service.hm-opa:7002/data/callback_report",{"headers":{"Authorization":"Bearer MY_JWT"}})]}'

See also: What are the values I can set inside callbacks?

hongbo-miao commented 2 years ago

Oh, missed the headers in OPAL_DEFAULT_UPDATE_CALLBACKS. I thought once having OPAL_CLIENT_TOKEN, then no need. Because we can still point to other servers, it makes sense!

hongbo-miao commented 2 years ago

Hi @asafc for better user experience, I actually think if we only provide OPAL_SHOULD_REPORT_ON_DATA_UPDATES=True without OPAL_DEFAULT_UPDATE_CALLBACKS, then it can point to opal-server by default.

asafc commented 2 years ago

That's a good idea, and in general I have big plans for OPAL usability :) We will soon move to a yaml based configuration with much better docs and sane defaults, which is more cloud-native-y and suitable for the kubernetes ecosystem.

hongbo-miao commented 2 years ago

Hmm, I was happy too early, I still have issue.

Here is my OPAL_DEFAULT_UPDATE_CALLBACKS. You can see the token inside above is same with my OPAL_CLIENT_TOKEN:

image

However, I still got

[2021-10-22 21:49:34 +0000] [1] [INFO] Starting gunicorn 20.1.0
[2021-10-22 21:49:34 +0000] [1] [INFO] Listening at: http://0.0.0.0:7000 (1)
[2021-10-22 21:49:34 +0000] [1] [INFO] Using worker: uvicorn.workers.UvicornWorker
[2021-10-22 21:49:34 +0000] [9] [INFO] Booting worker with pid: 9
2021-10-22T21:49:35.431731+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'FastApiRpcFetchProvider' found at: <class 'opal_common.fetcher.providers.fastapi_rpc_fetch_provider.FastApiRpcFetchProvider'>
2021-10-22T21:49:35.432953+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'HttpFetchProvider' found at: <class 'opal_common.fetcher.providers.http_fetch_provider.HttpFetchProvider'>
2021-10-22T21:49:35.460963+0000 | opal_common.fetcher.fetcher_register    | INFO  | Loading FetcherProvider 'PostgresFetchProvider' found at: <class 'opal_fetcher_postgres.provider.PostgresFetchProvider'>
2021-10-22T21:49:35.461154+0000 | opal_common.fetcher.fetcher_register    | INFO  | Fetcher Register loaded
2021-10-22T21:49:35.462163+0000 | opal_client.callbacks.register          | INFO  | Callbacks register loaded
2021-10-22T21:49:35.462503+0000 | opal_client.client                      | INFO  | API authentication disabled (public encryption key was not provided)
2021-10-22T21:49:35.506748+0000 | uvicorn.server                          | INFO  | Started server process [9]
2021-10-22T21:49:35.506914+0000 | uvicorn.lifespan.on                     | INFO  | Waiting for application startup.
2021-10-22T21:49:35.507113+0000 | opal_client.opa.runner                  | INFO  | Launching opa runner
2021-10-22T21:49:35.507249+0000 | uvicorn.lifespan.on                     | INFO  | Application startup complete.
2021-10-22T21:49:35.509496+0000 | opal_client.opa.runner                  | INFO  | Running OPA inline: opa run --server --addr=:8181 --authentication=off --authorization=off --config-file=/data/opa/config.yaml --log-level=info
2021-10-22T21:49:35.533237+0000 | opal_client.opa.logger                  | INFO  | Initializing server. {'addrs': [':8181'], 'diagnostic-addrs': [], 'time': '2021-10-22T21:49:35Z'}
2021-10-22T21:49:35.534107+0000 | opal_client.opa.logger                  | INFO  | Starting decision logger. {'plugin': 'decision_logs', 'time': '2021-10-22T21:49:35Z'}
2021-10-22T21:49:36.515751+0000 | opal_client.opa.runner                  | INFO  | Running OPA initial start callbacks
2021-10-22T21:49:36.516605+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-22T21:49:36.519933+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41718', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.521234+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41718', 'req_id': 1, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 1.3585, 'resp_status': 200, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.522459+0000 | opal_client.data.updater                | INFO  | Launching data updater
2021-10-22T21:49:36.522656+0000 | opal_client.policy.updater              | INFO  | Launching policy updater
2021-10-22T21:49:36.522757+0000 | opal_client.data.updater                | INFO  | Subscribing to topics: ['policy_data']
2021-10-22T21:49:36.522958+0000 | opal_client.policy.updater              | INFO  | Subscribing to topics: ['policy:.']
2021-10-22T21:49:36.523243+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.523601+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.526468+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Trying to connect to Pub/Sub server - ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.526750+0000 | fastapi_websocket_rpc.websocket_rpc_c...| INFO  | Trying server - ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.588082+0000 | opal_client.data.updater                | INFO  | Connected to server
2021-10-22T21:49:36.588235+0000 | opal_client.data.updater                | INFO  | Performing data configuration, reason: Initial load
2021-10-22T21:49:36.588487+0000 | opal_client.data.updater                | INFO  | Getting data-sources configuration from 'http://opal-server-service.hm-opa:7002/data/config'
2021-10-22T21:49:36.597174+0000 | opal_client.data.updater                | INFO  | Triggering data update with id: 4161dbc4465d48dd869157022fd91cdb
2021-10-22T21:49:36.597429+0000 | opal_client.data.updater                | INFO  | Fetching policy data
2021-10-22T21:49:36.597797+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: postgresql://admin@opa-db-service.hm-opa:40072/opa_db
2021-10-22T21:49:36.615697+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.685550+0000 | opal_client.policy.updater              | INFO  | Connected to server
2021-10-22T21:49:36.685856+0000 | opal_client.policy.updater              | INFO  | Refetching policy code (full bundle)
2021-10-22T21:49:36.695059+0000 | opal_client.data.updater                | INFO  | Saving fetched data to policy-store: source url='postgresql://admin@opa-db-service.hm-opa:40072/opa_db', destination path='/roles'
2021-10-22T21:49:36.700004+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41734', 'req_id': 2, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.701573+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41734', 'req_id': 2, 'req_method': 'PUT', 'req_path': '/v1/data/roles', 'resp_bytes': 0, 'resp_duration': 0.3013, 'resp_status': 204, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.702588+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '4161dbc4465d48dd869157022fd91cdb', 'actions': ['set_policy_data'], 'success': True, 'error': ''}
2021-10-22T21:49:36.702711+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=false, healthy=false
2021-10-22T21:49:36.707346+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41736', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.709756+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41736', 'req_id': 3, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 2.0526, 'resp_status': 200, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.711007+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks
2021-10-22T21:49:36.711655+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-22T21:49:36.714894+0000 | opal_client.policy.updater              | INFO  | Got policy bundle with 1 rego files, 0 data files, commit hash: '910d43e7e4da262d838c562ce3ae9910600fe445'
2021-10-22T21:49:36.717767+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41740', 'req_id': 4, 'req_method': 'GET', 'req_path': '/v1/policies', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.719894+0000 | opal_client.callbacks.reporter          |ERROR  | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
2021-10-22T21:49:36.723232+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41740', 'req_id': 4, 'req_method': 'GET', 'req_path': '/v1/policies', 'resp_bytes': 5056, 'resp_duration': 5.1659, 'resp_status': 200, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.726057+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41742', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.728361+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41742', 'req_id': 5, 'req_method': 'PUT', 'req_path': '/v1/policies/rbac.rego', 'resp_bytes': 2, 'resp_duration': 2.3008, 'resp_status': 200, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.729521+0000 | opal_client.policy_store.opa_client     | INFO  | processing store transaction: {'id': '910d43e7e4da262d838c562ce3ae9910600fe445', 'actions': ['set_policies'], 'success': True, 'error': ''}
2021-10-22T21:49:36.729683+0000 | opal_client.policy_store.opa_client     | INFO  | persisting health check policy: ready=true, healthy=true
2021-10-22T21:49:36.731456+0000 | opal_client.callbacks.reporter          | INFO  | Reporting the update to requested callbacks
2021-10-22T21:49:36.731769+0000 | opal_client.data.fetcher                | INFO  | Fetching data from url: http://opal-server-service.hm-opa:7002/data/callback_report
2021-10-22T21:49:36.734616+0000 | opal_client.opa.logger                  | INFO  | Received request.    {'client_addr': '[::1]:41744', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.738409+0000 | opal_client.opa.logger                  | INFO  | Sent response.       {'client_addr': '[::1]:41744', 'req_id': 6, 'req_method': 'PUT', 'req_path': '/v1/policies/opa/healthcheck/opal.rego', 'resp_bytes': 2, 'resp_duration': 3.7296, 'resp_status': 200, 'time': '2021-10-22T21:49:36Z'}
2021-10-22T21:49:36.741288+0000 | fastapi_websocket_pubsub.pub_sub_client | INFO  | Connected to PubSub server ws://opal-server-service.hm-opa:7002/ws
2021-10-22T21:49:36.747092+0000 | opal_client.callbacks.reporter          |ERROR  | Failed to send report to http://opal-server-service.hm-opa:7002/data/callback_report, got response code 401 with error: {'detail': {'error': 'access token was not provided', 'token': None}}
asafc commented 2 years ago

Hey @Hongbo-Miao, you are right, we have a bug in 0.1.16 with Pydantic model unpacking of OPAL_DEFAULT_UPDATE_CALLBACKS, thanks for reporting!! :)

I was not able to fix it on the fly (will keep this issue open), but i have a quick workaround for you.

use this configuration values instead: (you simply move the callback http config to a different config var - OPAL_DEFAULT_UPDATE_CALLBACK_CONFIG)

opal_client:
    ...
    environment:
      ...
      - OPAL_CLIENT_TOKEN=MY_CLIENT_JWT_TOKEN
      - OPAL_DEFAULT_UPDATE_CALLBACKS={"callbacks":["http://opal_server:7002/data/callback_report"]}
      - OPAL_DEFAULT_UPDATE_CALLBACK_CONFIG={"method":"post","headers":{"Authorization":"Bearer MY_CLIENT_JWT_TOKEN","content-type":"application/json"},"process_data":"false"}

My full configuration ahead for completeness of the answer (i used volumes for ssh keys for speed, but you can just put the actual public and private key inline as you always do):

version: "3.8"
services:
  # When scaling the opal-server to multiple nodes and/or multiple workers, we use
  # a *broadcast* channel to sync between all the instances of opal-server.
  # Under the hood, this channel is implemented by encode/broadcaster (see link below).
  # At the moment, the broadcast channel can be either: postgresdb, redis or kafka.
  # The format of the broadcaster URI string (the one we pass to opal server as `OPAL_BROADCAST_URI`) is specified here:
  # https://github.com/encode/broadcaster#available-backends
  broadcast_channel:
    image: postgres:alpine
    environment:
      - POSTGRES_DB=postgres
      - POSTGRES_USER=postgres
      - POSTGRES_PASSWORD=postgres
  opal_server:
    # by default we run opal-server from latest official image
    image: authorizon/opal-server:next
    environment:
      # the broadcast backbone uri used by opal server workers (see comments above for: broadcast_channel)
      - OPAL_BROADCAST_URI=postgres://postgres:postgres@broadcast_channel:5432/postgres
      # number of uvicorn workers to run inside the opal-server container
      - UVICORN_NUM_WORKERS=4
      # the git repo hosting our policy
      # - if this repo is not public, you can pass an ssh key via `OPAL_POLICY_REPO_SSH_KEY`)
      # - the repo we pass in this example is *public* and acts as an example repo with dummy rego policy
      # - for more info, see: https://github.com/authorizon/opal/blob/master/docs/HOWTO/track_a_git_repo.md
      - OPAL_POLICY_REPO_URL=https://github.com/authorizon/opal-example-policy-repo
      # in this example we will use a polling interval of 30 seconds to check for new policy updates (git commits affecting the rego policy).
      # however, it is better to utilize a git *webhook* to trigger the server to check for changes only when the repo has new commits.
      # for more info see: https://github.com/authorizon/opal/blob/master/docs/HOWTO/track_a_git_repo.md
      - OPAL_POLICY_REPO_POLLING_INTERVAL=30
      # configures from where the opal client should initially fetch data (when it first goes up, after disconnection, etc).
      # the data sources represents from where the opal clients should get a "complete picture" of the data they need.
      # after the initial sources are fetched, the client will subscribe only to update notifications sent by the server.
      - OPAL_DATA_CONFIG_SOURCES={"config":{"entries":[{"url":"http://opal_server:7002/policy-data","config":{"headers":{"Authorization":"Bearer MY_CLIENT_JWT_TOKEN"}},"topics":["policy_data"]}]}}
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      - OPAL_AUTH_PUBLIC_KEY=/var/secrets/opal_rsa.pub
      - OPAL_AUTH_PRIVATE_KEY=/var/secrets/opal_rsa
      - OPAL_AUTH_MASTER_TOKEN=secret
    ports:
      # exposes opal server on the host machine, you can access the server at: http://localhost:7002
      - "7002:7002"
    volumes:
      - ~/.ssh:/var/secrets
    depends_on:
      - broadcast_channel
  opal_client:
    # by default we run opal-client from latest official image
    image: authorizon/opal-client:next
    environment:
      - OPAL_SERVER_URL=http://opal_server:7002
      - OPAL_CLIENT_TOKEN=MY_CLIENT_JWT_TOKEN
      - OPAL_LOG_FORMAT_INCLUDE_PID=true
      - OPAL_INLINE_OPA_LOG_FORMAT=http
      # update callbacks config ----------------------------------
      # this var turns on a callback (HTTP call to a configurable url) after every successful data update
      # and allows you to track which data updates completed successfully and were saved to OPA cache.
      - OPAL_SHOULD_REPORT_ON_DATA_UPDATES=True
      # we configure a callback to go to a default location in the OPAL server, but you can configure
      # a callback to any url you'd like. Each callback is either the url alone, or a tuple of
      # (url, HttpFetcherConfig).
      # We show here both ways to configure the same endpoint, one of them demonstrate how to
      # add extra HTTP headers (the header shown is ignored, only here for example).
      - OPAL_DEFAULT_UPDATE_CALLBACKS={"callbacks":["http://opal_server:7002/data/callback_report"]}
      - OPAL_DEFAULT_UPDATE_CALLBACK_CONFIG={"method":"post","headers":{"Authorization":"Bearer MY_CLIENT_JWT_TOKEN","content-type":"application/json"},"process_data":"false"}
      # OPAL can load a special policy into OPA that acts as a healthcheck policy (Not directly related to the callback feature).
      # This policy defines two opa rules you can query:
      # ready rule (POST http://localhost:8181/data/system/opal/ready): signals that OPA is ready to accept authorization queries.
      # healthy rule (POST http://localhost:8181/data/system/opal/ready): signals that the last policy and data updates succeeded.
      - OPAL_OPA_HEALTH_CHECK_POLICY_ENABLED=True
      # end of update callbacks config ---------------------------
    ports:
      # exposes opal client on the host machine, you can access the client at: http://localhost:7000
      - "7000:7000"
      # exposes the OPA agent (being run by OPAL) on the host machine
      # you can access the OPA api that you know and love at: http://localhost:8181
      # OPA api docs are at: https://www.openpolicyagent.org/docs/latest/rest-api/
      - "8181:8181"
    depends_on:
      - opal_server
    # this command is not necessary when deploying OPAL for real, it is simply a trick for dev environments
    # to make sure that opal-server is already up before starting the client.
    command: sh -c "/usr/wait-for.sh opal_server:7002 --timeout=20 -- /start.sh"

Btw, found another bug while investigating this: already have outstanding pull with #168 and will merge on Sunday.