bug: google-cloud-logging plugin refresh_access_token(): failed to refresh google oauth access token

zkl94 commented 3 months ago

Current Behavior

it is not able to send request logs to google cloud logging now due to this bug.

Expected Behavior

The logs should be sent to google cloud logging successfully

Error Logs

2024/04/18 19:03:44 [error] 55#55: 67159 [lua] oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443 2024/04/18 19:03:44 [error] 55#55: 67159 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443

Steps to Reproduce

run apisix helm chart https://artifacthub.io/packages/helm/apisix/apisix on k8s 1.27 on GKE

create the below ApisixGlobalRule CR (the GCP SA has owner access just in case of permission issue)


apiVersion: apisix.apache.org/v2
kind: ApisixGlobalRule
metadata:
name: google-cloud-logging
spec:
plugins:
- name: google-cloud-logging
  enable: true
  config:
    auth_config:
      client_email: "xxxxxx"
      private_key_id: "xxxxx"
      client_id: "xxxxx"
      private_key: "xxxxx"
      project_id: "xxxxx"
      token_uri: "https://oauth2.googleapis.com/token"
      entries_uri: "https://logging.googleapis.com/v2/entries:write"
      auth_uri: "https://accounts.google.com/o/oauth2/auth"
      ssl_verify: false
    batch_max_size: 1
    retry_delay: 1
    inactive_timeout: 10
    max_retry_count: 10

3. just in case it is not applied globally, I added the google-cloud-logging section to ApisixRoute as well:

apiVersion: apisix.apache.org/v2 kind: ApisixRoute metadata: name: apisix-dashboard spec: http:

name: dashboard match: hosts:
- xxxx.example.com paths:
- /* backends:
  - serviceName: apisix-dashboard servicePort: 80 plugins:
  - name: redirect enable: true config: http_to_https: true
  - name: google-cloud-logging enable: true config: auth_config: client_email: "xxxxx" private_key_id: "xxxx" client_id: "xxx" private_key: "xxxx" project_id: "xxxxx" token_uri: "https://oauth2.googleapis.com/token" entries_uri: "https://logging.googleapis.com/v2/entries:write" auth_uri: "https://accounts.google.com/o/oauth2/auth" ssl_verify: false batch_max_size: 1 retry_delay: 1 inactive_timeout: 10 max_retry_count: 10
```
4. visit xxxx.example.com on browser multiple times
5. check the logs of apisix pods inside kubernetes and we get the error logs:
```
    2024/04/18 19:03:53 [error] 55#55: 67552 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443 2024/04/18 19:03:53 [error] 55#55: 67552 [lua] batch-processor.lua:104: Batch Processor[google-cloud-logging] exceeded the max_retry_count[11] dropping the entries, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443 2024/04/18 19:03:53 [error] 55#55: 67554 [lua] oauth.lua:62: refresh_access_token(): failed to refresh google oauth access token, 20: unable to get local issuer certificate, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443 2024/04/18 19:03:53 [error] 55#55: 67554 [lua] batch-processor.lua:95: Batch Processor[google-cloud-logging] failed to process entries: failed to get google oauth token, context: ngx.timer, client: 10.186.156.41, server: 0.0.0.0:9443
```
6. I checked google-cloud-logging as well, the request logs are not received
```

Environment

APISIX version (run apisix version): 3.8.0
Operating system (run uname -a): (I run it inside the apisix pod) Linux apisix-6bc8f67474-mk5xj 5.15.133+ SMP Wed Nov 8 17:30:28 UTC 2023 x86_64 GNU/Linux
OpenResty / Nginx version (run openresty -V or nginx -V): nginx version: openresty/1.21.4.2
etcd version, if relevant (run curl http://127.0.0.1:9090/v1/server_info):
APISIX Dashboard version, if relevant:
Plugin runner version, for issues related to plugin runners:
LuaRocks version, for installation issues (run luarocks --version):

nitishfy commented 3 months ago

Thanks for raising this issue! It seems like the SSL/TLS certificate of the server issuing the OAuth token cannot be verified.

zkl94 commented 3 months ago

@nitishfy Could you suggest your solution? According to my understanding, Google is the entity issuing the OAuth token. What is the wrong configuration that you see here? Are you saying that google SSL/TLS certificate cannot be verified?

balaji-n4 commented 2 months ago

@nitishfy could you explain how to fix the issue or any work around for the issue?

apache / apisix