Add monitoring to GCP cloud Logging

xellos00 commented 5 months ago

1. Type of change

Please delete options that are not relevant.

[x] New feature
[ ] Enhancement
[ ] Bug/fix (non-breaking change which fixes an issue)
[ ] others (anything other than above)

2. Summary

Please include a summary of the changes and which issue is fixed or solved.

close #556

Summary

Add feature that register every minute that VAtz status into Google Cloud logging

3. Comments

Please, leave a comments if there's further action that requires.

xellos00 commented 4 months ago

@heejin-github I found some bugs on the codes, will update to tomorrow, and send gcp credentials as well.

xellos00 commented 4 months ago

Confirmed that plugin status changes through its current status.

@dsrvlabs/validator Please, follow the instruction to test this PR

https://docs.google.com/document/d/1e4nQvPXGL8pJqNYtzEwwIfX8FOBrjfUdhTej8VM6Zdc/edit?usp=sharing

heejin-github commented 4 months ago

with APIKey type credential, I got this error from vatz.

2024/07/08 13:06:33 API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2024-07-08T13:06:33Z INF start rpc server module=rpc
2024-07-08T13:06:33Z INF start gRPC gateway server :19091 module=rpc
2024-07-08T13:06:33Z INF start gRPC server :19090 module=rpc
2024-07-08T13:06:33Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more inf                                                                                            ormation module="monitoring > Init"

It seems that API key support for gRPC API from google api client library v0.155.0 https://github.com/googleapis/google-api-go-client/commit/9dbfb73d31f050b998cf111b09a084c07583bcb3

If I updated google api client library from current v0.114.0 to the latest(v0.187.0) this issue was fixed. should we update this package?

heejin-github commented 4 months ago

There's another issue on IP address restriction for api key. It seems that IP address restriction based on IPv4 address. however in some cases, It was failed to use api key due to it was originating with IPv6 address(2a03:b0c0:3:d0::12e4:c001) from the server like below log.

2024/07/08 13:23:01 logging client: rpc error: code = PermissionDenied desc = The provided API key has an IP address restriction. The originating IP address of the call (2a03:b0c0:3:d0::12e4:c001) violates this restriction.
error details: name = ErrorInfo reason = API_KEY_IP_ADDRESS_BLOCKED domain = googleapis.com metadata = map[consumer:projects/185011769942 service:logging.googleapis.com]

I think below fix would be one of idea for this issue, we explicitly uses "tcp4", which forces IPv4 in grpc dialer.

diff --git a/monitoring/gcp/gcp.go b/monitoring/gcp/gcp.go
index 8df32b9..ad0e1a9 100644
--- a/monitoring/gcp/gcp.go
+++ b/monitoring/gcp/gcp.go
@@ -12,6 +12,8 @@ import (
        "google.golang.org/api/option"
        "sync"
        "time"
+       "net"
+       "google.golang.org/grpc"
 )

 type GCP interface {
@@ -53,19 +55,25 @@ func getClient(ctx context.Context, projectID string, credType tp.CredentialOpti
        var client *logging.Client
        var err error

+       customDialer := func(ctx context.Context, addr string) (net.Conn, error) {
+               return (&net.Dialer{}).DialContext(ctx, "tcp4", addr)
+       }
+
+       dialOption := grpc.WithContextDialer(customDialer)
+
        switch credType {
        case tp.ApplicationDefaultCredentials:
-               client, err = logging.NewClient(ctx, projectID)
+               client, err = logging.NewClient(ctx, projectID, option.WithGRPCDialOption(dialOption))
        case tp.ServiceAccountCredentials:
-               client, err = logging.NewClient(ctx, projectID, option.WithCredentialsFile(credentials))
+               client, err = logging.NewClient(ctx, projectID, option.WithCredentialsFile(credentials), option.WithGRPCDialOption(dialOption))
        case tp.APIKey:
-               client, err = logging.NewClient(ctx, projectID, option.WithAPIKey(credentials))
+               client, err = logging.NewClient(ctx, projectID, option.WithAPIKey(credentials), option.WithGRPCDialOption(dialOption))
        case tp.OAuth2:
                tokenSource, err := google.DefaultTokenSource(ctx, logging.WriteScope)
                if err != nil {
                        return nil, err
                }
-               client, err = logging.NewClient(ctx, projectID, option.WithTokenSource(tokenSource))
+               client, err = logging.NewClient(ctx, projectID, option.WithTokenSource(tokenSource), option.WithGRPCDialOption(dialOption))
                if err != nil {
                        return nil, err
                }

heejin-github commented 4 months ago

Finally, I got IAM permission denied error. 😱

2024-07-08T14:52:00Z INF Store Logs into Cloud logging for chainlink, chainlink-agoric-devnet-node1-do-fra1 module=monitoring
2024/07/08 14:52:01 logging client: rpc error: code = PermissionDenied desc = Permission 'logging.logEntries.create' denied on resource (or it may not exist).
error details: name = ErrorInfo reason = IAM_PERMISSION_DENIED domain = iam.googleapis.com metadata = map[permission:logging.logEntries.create]
error details: name = Unknown  desc = log_entry_errors:{key:0 value:{code:7 message:"Permission 'logging.logEntries.create' denied on resource (or it may not exist)."}} log_entry_errors:{key:1 value:{code:7 message:"Permission 'logging.logEntries.create' denied on resource (or it may not exist)."}}

xellos00 commented 4 months ago

with APIKey type credential, I got this error from vatz.

2024/07/08 13:06:33 API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2024-07-08T13:06:33Z INF start rpc server module=rpc
2024-07-08T13:06:33Z INF start gRPC gateway server :19091 module=rpc
2024-07-08T13:06:33Z INF start gRPC server :19090 module=rpc
2024-07-08T13:06:33Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more inf                                                                                            ormation module="monitoring > Init"

It seems that API key support for gRPC API from google api client library v0.155.0 googleapis/google-api-go-client@9dbfb73

If I updated google api client library from current v0.114.0 to the latest(v0.187.0) this issue was fixed. should we update this package?

Finally, I got IAM permission denied error. 😱

2024-07-08T14:52:00Z INF Store Logs into Cloud logging for chainlink, chainlink-agoric-devnet-node1-do-fra1 module=monitoring
2024/07/08 14:52:01 logging client: rpc error: code = PermissionDenied desc = Permission 'logging.logEntries.create' denied on resource (or it may not exist).
error details: name = ErrorInfo reason = IAM_PERMISSION_DENIED domain = iam.googleapis.com metadata = map[permission:logging.logEntries.create]
error details: name = Unknown  desc = log_entry_errors:{key:0 value:{code:7 message:"Permission 'logging.logEntries.create' denied on resource (or it may not exist)."}} log_entry_errors:{key:1 value:{code:7 message:"Permission 'logging.logEntries.create' denied on resource (or it may not exist)."}}

Thanks for the all comments, let me figure this our with our credentials. will get back to you shortly.

xellos00 commented 4 months ago

with APIKey type credential, I got this error from vatz.
2024/07/08 13:06:33 API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2024-07-08T13:06:33Z INF start rpc server module=rpc
2024-07-08T13:06:33Z INF start gRPC gateway server :19091 module=rpc
2024-07-08T13:06:33Z INF start gRPC server :19090 module=rpc
2024-07-08T13:06:33Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more inf                                                                                            ormation module="monitoring > Init"
It seems that API key support for gRPC API from google api client library v0.155.0 googleapis/google-api-go-client@9dbfb73

If I updated google api client library from current v0.114.0 to the latest(v0.187.0) this issue was fixed. should we update this package?

You don't have to even if you upgrade it, it would create another error something like below which doesn't send any message to GCP logging.

2024-07-08T18:49:00-05:00 INF Store Logs into Cloud logging for hyperlane, hyperlane-mainnet-validator12-mantapacific-aws-frankfurt module=monitoring
2024/07/08 18:49:01 logging client: rpc error: code = PermissionDenied desc = Permission 'logging.logEntries.create' denied on resource (or it may not exist).
error details: name = ErrorInfo reason = IAM_PERMISSION_DENIED domain = iam.googleapis.com metadata = map[permission:logging.logEntries.create]

@heejin-github You can skip or ignore this error message because it frankly send message to GCP. btw will update code soon.

xellos00 commented 4 months ago

with APIKey type credential, I got this error from vatz.
2024/07/08 13:06:33 API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2024-07-08T13:06:33Z INF start rpc server module=rpc
2024-07-08T13:06:33Z INF start gRPC gateway server :19091 module=rpc
2024-07-08T13:06:33Z INF start gRPC server :19090 module=rpc
2024-07-08T13:06:33Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more inf                                                                                            ormation module="monitoring > Init"
It seems that API key support for gRPC API from google api client library v0.155.0 googleapis/google-api-go-client@9dbfb73 If I updated google api client library from current v0.114.0 to the latest(v0.187.0) this issue was fixed. should we update this package?

You don't have to even if you upgrade it, it would create another error something like below which doesn't send any message to GCP logging.

2024-07-08T18:49:00-05:00 INF Store Logs into Cloud logging for hyperlane, hyperlane-mainnet-validator12-mantapacific-aws-frankfurt module=monitoring
2024/07/08 18:49:01 logging client: rpc error: code = PermissionDenied desc = Permission 'logging.logEntries.create' denied on resource (or it may not exist).
error details: name = ErrorInfo reason = IAM_PERMISSION_DENIED domain = iam.googleapis.com metadata = map[permission:logging.logEntries.create]

@heejin-github

With previous version 2dc55505, you can send Plugin Status to Cloud Logger with the current APIKey without update googleapis to version you mentioned above, even if you see the message that API keys are not supported for gRPC APIs. Please, test without any change, and ignore API keys are not supported for gRPC APIs.
The latest commit version, which has upgraded the gRPC API from google api and changed it to be processed in Rest API format, does not allow google cloud API to create messages in google cloud using an API key. Ultimately, the current Service Account method must be used.

xellos00 commented 4 months ago

from @rootwarp

Logger 만 안된다는 것이 이상하다. API key
bug 인지, 아니면 library 를 확인해야한다.

Research on latest version of google-api-key by this week.

xellos00 commented 4 months ago

@heejin-github @rootwarp @meetrick Can you guys review again? Please, let me know if you guys would like to register your own ip address in apiKey restriction.

Reference to
- https://docs.google.com/document/d/1e4nQvPXGL8pJqNYtzEwwIfX8FOBrjfUdhTej8VM6Zdc/edit?usp=sharing

heejin-github commented 4 months ago

@heejin-github @rootwarp @meetrick Can you guys review again? Please, let me know if you guys would like to register your own ip address in apiKey restriction.

Reference to

https://docs.google.com/document/d/1e4nQvPXGL8pJqNYtzEwwIfX8FOBrjfUdhTej8VM6Zdc/edit?usp=sharing

@xellos00 As I told you in the bi-weekly meeting, failed to get GCP client with following error.

2024-07-11T11:07:09Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information module="monitoring > Init"

Can I ask any wrong in my config?

root@chainlink-agoric-devnet-node1-do-fra1:~/dsrv/config# cat vatz_config.yaml
vatz_protocol_info:
  home_path: "~/.vatz"
  protocol_identifier: "chainlink"
  port: 9050
  health_checker_schedule:
    - "0 1 * * *"
  notification_info:
    host_name: "chainlink-agoric-devnet-node1-do-fra1"
    default_reminder_schedule:
      - "*/30 * * * *"
    dispatch_channels:
      - channel: "discord"
        secret: <REMOVED>
  rpc_info:
    enabled: true
    address: "127.0.0.1"
    grpc_port: 19090
    http_port: 19091
  monitoring_info:
    gcp:
      gcp_cloud_logging_info:
        enabled: true
        cloud_logging_credential_info:
          project_id: "validator-test-unlimited"
          credentials_type: "APIKey"
          credentials: <REMOVED>
          checker_schedule:
            - "* * * * *"

xellos00 commented 4 months ago

@heejin-github @rootwarp @meetrick Can you guys review again? Please, let me know if you guys would like to register your own ip address in apiKey restriction.

Reference to

https://docs.google.com/document/d/1e4nQvPXGL8pJqNYtzEwwIfX8FOBrjfUdhTej8VM6Zdc/edit?usp=sharing

@xellos00 As I told you in the bi-weekly meeting, failed to get GCP client with following error.

2024-07-11T11:07:09Z ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information module="monitoring > Init"

Can I ask any wrong in my config?

root@chainlink-agoric-devnet-node1-do-fra1:~/dsrv/config# cat vatz_config.yaml
vatz_protocol_info:
  home_path: "~/.vatz"
  protocol_identifier: "chainlink"
  port: 9050
  health_checker_schedule:
    - "0 1 * * *"
  notification_info:
    host_name: "chainlink-agoric-devnet-node1-do-fra1"
    default_reminder_schedule:
      - "*/30 * * * *"
    dispatch_channels:
      - channel: "discord"
        secret: <REMOVED>
  rpc_info:
    enabled: true
    address: "127.0.0.1"
    grpc_port: 19090
    http_port: 19091
  monitoring_info:
    gcp:
      gcp_cloud_logging_info:
        enabled: true
        cloud_logging_credential_info:
          project_id: "validator-test-unlimited"
          credentials_type: "APIKey"
          credentials: <REMOVED>
          checker_schedule:
            - "* * * * *"

I will get back to you shortly after I confirm its solution, I think your config is all good.

meetrick commented 4 months ago

Test with Local (Macboook air)

~/project_local/dsrv
base ❯ git clone https://github.com/dsrvlabs/vatz.git
Cloning into 'vatz'...
remote: Enumerating objects: 1279, done.
remote: Counting objects: 100% (582/582), done.
remote: Compressing objects: 100% (317/317), done.
remote: Total 1279 (delta 325), reused 423 (delta 249), pack-reused 697
Receiving objects: 100% (1279/1279), 532.00 KiB | 8.72 MiB/s, done.
Resolving deltas: 100% (653/653), done.

~/project_local/dsrv
base ❯ cd vatz

~/project_local/dsrv/vatz main*
base ❯ ./vatz init
2024-07-16T04:36:43+09:00 INF Initialize DB /Users/hwangjae/.vatz/vatz.db module=db

~/project_local/dsrv/vatz main*
base ❯ ./vatz plugin install github.com/dsrvlabs/vatz-plugin-sysutil/plugins/cpu_monitor cpu_monitor
2024-07-16T04:37:35+09:00 INF A new plugin cpu_monitor is successfully installed. module=plugin

default.yaml

~/project_local/dsrv/vatz main* 7s
base ❯ cat default.yaml
vatz_protocol_info:
home_path: "~/.vatz"
protocol_identifier: "GCP cloud logging test"
port: 9090
health_checker_schedule:
- "0 1 * * *"
notification_info:
host_name: "Hwangjae's macbook air"
default_reminder_schedule:
  - "*/30 * * * *"
dispatch_channels:
  - channel: "discord"
    secret: "https://discord.com/api/webhooks/1030633613420675153/k4sfMQT_QCkiJgC92FI_odbX75Efp7IrP8nrPdAV8Qdh7-7LHGFzPBbUPG5iSo51VzjV"
    reminder_schedule:
      - "*/5 * * * *"
rpc_info:
enabled: true
address: "127.0.0.1"
grpc_port: 19090
http_port: 19091
monitoring_info:
gcp:
  gcp_cloud_logging_info:
    enabled: true
    cloud_logging_credential_info:
      project_id: "Please, Set your GCP Project id"
      credentials_type: "Check the Credential Type: ADC: Application, SAC: Default Credentials, Service Account Credentials, APIKey: API Key, OAuth: OAuth2"
      credentials: "Put your credential Info"
      checker_schedule:
        - "* * * * *"
prometheus:
  enabled: true
  address: "127.0.0.1"
  port: 18080
plugins_infos:
default_verify_interval: 15
default_execute_interval: 30
default_plugin_name: "vatz-plugin"
plugins:
- plugin_name: "cpu_monitor"
  plugin_address: "localhost"
  plugin_port: 9001
  executable_methods:
    - method_name: "cpu_monitor"

~/project_local/dsrv/vatz main*
base ❯ ./vatz plugin start --plugin cpu_monitor --args "-port 9001" --log cpu_monitor.logs
2024-07-16T04:45:12+09:00 INF Start plugin cpu_monitor -port 9001 module=plugin
2024-07-16T04:45:12+09:00 INF Plugin cpu_monitor is successfully started. module=plugin

~/project_local/dsrv/vatz main*
base ❯ tail -f cpu_monitor.logs
2024-07-16T04:45:13+09:00 INF Register module=grpc
2024-07-16T04:45:13+09:00 INF Start 127.0.0.1 9001 module=sdk
2024-07-16T04:45:13+09:00 INF Start module=grpc

~/project_local/dsrv/vatz main* 14s
base ❯ ./vatz start --config default.yaml >> vatz.log 2>&1 &
[1] 99609

~/project_local/dsrv/vatz main* 53s
base ❯ tail -f vatz.log
2024-07-16T04:45:33+09:00 INF Initialize Server module=main
2024-07-16T04:45:33+09:00 INF Start VATZ Server on Listening Port: :9090 module=main
2024-07-16T04:45:33+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:45:33+09:00 INF start metric server: 127.0.0.1:18080 module=main
2024-07-16T04:45:33+09:00 INF start rpc server module=rpc
2024-07-16T04:45:33+09:00 INF start gRPC gateway server 127.0.0.1:19091 module=rpc
2024-07-16T04:45:33+09:00 INF start gRPC server 127.0.0.1:19090 module=rpc
2024-07-16T04:45:33+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:46:03+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-16T04:46:03+09:00 INF response: SUCCESS module=executor
2024-07-16T04:46:35+09:00 INF Initialize Server module=main
2024-07-16T04:46:35+09:00 INF Start VATZ Server on Listening Port: :9090 module=main
2024-07-16T04:46:35+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:46:35+09:00 INF start metric server: 127.0.0.1:18080 module=main
2024-07-16T04:46:35+09:00 INF start rpc server module=rpc
2024-07-16T04:46:35+09:00 INF start gRPC server 127.0.0.1:19090 module=rpc
2024-07-16T04:46:35+09:00 INF start gRPC gateway server 127.0.0.1:19091 module=rpc
2024-07-16T04:46:35+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:47:05+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-16T04:47:05+09:00 INF response: SUCCESS module=executor

Try-2

~/project_local/dsrv/vatz main*
base ❯ cat default.yaml
vatz_protocol_info:
  home_path: "~/.vatz"
  protocol_identifier: "GCP cloud logging test"
  port: 9090
  health_checker_schedule:
    - "0 1 * * *"
  notification_info:
    host_name: "Hwangjae's macbook air"
    default_reminder_schedule:
      - "*/30 * * * *"
    dispatch_channels:
      - channel: "discord"
        secret: "https://discord.com/api/webhooks/1030633613420675153/k4sfMQT_QCkiJgC92FI_odbX75Efp7IrP8nrPdAV8Qdh7-7LHGFzPBbUPG5iSo51VzjV"
        reminder_schedule:
          - "*/5 * * * *"
  rpc_info:
    enabled: true
    address: "127.0.0.1"
    grpc_port: 19090
    http_port: 19091
  monitoring_info:
    gcp:
      gcp_cloud_logging_info:
        enabled: true
        cloud_logging_credential_info:
          project_id: "validator-test-unlimited"
          credentials_type: "APIKey"
          credentials: "AIzaSyBtSNwa90y5J4SMl-hGk0xXd5o6uFE1PmA"
          checker_schedule:
            - "* * * * *"
    prometheus:
      enabled: true
      address: "127.0.0.1"
      port: 18080
plugins_infos:
  default_verify_interval: 15
  default_execute_interval: 30
  default_plugin_name: "vatz-plugin"
  plugins:
    - plugin_name: "cpu_monitor"
      plugin_address: "localhost"
      plugin_port: 9001
      executable_methods:
        - method_name: "cpu_monitor"

~/project_local/dsrv/vatz main*
base ❯ ./vatz plugin start --plugin cpu_monitor --args "-port 9001" --log cpu_monitor.logs
2024-07-16T04:53:19+09:00 INF Start plugin cpu_monitor -port 9001 module=plugin
2024-07-16T04:53:20+09:00 INF Plugin cpu_monitor is successfully started. module=plugin

~/project_local/dsrv/vatz main*
base ❯ tail -f cpu_monitor.logs

2024-07-16T04:53:20+09:00 INF Register module=grpc
2024-07-16T04:53:20+09:00 INF Start 127.0.0.1 9001 module=sdk
2024-07-16T04:53:20+09:00 INF Start module=grpc

~/project_local/dsrv/vatz main*
base ❯ ./vatz start --config default.yaml >> vatz.log 2>&1 &
[1] 1092

~/project_local/dsrv/vatz main*
base ❯ tail -f vatz.log

2024-07-16T04:53:47+09:00 INF Initialize Server module=main
2024-07-16T04:53:47+09:00 INF Start VATZ Server on Listening Port: :9090 module=main
2024-07-16T04:53:47+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:53:47+09:00 INF start rpc server module=rpc
2024-07-16T04:53:47+09:00 INF start gRPC server 127.0.0.1:19090 module=rpc
2024/07/16 04:53:47 API keys are not supported for gRPC APIs. Remove the WithAPIKey option from your client-creating call.
2024-07-16T04:53:47+09:00 INF start gRPC gateway server 127.0.0.1:19091 module=rpc
2024-07-16T04:53:47+09:00 ERR get GCP client for Logging Error: google: could not find default credentials. See https://cloud.google.com/docs/authentication/external/set-up-adc for more information module="monitoring > Init"
2024-07-16T04:53:47+09:00 INF start metric server: 127.0.0.1:18080 module=main
2024-07-16T04:53:47+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-16T04:54:17+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-16T04:54:17+09:00 INF response: SUCCESS module=executor
2024-07-16T04:54:47+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-16T04:54:47+09:00 INF response: SUCCESS module=executor

xellos00 commented 4 months ago

@rootwarp @heejin-github @meetrick

I apologize for the delay. I've updated the code to download Service Account (SA) credentials from S3, which is now secured by a bucket policy restricted to specific IP addresses. Additionally, the GCP client now utilizes SA credentials stored in memory on VATZ.

Please update your configuration file as per the guide available at:

Configuration Guide

If you have any questions or need further assistance, please let me know. oh, if you would like to test on your local, I need to add your local ip address, so let me know.

Thank you!

meetrick commented 4 months ago

2024-07-23 Vatz test in Macbook Pro

☁  vatz [main] git fetch origin pull/569/head:pr-569
remote: Enumerating objects: 130, done.
remote: Counting objects: 100% (130/130), done.
remote: Compressing objects: 100% (56/56), done.
remote: Total 93 (delta 51), reused 77 (delta 36), pack-reused 0
Unpacking objects: 100% (93/93), 24.88 KiB | 296.00 KiB/s, done.
From https://github.com/dsrvlabs/vatz
 * [new ref]         refs/pull/569/head -> pr-569
☁  vatz [main] git checkout pr-569
Switched to branch 'pr-569'
☁  vatz [pr-569] go build

☁  vatz [pr-569] ./vatz init
2024-07-23T10:29:55+09:00 INF Initialize DB /Users/meetrick/.vatz/vatz.db module=db

☁  vatz [pr-569] cat default.yaml
vatz_protocol_info:
  home_path: "~/.vatz"
  protocol_identifier: "Vatz GCP cloud logging test"
  port: 9090
  health_checker_schedule:
    - "0 1 * * *"
  notification_info:
    host_name: "Local - meetrick MacBook Pro"
    default_reminder_schedule:
      - "*/30 * * * *"
    dispatch_channels:
      - channel: "discord"
        secret: "https://discord.com/api/webhooks/945944481872629770/Rz2-5fwWPKKFDONPXSFHw5q-HE-3rvY-NbnAgqKf4mNvCbICXHIA1Xdcj1qMTt4B7JCZ"
  rpc_info:
    enabled: true
    address: "127.0.0.1"
    grpc_port: 19090
    http_port: 19091
  monitoring_info:
    gcp:
      gcp_cloud_logging_info:
        enabled: true
        cloud_logging_credential_info:
          project_id: "validator-test-unlimited"
          credentials_type: "SAC"
          credentials: "https://vatz-logging.s3.eu-central-1.amazonaws.com/validator-test-unlimited-sample.json"
          checker_schedule:
            - "* * * * *"
    prometheus:
      enabled: true
      address: "127.0.0.1"
      port: 18080
plugins_infos:
  default_verify_interval: 15
  default_execute_interval: 30
  default_plugin_name: "vatz-plugin"
  plugins:
    - plugin_name: "cpu_monitor"
      plugin_address: "localhost"
      plugin_port: 9001
      executable_methods:
        - method_name: "cpu_monitor"

☁  vatz [pr-569] ./vatz plugin install github.com/dsrvlabs/vatz-plugin-sysutil/plugins/cpu_monitor cpu_monitor
2024-07-23T10:36:27+09:00 INF A new plugin cpu_monitor is successfully installed. module=plugin

☁  vatz [pr-569] ./vatz start --config default.yaml >> vatz.log 2>&1 &
[1] 3260

☁  vatz [pr-569] ./vatz plugin start --plugin cpu_monitor --args 9001 --log cpu_monitor.log
2024-07-23T10:54:03+09:00 INF Start plugin cpu_monitor 9001 module=plugin
2024-07-23T10:54:03+09:00 INF Plugin cpu_monitor is successfully started. module=plugin

Log

☁  vatz [pr-569] tail -f cpu_monitor.log
2024-07-23T12:08:14+09:00 INF Register module=grpc
2024-07-23T12:08:14+09:00 INF Start 127.0.0.1 9001 module=sdk
2024-07-23T12:08:14+09:00 INF Start module=grpc
2024-07-23T12:08:45+09:00 INF Execute module=grpc
2024-07-23T12:08:45+09:00 DBG cpu_monitor CPU Usage=9 Urgent=95 Warning=90 module=plugin
2024-07-23T12:09:15+09:00 INF Execute module=grpc
2024-07-23T12:09:15+09:00 DBG cpu_monitor CPU Usage=10 Urgent=95 Warning=90 module=plugin
2024-07-23T12:09:45+09:00 INF Execute module=grpc
2024-07-23T12:09:45+09:00 DBG cpu_monitor CPU Usage=19 Urgent=95 Warning=90 module=plugin

☁  vatz [pr-569] tail -f vatz.log
 2024-07-23T12:08:14+09:00 INF Initialize Server module=main
2024-07-23T12:08:14+09:00 INF Start VATZ Server on Listening Port: :9090 module=main
2024-07-23T12:08:15+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-23T12:08:15+09:00 INF start rpc server module=rpc
2024-07-23T12:08:15+09:00 INF start gRPC gateway server 127.0.0.1:19091 module=rpc
2024-07-23T12:08:15+09:00 INF start gRPC server 127.0.0.1:19090 module=rpc
2024-07-23T12:08:16+09:00 INF start metric server: 127.0.0.1:18080 module=main
2024-07-23T12:08:16+09:00 INF Client successfully connected to localhost:9001 (plugin:cpu_monitor). module=util
2024-07-23T12:08:45+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-23T12:08:45+09:00 INF response: SUCCESS module=executor
2024-07-23T12:09:02+09:00 INF Store Logs into Cloud logging for Vatz GCP cloud logging test, Local - meetrick MacBook Pro module=monitoring
2024-07-23T12:09:15+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-23T12:09:15+09:00 INF response: SUCCESS module=executor
2024-07-23T12:09:45+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-23T12:09:45+09:00 INF response: SUCCESS module=executor
2024-07-23T12:10:00+09:00 INF Store Logs into Cloud logging for Vatz GCP cloud logging test, Local - meetrick MacBook Pro module=monitoring
2024-07-23T12:10:15+09:00 INF Executor send request to cpu_monitor module=executor
2024-07-23T12:10:15+09:00 INF response: SUCCESS module=executor

https://console.cloud.google.com/logs/query?hl=en&project=validator-test-unlimited

heejin-github commented 3 months ago

@xellos00 Working well with service account key. 👍 I'll keep the current state to monitor it.

and I have some questions.

Which permissions does this service account key have??
Can I ask any plan to rotating service account key?
how long will the logs stored in GCP cloud logging storage?

xellos00 commented 3 months ago

keep the current state to monitor it.

@xellos00 Working well with service account key. 👍 I'll keep the current state to monitor it.

and I have some questions.

Which permissions does this service account key have??

Can I ask any plan to rotating service account key?

how long will the logs stored in GCP cloud logging storage?

Test SA credentials has cloud logging admin only for test, but will be limited to have logger.create only for monitoring real environments
Do not have a plan for now, but further discussion may needs with @rootwarp overall include other credentials as well.
The default retention period is 30 days, but you can also configure the retention period for these logs. Refer to official Doc

@heejin-github Can you approve if you confirmed it works properly?

dsrvlabs / vatz