search

GoogleCloudPlatform / gcpdiag

gcpdiag is a command-line diagnostics tool for GCP customers.
https://gcpdiag.dev/
Apache License 2.0
288 stars 67 forks source link

Intermittent warning/skips on Composer rules #72

Open qgallet opened 1 year ago

qgallet commented 1 year ago

This was working fine on 0.59, now there seems to be some lock issue on Composer API queries. Here's the command used:

gcpdiag lint --project=my-project --include=composer

At the start of the command, the following log will be temporarily visible on the console, before the results are printed.

... still fetching logs (project: my-project, resource type: cloud_composer_environment, max wait: 117s)

After two minutes, some of the checks are skipped because of (I guess) a timeout:

🔎  composer/BP/2023_001: Cloud Composer logging level is set to INFO
   - my-project/europe-west1/my-composer               [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/BP/2023_003 
🔎  composer/ERR/2022_001: Composer Service Agent permissions
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/ERR/2023_001 
🔎  composer/WARN/2022_001: Composer Service Agent permissions for Composer 2.x
   - my-project                                        [ OK ]

[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2022_003 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_001 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_002 
[WARNING] RuntimeError: Couldn't acquire lock for get_environments. while processing rule: composer/WARN/2023_003 
🔎  composer/WARN/2023_004: Cloud Composer database CPU usage does not exceed 80%
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_005: Cloud Composer is consistently in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_006: Airflow schedulers are healthy for the last hour
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_007: Cloud Composer Scheduler CPU limit exceeded.
   - my-project/europe-west1/my-composer               [ OK ]

🔎  composer/WARN/2023_008: Cloud Composer Airflow database is in healthy state
   - my-project/europe-west1/my-composer               [ OK ]

Rules summary: 9 skipped, 8 ok, 0 failed

If I launch the command with a more restrictive parameter, like --include=composer/WARN, this still happens although less frequently.

ropeck commented 1 year ago

Thanks for the issue report. Can you check if this still occurs with v0.63 recently released?