hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference

lukas-at-harren commented 2 years ago

Describe the bug

When selecting from the gcp_project table, there is an error.

select * from gcp.gcp_project

Steampipe version (steampipe -v): 0.16.1

Plugin version (steampipe plugin list)

+--------------------------------------------------------+---------+-----------------+
| Name                                                   | Version | Connections     |
+--------------------------------------------------------+---------+-----------------+
| hub.steampipe.io/plugins/turbot/gcp@latest             | 0.26.0  | gcp             |
| hub.steampipe.io/plugins/turbot/googlesheets@latest    | 0.2.0   | googlesheets    |
| hub.steampipe.io/plugins/turbot/googleworkspace@latest | 0.4.0   | googleworkspace |
| hub.steampipe.io/plugins/turbot/kubernetes@latest      | 0.10.0  | kubernetes      |
| hub.steampipe.io/plugins/turbot/net@latest             | 0.7.0   | net             |
+--------------------------------------------------------+---------+-----------------+

To reproduce

Run steampipe with roles/viewer permissions on the GCP project.

Expected behavior

A list of projects.

Additional context

The immediate response from SQL:

Query 1 ERROR: ERROR:  rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference

The logs say:

2022-09-01 17:05:59.842 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1662051959333: recovered a panic from a wrapped hydrate function: runtime error: invalid memory address or nil pointer dereference
2022-09-01 17:05:59.842 UTC [ERROR] steampipe-plugin-gcp.plugin: [ERROR] 1662051959333: streamRows error chan select: rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-09-01 17:05:59.842 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1662051959333: Execute call failed err: rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference cancelled: <nil> - cancelling pending item in cache

cbruno10 commented 2 years ago

Hey @lukas-at-harren , sorry you encountered this issue.

Can you please share your ~/.steampipe/config/gcp.spc and also provide more details on how you're authenticating?

Also, do you get this error with multiple different projects, or does listing some projects work? Also, do any other GCP tables return results successfully, or do you get an error with those too?

lukas-at-harren commented 2 years ago

We are using Application Default Credentials (using a GKE Workload Identity that was verified to work, and also in steampipe other queries seem to work fine with that SA).

Note that the environment variable CLOUDSDK_CORE_PROJECT is set to the quota project of the SA.

connection "gcp" {
  plugin    = "gcp"

  # `project` (optional) - The project ID to connect to. This is the project ID (string), not the
  # project name or number. If the `project` argument is not specified for a connection,
  # the project will be determined in the following order:
  #   - The standard gcloud SDK `CLOUDSDK_CORE_PROJECT` environment variable, if set; otherwise
  #   - The `GCP_PROJECT` environment variable, if set (this is deprecated); otherwise
  #   - The current active project project, as returned by the `gcloud config get-value project` command
  #project = "YOUR_PROJECT_NAME"

  # `credentials` (optional) - Either the path to a JSON credential file that contains Google application credentials,
  # or the contents of a service account key file in JSON format. If `credentials` is not specified in a connection,
  # credentials will be loaded from:
  #   - The path specified in the `GOOGLE_APPLICATION_CREDENTIALS` environment variable, if set; otherwise
  #   - The standard location (`~/.config/gcloud/application_default_credentials.json`)
  #credentials = "~/.config/gcloud/application_default_credentials.json"

  # `impersonate_service_account` (optional) - The GCP service account (string) which should be impersonated.
  # If not set, no impersonation is done.
  #impersonate_service_account = "YOUR_SERVICE_ACCOUNT"
}

lukas-at-harren commented 2 years ago

@cbruno10 any info on this? Can we help debug this issue?

cbruno10 commented 2 years ago

Hi @lukas-at-harren , sorry for missing your latest response!

Can you please try the following queries with STEAMPIPE_LOG=TRACE enabled:

select name, project_id, project_number from gcp_project;
select self_link from gcp_project;
select title, tags, akas from gcp_project;
select access_approval_settings from gcp_project;

I've broken up the queries into different sets based on what hydrates/data they rely on, which I believe will help us narrow down which functions specifically are causing the nil pointer exception.

If any of the queries above fail, can you please look for the error in your plugin logs, and around or before the error message, you should see something like listGCPProjects, getProjectAka, getProjectAccessApprovalSettings.

If all of the queries above though work, can you please try running select * from gcp_project again and then look in the plugin logs for any of the logging messages above? Thanks!

jchrisfarris commented 2 years ago

What operating system are you running Steampipe on @lukas-at-harren ? I see similar weirdness in MacOS, but not Linux.

lukas-at-harren commented 2 years ago

@jchrisfarris I have this error both on Kubernetes (Linux, in a container) and on macOS.

lukas-at-harren commented 2 years ago

@cbruno10 thank you for looking into it. Please give me some time, I will attempt to execute these queries and report back here.

ashishsecdev commented 1 year ago

@cbruno10 I am also facing similar issue, I noticed this issue while looking at GCP Dashboard mod and followed your above steps.

You can refer to below logs, currently, I am using Kali Linux on Virtualbox but I faced the same issue on GCP Cloud instance.


2022-10-15 16:35:56.200 UTC [WARN]  Total cache size 0Mb
2022-10-15 16:35:56.263 UTC [WARN]  plugin configured with a nil SecureConfig
2022-10-15 16:52:29.274 UTC [WARN]  PluginManager setPluginCacheSizeMap: 2 connections.
2022-10-15 16:52:29.274 UTC [WARN]  Total cache size 0Mb
2022-10-15 16:52:29.339 UTC [WARN]  plugin configured with a nil SecureConfig
2022-10-15 16:55:40.317 UTC [ERROR] steampipe-plugin-gcp.plugin: [ERROR] 1665852940501: listIamRoles: roleType=ALL
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: recovered a panic from a wrapped hydrate function: runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: doList callHydrateWithRetries (gcp_steampipe-1665853348796) returned err rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: QueryData StreamError rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference (gcp_steampipe-1665853348796)
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: streamRows for gcp_steampipe-1665853348796 - execution has failed (rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference) - calling queryCache.AbortSet
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: executeForConnection gcp_steampipe returned error rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:29.913 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: error channel received rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: recovered a panic from a wrapped hydrate function: runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: doList callHydrateWithRetries (gcp_steampipe-1665853348349) returned err rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: QueryData StreamError rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference (gcp_steampipe-1665853348349)
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: streamRows for gcp_steampipe-1665853348349 - execution has failed (rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference) - calling queryCache.AbortSet
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: executeForConnection gcp_steampipe returned error rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:30.106 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853348349: error channel received rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:48.601 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853368132: waitForPendingItem (gcp_steampipe-1665853368290) - pending item 0xc0147e1bc0, key gcp_steampipegcp_compute_instancecreation_timestamp-1, transferCompleteChan was signalled but getCachedResult returned error: Entry not found
2022-10-15 17:02:48.609 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853368132: waitForPendingItem item still not in the cache - add pending item, (gcp_steampipe-1665853368290) indexBucketKey: index__gcp_steampipegcp_compute_instance, item key gcp_steampipegcp_compute_instancecreation_timestamp-1
2022-10-15 17:02:48.609 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853368132: waitForPendingItem (gcp_steampipe-1665853368132) - pending item 0xc0147e1bc0, key gcp_steampipegcp_compute_instancecreation_timestamp-1, transferCompleteChan was signalled but getCachedResult returned error: Entry not found
2022-10-15 17:02:48.609 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853368132: waitForPendingItem item still not in the cache - add pending item, (gcp_steampipe-1665853368132) indexBucketKey: index__gcp_steampipegcp_compute_instance, item key gcp_steampipegcp_compute_instancecreation_timestamp-1
2022-10-15 17:02:58.298 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853378211: recovered a panic from a wrapped hydrate function: runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:58.298 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853378211: doList callHydrateWithRetries (gcp_steampipe-1665853378580) returned err rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
2022-10-15 17:02:58.298 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853378211: QueryData StreamError rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference (gcp_steampipe-1665853378580)
2022-10-15 17:02:58.298 UTC [WARN]  steampipe-plugin-gcp.plugin: [WARN]  1665853378211: streamRows for gcp_steampipe-1665853378580 - execution has failed (rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference) - calling queryCache.AbortSet

ashishsecdev commented 1 year ago

I noticed something similar forvault as per this - https://github.com/theapsgroup/steampipe-plugin-vault/issues/19

github-actions[bot] commented 1 year ago

'This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 30 days.'

kraft87 commented 1 year ago

Is there any update on this issue? I am having the same issue. I am currently authenticating via "gcloud auth application-default login".

misraved commented 1 year ago

Sorry for the delayed response @kraft87, Could you please execute the following queries STEAMPIPE_LOG=TRACE enabled :

select name, project_id, project_number from gcp_project;
select self_link from gcp_project;
select title, tags, akas from gcp_project;
select access_approval_settings from gcp_project;

craig-fidus commented 1 year ago

@misraved I get the following output:

2023-01-13 18:27:10.610 UTC [TRACE] steampipe: ensureInstallDir /home/kali/.steampipe
2023-01-13 18:27:10.611 UTC [TRACE] steampipe: No memory limit set
2023-01-13 18:27:10.611 UTC [TRACE] steampipe: modfile exists in workspace folder - creating pseudo-resources and loading files recursively
2023-01-13 18:27:10.639 UTC [TRACE] steampipe: parse complete after 1 decode passes
2023-01-13 18:27:10.744 UTC [TRACE] steampipe: parse complete after 2 decode passes
2023-01-13 18:27:10.748 UTC [TRACE] steampipe: telemetry.Init service 'steampipe', tracingEnabled: false, metricsEnabled: false
2023-01-13 18:27:10.748 UTC [TRACE] steampipe: metrics and tracing disabled' - returning
2023-01-13 18:27:10.749 UTC [TRACE] steampipe: GetRunStatus - loadRunningInstanceInfo returned nil
2023-01-13 18:27:10.749 UTC [TRACE] steampipe: StartDB invoker query
2023-01-13 18:27:10.749 UTC [TRACE] steampipe: StartDB started plugin manager
2023-01-13 18:27:10.750 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
2023-01-13 18:27:10.774 UTC [TRACE] steampipe: SERVICE: 2023-01-13 18:27:10.773 UTC [11949] LOG:  redirecting log output to logging collector process
2023-01-13 18:27:10.774 UTC [TRACE] steampipe: SERVICE: 2023-01-13 18:27:10.773 UTC [11949] HINT:  Future log output will appear in directory "/home/kali/.steampipe/logs".
2023-01-13 18:27:10.957 UTC [TRACE] steampipe: Connection string:  host=localhost port=9193 user=root dbname=postgres sslmode=disable
2023-01-13 18:27:11.014 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:postgres host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:root]
2023-01-13 18:27:11.014 UTC [TRACE] steampipe: PSQLInfo >>> [sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key host=localhost port=9193 user=root dbname=postgres sslmode=verify-ca]
2023-01-13 18:27:11.078 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:root]
2023-01-13 18:27:11.078 UTC [TRACE] steampipe: PSQLInfo >>> [sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key port=9193 user=root dbname=steampipe host=localhost sslmode=verify-ca sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt]
2023-01-13 18:27:11.161 UTC [TRACE] steampipe: plugin manager state file not found
2023-01-13 18:27:11.161 UTC [TRACE] steampipe: plugin manager state file not found
2023-01-13 18:27:11.182 UTC [TRACE] steampipe: start: started plugin manager, pid 11969
2023-01-13 18:27:11.182 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:steampipe]
2023-01-13 18:27:11.182 UTC [TRACE] steampipe: PSQLInfo >>> [sslmode=verify-ca sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt host=localhost port=9193 user=steampipe dbname=steampipe sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key]
2023-01-13 18:27:11.190 UTC [TRACE] steampipe: CreatePreparedStatements
2023-01-13 18:27:11.249 UTC [TRACE] steampipe: created local client 0xc0013e74e0
2023-01-13 18:27:11.249 UTC [TRACE] steampipe: getSchemaHashesForDynamicSchemas
2023-01-13 18:27:11.249 UTC [TRACE] steampipe: fetched schema for 0 dynamic plugins
2023-01-13 18:27:11.249 UTC [TRACE] steampipe: RefreshConnections: no updates required
2023-01-13 18:27:11.250 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:root]
2023-01-13 18:27:11.250 UTC [TRACE] steampipe: PSQLInfo >>> [port=9193 user=root dbname=steampipe sslmode=verify-ca sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key host=localhost]
2023-01-13 18:27:11.310 UTC [TRACE] steampipe: setting user search path to [public aws gcp steampipe internal]
2023-01-13 18:27:11.311 UTC [TRACE] steampipe: ensureSessionSearchPath
2023-01-13 18:27:11.311 UTC [TRACE] steampipe: updated the required search path to "public","aws","gcp","steampipe","internal"
2023-01-13 18:27:11.311 UTC [TRACE] steampipe: session search path will be updated to  "public","aws","gcp","steampipe","internal"
2023-01-13 18:27:11.312 UTC [TRACE] steampipe: user search path sql: alter user "steampipe" set search_path to "public","aws","gcp","steampipe","internal";
2023-01-13 18:27:11.313 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:root]
2023-01-13 18:27:11.313 UTC [TRACE] steampipe: PSQLInfo >>> [user=root dbname=steampipe host=localhost sslmode=verify-ca sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key port=9193]
2023-01-13 18:27:11.372 UTC [TRACE] steampipe: ensureSessionSearchPath
2023-01-13 18:27:11.372 UTC [TRACE] steampipe: updated the required search path to "public","aws","gcp","steampipe","internal"
2023-01-13 18:27:11.372 UTC [TRACE] steampipe: session search path is already correct - nothing to do
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: ensureSessionSearchPath
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: updated the required search path to "public","aws","gcp","steampipe","internal"
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: session search path is already correct - nothing to do
Warning: executeQueries: query 1 of 1 failed: syntax error at or near "-"
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: close local client 0xc0013e74e0
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: local client not NIL
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: DbClient.Close &{1 0 0 0xc0009ab300 0xc0007a3aa0 <nil> 0x1199c80 <nil> <nil> 1 1 600000000000 0 60000000000 60000000000 0xc000b8f200 {0 {0 0}} 0xc000b8f260}
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: local client close complete
2023-01-13 18:27:11.373 UTC [TRACE] steampipe: shutdown local service query
2023-01-13 18:27:11.374 UTC [TRACE] steampipe: SQLInfoMap >>> map[dbname:steampipe host:localhost port:9193 sslcert:/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey:/home/kali/.steampipe/db/14.2.0/data/server.key sslmode:verify-ca sslrootcert:/home/kali/.steampipe/db/14.2.0/data/root.crt user:root]
2023-01-13 18:27:11.374 UTC [TRACE] steampipe: PSQLInfo >>> [user=root dbname=steampipe sslrootcert=/home/kali/.steampipe/db/14.2.0/data/root.crt sslcert=/home/kali/.steampipe/db/14.2.0/data/server.crt sslkey=/home/kali/.steampipe/db/14.2.0/data/server.key sslmode=verify-ca host=localhost port=9193]
2023-01-13 18:27:11.434 UTC [TRACE] steampipe: StopDB invoker query, force false
2023-01-13 18:27:11.434 UTC [TRACE] steampipe: plugin manager stop
2023-01-13 18:27:11.435 UTC [TRACE] steampipe: pluginManager.Shutdown
2023-01-13 18:27:11.435 UTC [TRACE] steampipe: PluginManagerClient Shutdown
2023-01-13 18:27:11.435 UTC [TRACE] steampipe: pluginManager state.kill

craig-fidus commented 1 year ago

@misraved Hey it seems I only get this issue when using the --tag cis_level=1 --tag cis=true flags

e-gineer commented 1 year ago

Can you please share the full CLI command you are running @craig-fidus?

This line seems to indicate that it's trying to run a query called -:

Warning: executeQueries: query 1 of 1 failed: syntax error at or near "-"

craig-fidus commented 1 year ago

@e-gineer

This is the one I ran └─$ steampipe query "select name, project_id, project_number from XXXXX-XXXXX-XXXXX;"

craig-fidus commented 1 year ago

@e-gineer

If I run the command below I get the error

└─$ steampipe check benchmark.cis_v130  --tag cis_level=1 --tag cis=true                                                                                                       
Error: failed to create introspection tables: ERROR: column "sql" of relation "steampipe_control" does not exist (SQLSTATE 42703)

If I run without it works, but I only want Level 1

└─$ steampipe check benchmark.cis_v130 

CIS v1.3.0 ............................................................................................................................................. 108 / 139 [==========]
| 
+ 1 Identity and Access Management .....................................................................................................................  17 /  23 [===       ]
| | 
| + 1.1 Ensure that corporate login credentials are used ...............................................................................................   0 /   0 [          ]

Just noticed now I get this error also: | | ERROR: rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference (SQLSTATE HV000)

NightHawk4 commented 1 year ago

Hi, Is there any update on this. Is this now a known bug or just a few people with this issue. I am now having the same problem. :-(

pmalkki commented 1 year ago

Might be obvious but wanted to point this out still: If you don't have proper APIs enabled in your project you seem to get similar errors. Example logs if API cloudresourcemanager.googleapis.com is disabled:

[WARN]  steampipe-plugin-gcp.plugin: [WARN]  1674825580267: recovered a panic from a wrapped hydrate function: runtime error: invalid memory address or nil pointer dereference
[TRACE] steampipe-plugin-gcp.plugin: [TRACE] 1674825580267: hydrateWithIgnoreError returned error rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
[TRACE] steampipe-plugin-gcp.plugin: [TRACE] 1674825580267: wrapped hydrate call listGcpAuditPolicies returned error googleapi: Error 403: Cloud Resource Manager API has not been used in project xxxxxxxxxxxx before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=xxxxxxxxxxxx then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry.
[DEBUG] steampipe-plugin-gcp.plugin: Details:
[DEBUG] steampipe-plugin-gcp.plugin: [
[DEBUG] steampipe-plugin-gcp.plugin:   {
[DEBUG] steampipe-plugin-gcp.plugin:     "@type": "type.googleapis.com/google.rpc.Help",
[DEBUG] steampipe-plugin-gcp.plugin:     "links": [
[DEBUG] steampipe-plugin-gcp.plugin:       {
[DEBUG] steampipe-plugin-gcp.plugin:         "description": "Google developers console API activation",
[DEBUG] steampipe-plugin-gcp.plugin:         "url": "https://console.developers.google.com/apis/api/cloudresourcemanager.googleapis.com/overview?project=xxxxxxxxxxxx"
[DEBUG] steampipe-plugin-gcp.plugin:       }
[DEBUG] steampipe-plugin-gcp.plugin:     ]
[DEBUG] steampipe-plugin-gcp.plugin:   },
[DEBUG] steampipe-plugin-gcp.plugin:   {
[DEBUG] steampipe-plugin-gcp.plugin:     "@type": "type.googleapis.com/google.rpc.ErrorInfo",
[DEBUG] steampipe-plugin-gcp.plugin:     "domain": "googleapis.com",
[DEBUG] steampipe-plugin-gcp.plugin:     "metadata": {
[DEBUG] steampipe-plugin-gcp.plugin:       "consumer": "projects/xxxxxxxxxxxx",
[DEBUG] steampipe-plugin-gcp.plugin:       "service": "cloudresourcemanager.googleapis.com"
[DEBUG] steampipe-plugin-gcp.plugin:     },
[DEBUG] steampipe-plugin-gcp.plugin:     "reason": "SERVICE_DISABLED"
[DEBUG] steampipe-plugin-gcp.plugin:   }
[DEBUG] steampipe-plugin-gcp.plugin: ]
[DEBUG] steampipe-plugin-gcp.plugin: , accessNotConfigured, ignore config ShouldIgnoreError: func1
[TRACE] steampipe-plugin-gcp.plugin: [TRACE] 1674825580267: shouldRetryError err: rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference, retryConfig:
[WARN]  steampipe-plugin-gcp.plugin: [WARN]  1674825580267: doList callHydrateWithRetries (gcp-1674825578936) returned err rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference
[WARN]  steampipe-plugin-gcp.plugin: [WARN]  1674825580267: QueryData StreamError rpc error: code = Internal desc = hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference (gcp-1

misraved commented 1 year ago

@NightHawk4 @craig-fidus apologies for the delayed response.

I haven't managed to successfully replicate the issue and hence the delay 😞.

Taking a fresh stab at it, @NightHawk4, you mentioned that you have been getting this error of late, does that mean it was working correctly for you before and now it has stopped working?

If that is the case, could you please run the following gcloud command and confirm if it works correctly: gcloud projects list?

Thank you everyone for being patient with us and helping us by providing useful logs, we will try to figure out the root cause of this error as soon as possible 👍 .

NightHawk4 commented 1 year ago

Hi @misraved I'm currently using version 0.17.4. I have checked running services with 'gcloud services list' in my project. The cloudresourcemanager.googleapis.com is running in that project.

I can run gcloud projects list and get the appropriate responses.

I'll upgrade to the latest version which I think is v.0.18.2 using 'brew upgrade turbot/tap/steampipe' and see how that goes, but for now it's still not working.

UPDATE: New version 0.18.2. Now it seems I have to enable another API... !? See below.

For more information, type .help

select * from gcp_project

Error: googleapi: Error 403: Access Approval API has not been used in project 123456789 before or it is disabled. Enable it by visiting https://console.developers.google.com/apis/api/accessapproval.googleapis.com/overview?project=123456789 then retry. If you enabled this API recently, wait a few minutes for the action to propagate to our systems and retry. Details: [ { "@type": "type.googleapis.com/google.rpc.Help", "links": [ { "description": "Google developers console API activation", "url": "https://console.developers.google.com/apis/api/accessapproval.googleapis.com/overview?project=123456789" } ] }, { "@type": "type.googleapis.com/google.rpc.ErrorInfo", "domain": "googleapis.com", "metadata": { "consumer": "projects/123456789", "service": "accessapproval.googleapis.com" }, "reason": "SERVICE_DISABLED" } ] , accessNotConfigured (SQLSTATE HV000)

*If I enable this API above, I get an error on the following 'select from gcp_project shown below :**

Welcome to Steampipe v0.18.2 For more information, type .help

select * from gcp_project

Error: googleapi: Error 400: Precondition check failed., failedPrecondition (SQLSTATE HV000)

BUT ... If I run against another table I get the following correct output :

select * from gcp_project_organization_policy

+--------------------------------+----------------------+---------+------------------+-------------+----------------+-----------------+--> | id | update_time | version | etag | list_policy | boolean_policy | restore_default | t> +--------------------------------+----------------------+---------+------------------+-------------+----------------+-----------------+--> | storage.publicAccessPrevention | 2023-01-23T13:11:44Z | 0 | CJCPup4GENjzt9AC | | {} | | s> +--------------------------------+----------------------+---------+------------------+-------------+----------------+-----------------+-->

Now I'm confused, my authentication must be okay, and I have the latest version with a new required API enabled..

mdaguete commented 1 year ago

Hi,

As additional info, I'm getting the same error, but only when the default project isn't set, when setted I only get a empty result. Perhaps related to line 116 of file table_gcp_project.go

resp, err := service.Projects.List().Filter("id=" + project).Do()

I'm using an organization level identity to list multiple projects.

Regards.

cbruno10 commented 1 year ago

Hi everyone, thanks for your patience on this longstanding issue and also thanks for all of the helpful debug messages and testing!

Based off of all of your feedback and some testing I did, I just pushed a new version, v0.33.1, which includes the following fix:

Fixed the plugin failing to detect when an active project is not set in the connection config, environment variables, or the gcloud CLI.

This bug originated around how we checked for an active project for a given connection. Today, we check for an active project from the following locations (in order):

The project connection config arg
The CLOUDSDK_CORE_PROJECT env variable
The GCP_PROJECT env variable
The results from running gcloud config get-value project

An active project must be set, otherwise the plugin doesn't know where to make the API requests.

While checking the results from running the gcloud command, we were handling the results incorrectly and passed up an empty object, resulting in the panic runtimes. This has now been fixed, and if an active project has not been set for a connection, Steampipe will report a better error:

> select * from gcp_project

Error: an active project must be set (SQLSTATE HV000)

This fix will affect any tables for resources that are created in projects, e.g., gcp_compute_instance, gcp_storage_bucket, but tables like gcp_organization can still return results (if permissions have been set correctly) even if an active project hasn't been set anywhere.

After updating to v0.33.1 and setting an active project, if anyone is still getting unexpected query results or errors, please feel free to add them in this issue (will leave open for a bit for more visibility)!

Also, if anyone is still receiving API disabled errors, please create a new issue so we can focus on that separately, as I believe those are not related to the original issue reported in this issue.

NightHawk4 commented 1 year ago

@cbruno10 Great News, nice work. I'll try when I get 5 mins.. :-).

cbruno10 commented 1 year ago

I'm closing this issue for now, as I think it's been resolved by having the plugin provide a more clear error/what to do next message, but will re-open if this is not the case based on user feedback.

turbot / steampipe-plugin-gcp

hydrate function listGCPProjects failed with panic runtime error: invalid memory address or nil pointer dereference #385