Performance issue with many GCP projects

[nirmesika]

Describe the bug have 7 gcp orgs with combined volume of 12K (12000) projects. when trying to load the service or refresh the connections it takes over an hour for steampipe to load the schema for those projects.

Steampipe version (steampipe -v) Example: v0.21.7

To reproduce have a gcp.spc file with 12K projects.

Expected behavior loading time of a few minutes.

Additional context running on an AWS EC2 instance with 16GB ram and 4 CPU's, Any ideas on how to resolve

[e-gineer]

Glad you got this resolved @nirmesika ... that's an impressive number of projects!

Steampipe will create schemas for connection (project) the first time you run. Future runs will not need to do that.

Some tips to help performance here:

• Use steampipe service so they are loaded once in a central service.
• Set import_schema to disabled if you don't need them, see https://steampipe.io/docs/reference/config-files/connection#supported-options
• Consider using Datatank in Turbot Pipes for datasets this large - https://turbot.com/pipes/blog/2023/10/datatank-launch

Please hit us up on Slack (or GitHub) if you have any questions or need any help as you scale Steampipe. Keen to learn from your experience.