Closed nirmesika closed 6 months ago
Glad you got this resolved @nirmesika ... that's an impressive number of projects!
Steampipe will create schemas for connection (project) the first time you run. Future runs will not need to do that.
Some tips to help performance here:
steampipe service
so they are loaded once in a central service.import_schema
to disabled
if you don't need them, see https://steampipe.io/docs/reference/config-files/connection#supported-optionsPlease hit us up on Slack (or GitHub) if you have any questions or need any help as you scale Steampipe. Keen to learn from your experience.
Describe the bug have 7 gcp orgs with combined volume of 12K (12000) projects. when trying to load the service or refresh the connections it takes over an hour for steampipe to load the schema for those projects.
Steampipe version (
steampipe -v
) Example: v0.21.7To reproduce have a gcp.spc file with 12K projects.
Expected behavior loading time of a few minutes.
Additional context running on an AWS EC2 instance with 16GB ram and 4 CPU's, Any ideas on how to resolve