Unable to test a new provider in debug mode

[andrewthetechie]

Describe the Bug

While trying to develop my own provider, I am having issues in attempting to test it. Cloudquery always does a lookup against the hub and when it fails to find my provider, it fails. My provider is not listed in the hub.

Expected Behavior

In debug mode, cloudquery uses the local version of my provider via CQ_REATTACH_PROVIERS without talking to the hub.

CloudQuery Version

Version: 0.23.4 Commit: 96ec2c96eb6fd8dc67e8f7f666618a8bcfbdd262 Date: 2022-05-17T14:33:18Z

Debug Output

{"@level":"debug","@message":"plugin address","@module":"Datadog","@timestamp":"2022-05-22T00:25:00.884348-05:00","address":"/var/folders/dl/4jg1h4h53dv_pzzpbt6sntyr0000gn/T/plugin3220677151","network":"unix"}
Provider started, to attach Cloudquery set the CQ_REATTACH_PROVIDERS env var:

        export CQ_REATTACH_PROVIDERS=/Users/andrew/Documents/code/cq-provider-datadog/.cq_reattach

In a second terminal

CQ_PROVIDER_DEBUG=1 CQ_REATTACH_PROVIDERS=/Users/andrew/Documents/code/cq-provider-datadog/.cq_reattach ./cloudquery fetch --no-verify
Anonymous telemetry collection and crash reporting enabled. Run with --no-telemetry to disable, or check docs at https://docs.cloudquery.io/docs/cli/telemetry
Syncing CloudQuery providers [datadog@v0.0.1]

Initializing CloudQuery Providers...

⚠️ cq-provider-datadog@v0.0.1 skipped verification...    0s  100 %

Finished provider initialization...

Checking available provider updates...

Diagnostics:

Type: Internal Severity: Error
        Summary: failed to find provider[datadog] latest version

Sync Diagnostics:

Type: Internal Severity: Error
        Summary: failed to find provider[datadog] latest version

Config.hcl

cloudquery {
  plugin_directory = "./cq/providers"
  policy_directory = "./cq/policies"

  provider "datadog" {
    version = "latest"
  }

  connection {
    username = "postgres"
    password = "pass"
    host     = "localhost"
    port     = 5432
    database = "postgres"
    sslmode  = "disable"
  }
}

// All Provider Configurations

provider "datadog" {
  configuration {
    accounts "main" {
      api_key = ""
      app_key = ""
    }
  }

}

Steps to Reproduce

1. Develop a new provider not yet on hub.cloudquery.com
2. build and run it in debug mode
3. Attempt to fetch using that provider and CQ_REATTASCH_PROVIDER

Additional Context

Fixing this might issue might be closely related to https://github.com/cloudquery/cloudquery/issues/738

[erezrokah]

Thanks for opening the issue @andrewthetechie ❤️ I've created a fix for this in https://github.com/cloudquery/cloudquery/pull/748

[andrewthetechie]

@erezrokah ty for the quick fix!

I built cloudquery from git with your fixes this morning, but I am still having the same issue.

CQ_REATTACH_PROVIDERS=/Users/andrew/Documents/code/cq-provider-datadog/.cq_reattach ./cloudquery fetch --no-verify
Anonymous telemetry collection and crash reporting enabled. Run with --no-telemetry to disable, or check docs at https://docs.cloudquery.io/docs/cli/telemetry
Syncing CloudQuery providers [datadog@latest]

Initializing CloudQuery Providers...

❌ failed to initialize provider: failed to download providers: failed to find provider[datadog] latest version.

Diagnostics:

Type: Internal Severity: Error
        Summary: failed to download providers: failed to find provider[datadog] latest version

Sync Diagnostics:

Type: Internal Severity: Error
        Summary: failed to download providers: failed to find provider[datadog] latest version```

./cloudquery version
Version: development
Commit: development
Date: unknown```

[erezrokah]

Thanks for testing! I'll re-open the issue while I'm taking a look at this

[erezrokah]

@andrewthetechie should handle this https://github.com/andrewthetechie/cq-provider-datadog/pull/1, and we'll look into a way of avoiding this pitfall in the future

[andrewthetechie]

Ty for the help @erezrokah - I had made that change here https://github.com/andrewthetechie/cq-provider-datadog/blob/initial-dev/resources/provider/provider.go#L22 but I see the issue - Datadog vs datadog.

[andrewthetechie]

Success! I changed 'Datadog' to 'datadog' and fetching went right through with your fixes.