Cannot retrieve publishers as data due to breaking API changes

[matthosking-mydeal]

The following error always occurs after API specifications recently changed:

│ Error: publishers.0.upgrade_failed_reason: '' expected type 'string', got unconvertible type 'map[string]interface {}'
│ 
│   with data.netskope_publishers.all,
│   on [netskope.tf](http://netskope.tf/) line 1, in data "netskope_publishers" "all":
│    1: data "netskope_publishers" "all" {
│ 

upgrade_failed_reason now looks like this in the API response:

        "upgrade_failed_reason": {
          "detail": "",
          "error_code": 0,
          "timestamp": 1677434996,
          "version": "7619"
        },

The data type for this field must be updated to match the API change.

[ns-sbrown]

Thanks for reaching out on this issue. I will look into this and hopefully be able to release a fix shortly.

[jeff-clearcover]

@ns-sbrown Any updates?

[nwhobart]

We've seen this issue crop up over the weekend. Oddly enough before and on Friday there were no issues but today we're seeing the same messaging as @matthosking-mydeal

[ns-sbrown]

I am looking into it this week.

[ns-sbrown]

@nwhobart @jeff-clearcover @matthosking-mydeal Can any of you share the TF that is causing this error as I am not able to reproduce.

[jeff-clearcover]

@ns-sbrown

The data structure in the netskope API client go doesn't match what the API actually returns, I updated it here: https://github.com/jeff-clearcover/netskope-api-client-go/blame/fix/publishers-data-source/nsgo/publishers.go#L18-L50

[jeff-clearcover]

@ns-sbrown The API now returns this for upgrade failed reason and when we try to refresh the data source it fails because it cannot marshal the map into the string. It looks like the schema will need to change to a TypeMap and that's what I started working on in my branch for the terraform-provider-netskope https://github.com/jeff-clearcover/terraform-provider-netskope/blob/fix/publishers-data-source/netskope/data_source_publishers.go#L122-L239

        "upgrade_failed_reason": {
          "detail": "exit status 100 (E: Sub-process /usr/bin/dpkg returned an error code (1))",
          "error_code": 516,
          "timestamp": 1687626688,
          "version": "8080"
        },

[jeff-clearcover]

@ns-sbrown It seems like it is the PublishersWithFilters function now returning a different data structure than intended. After being marshaled between structures the publisher object has things that no longer match the schema in the data source. In order to reproduce you'll need to have a publisher which failed to upgrade in the response for your data source lookup.

data "netskope_publishers" "default" {
  filter = "publisher_name sw CC-NPA and status eq connected"
}

[TimGebhardt]

@jeff-clearcover submitted a PR with the fix here: https://github.com/netskopeoss/netskope-api-client-go/pull/4

[ns-sbrown]

@jeff-clearcover I merged the API client updates just now and created v0.3.3 tag based on it. I will work on testing the provider using the updated API Client and hopefully I'll be able to publish it ASAP.

[jeff-clearcover]

Thanks so much @ns-sbrown, this provider is very valuable and useful for our company so I'm happy to help further if needed.

[ns-sbrown]

@jeff-clearcover FYI...I was hopeful that I would get this published this week, but ran into a limitation in SDKv2. I have a workaround but it may be cleaner to upgrade to the new plugin-sdk. Either way it looks like this may take a little bit longer to get published.

[nwhobart]

@ns-sbrown with the latest provider version update the error still exists however it's a little more verbose in the messaging:

│ Error: publishers.0.upgrade_failed_reason: '' expected type 'string', got unconvertible type 'map[string]interface {}', value: 'map[detail: error_code:0 timestamp:1.687626589e+09 version:8080]'

[ns-sbrown]

All...I just published v0.2.4 which as a workaround forces nested elements to strings instead of bool, int etc. This is because of how terraform SDKv2 handles nested maps. I investigated upgrading to the new plugin framework but this is a lot larger effort. Please let me know if this solves the current issue.

@nwhobart @jeff-clearcover

[nwhobart]

@ns-sbrown We ended up using https://github.com/Mastercard/terraform-provider-restapi to make direct calls to the Netskope v2 API to circumvent the issue.

[TimGebhardt]

@ns-sbrown we're happy to revisit this provider when it and the underlying Netskope API seem more production ready. But for now this has highlighted it's in a pre-alpha state and not really on the Netskope's radar. That makes us pause on basing our own infrastructure on top of it.

[ns-sbrown]

Sounds good guys. I am going to close this issue. If you come back to it and find other issues, please let me know.