Provider vs. Instana-Server v195: "Error: value not allowed for unary operator expression"

[ppuschmann]

The commands terraform plan and terraform apply crash against A Instana-Server (on-premise) with v195.

With Instana-Server v193 everything is still fine.

Instana-Server v197 is untested until now.

Terraform v0.15 terraform-provider-instana: v1.0.0

Output of TF_LOG=DEBUG terraform plan:

2021-04-26T16:13:04.460+0200 [INFO]  provider.terraform-provider-instana_v1.0.0: 2021/04/26 16:13:04 [WARN] Truncating attribute path of 0 diagnostics for TypeSet: timestamp=2021-04-26T16:13:04.460+0200
2021-04-26T16:13:04.460+0200 [WARN]  Provider "registry.terraform.io/gessnerfl/instana" produced an invalid plan for instana_alerting_config.alerting_eCom_general, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .integration_ids: planned value cty.UnknownVal(cty.Set(cty.String)) does not match config value cty.SetVal([]cty.Value{cty.UnknownVal(cty.String)})
2021-04-26T16:13:04.461+0200 [WARN]  Provider "registry.terraform.io/gessnerfl/instana" produced an invalid plan for instana_alerting_config.alerting_eCom_OOM, but we are tolerating it because it is using the legacy plugin SDK.
    The following problems may be the cause of any confusing errors from downstream operations:
      - .integration_ids: planned value cty.UnknownVal(cty.Set(cty.String)) does not match config value cty.SetVal([]cty.Value{cty.UnknownVal(cty.String)})
2021-04-26T16:13:04.461+0200 [INFO]  backend/local: plan operation completed
╷
│ Error: value not allowed for unary operator expression
│
│
╵
2021-04-26T16:13:04.608+0200 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = transport is closing"
2021-04-26T16:13:04.612+0200 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/gessnerfl/instana/1.0.0/darwin_amd64/terraform-provider-instana_v1.0.0 pid=50697
2021-04-26T16:13:04.612+0200 [DEBUG] provider: plugin exited

Need more Information? I'm happy to share.

[ppuschmann]

Update: Instana-Server v197 is fine. At least the resources I use are working.

[gessnerfl]

Hi @ppuschmann

Can you please share some more details (for sure without disclosing any confidential data):

• Is this happening after upgrading to version 1.0.0 or is there no migration involved?
• If this is a migration which version of the terrraform-provider-instana did you used before
• Can you provide the code of the affected resources?

It seems the issue is caused by an invalid application configuration or a conflict between the representation of the application configuration in terraform and the model required by the Instana REST API (internal mapping error). In any case the error handling should be improved

[ppuschmann]

Is this happening after upgrading to version 1.0.0 or is there no migration involved?

Yes, this happened after migration to 1.0.0

If this is a migration which version of the terrraform-provider-instana did you used before

I used v0.10.1 before.

Can you provide the code of the affected resources?

I think I can't pin-point it down to some resources. What should I look out for?

[gessnerfl]

I would be especially interested in instana_application_config resources. The error message indicates that the issue is in this area and I think that either I introduced some change which is causing this issue or Instana has changed something in the API. I faced a similar issue in the past where the API was changed and I had to adopt the parsing to fit to the changed API semantics.

[ppuschmann]

my_teams = {
  "team_awesome" = {
    name    = "Team Awesome",
    team    = "awesome",
  },
  "team_super" = {
    name    = "My Super-Team",
    team    = "Super",
  }
}

resource "instana_application_config" "applications" {
  for_each            = var.my_teams
  label               = each.value.name
  scope               = "INCLUDE_IMMEDIATE_DOWNSTREAM_DATABASE_AND_MESSAGING"
  boundary_scope      = "INBOUND"
  match_specification = "docker.label.company_name.team EQUALS '${each.value.team}'"
}

# Application perspective for all eCom-services
resource "instana_application_config" "application_all_ecom_services" {
  label               = "eCOM all services"
  scope               = "INCLUDE_IMMEDIATE_DOWNSTREAM_DATABASE_AND_MESSAGING"
  boundary_scope      = "INBOUND"
  match_specification = "docker.label.company_name.service NOT_EMPTY"
}

# Application perspective for App1
resource "instana_application_config" "application_app1" {
  label               = "App1"
  scope               = "INCLUDE_IMMEDIATE_DOWNSTREAM_DATABASE_AND_MESSAGING"
  boundary_scope      = "ALL"
  match_specification = "agent.tag.app1-public NOT_EMPTY OR agent.tag.app1-backen NOT_EMPTY"
}

# Application perspective for App2
resource "instana_application_config" "application_app2" {
  label               = "App2"
  scope               = "INCLUDE_IMMEDIATE_DOWNSTREAM_DATABASE_AND_MESSAGING"
  boundary_scope      = "ALL"
  match_specification = "agent.tag.app2-public NOT_EMPTY OR agent.tag.app2-backend NOT_EMPTY"
}

Like this.

2021-04-27T10:38:57.147+0200 [WARN]  Provider "registry.terraform.io/gessnerfl/instana" produced an unexpected new value for instana_application_config.applications["team_awesome"] during refresh.
      - .label: was cty.StringVal("Team Awesome"), but now cty.StringVal(" Team Awesome")
      - .match_specification: was cty.StringVal("docker.label.rewe.team EQUALS 'Awesome'"), but now cty.StringVal("docker.label.rewe.team@dest EQUALS 'Awesome'")

^ Maybe this is related?

[gessnerfl]

I tried to reproduce this without success. I tried the following steps:

1. Created a terraform project with the provided resources using instana provider version 0.10.1
2. terraform init && terraform apply
3. backup terraform state
4. change instana provider version to 1.0.0
5. terraform init --upgrade
6. terraform plan
7. terraform apply

All operations completed as expected.

@ppuschmann : Are the steps correct as described above? Do you use the default prefix/suffix feature in the provider configuration? This behaviour only occurs with on-prem version 195 but not with the other on-prem versions?

I only have access to the SaaS solution. Maybe this is the reason why I cannot reproduce it.

For the support of the import feature (#95) I had to fix several read operations and the handling of names/labels. Additionally, for application perspectives I had to adopt how match_specification is stored in the terraform state. These changes were carefully tested. Still I thought there might be a general issue with these changes but then I expect to be able to reproduce it also in the SaaS version and that it occurs also for all on-prem versions.

[ppuschmann]

@gessnerfl The steps are correct. There's only a terraform init -reconfigure to be inserted between 4. and 5.

It could also happen that this one statefile is somehow borken, since my other instances are working fine. Maybe I try to find out if I could upgrade the last v193 instance to v195 and try to reproduce the issue (if it was version-related). If this wouldn't help, I'd try to upgrade all instances to v197 and have another try.

With some luck this solves the issue. If not, maybe I've to do some deeper investigation on the statefile (and maybe create the configuration from scratch).

[gessnerfl]

@ppuschmann To be honest I was not aware of the -reconfigure option. However after reading the documentation I thought that this could cause the issue . The reason for this is that as the command prevents migration of any existing state the updated resources schema does not match the existing state representations.

However when I tried to reproduce this I get an error when running terraform init -reconfigure.

Error: Failed to query available provider packages Could not retrieve the list of available versions for provider gessnerfl/instana: locked provider registry.terraform.io/gessnerfl/instana 0.10.1 does not match configured version constraint 1.0.0; must use terraform init -upgrade to allow selection of new versions

Could it be that this command is executed between step 3 and 4 instead of 4 and 5?

[ppuschmann]

Yes, you're right. It has to be between 3. and 4.

I'll continue fiddling. With some luck I'll find something.

[ppuschmann]

I finally had to remove two items of "instana_application_config" from the statefile via terraform state rm ....

Previous inspection of the state via terraform state pull showed no obvious issues. There was no other change on the Instana-Server.

I'm fine with closing this issue.

[gessnerfl]

OK thanks for the update. Please let me know in case you face similar issues

[ppuschmann]

@gessnerfl I'm now facing these issues again, when trying to apply the configuration of a instana_application_config resource.

Now I'm using the latest provider v1.1.1 with terraform v1.0.8.

I again tried to remove the problematic resource from the statefile and tried to import the resource again.

This fails with:

terraform import instana_application_config.application_myapp bpac9c3ipt326rkqas00
instana_application_config.application_myapp: Importing from ID "bpac9c3ipt326rkqas00"...
instana_application_config.application_myapp: Import prepared!
  Prepared instana_application_config for import
instana_application_config.application_myapp: Refreshing state... [id=bpac9c3ipt326rkqas00]
╷
│ Error: no value must be assigned for unary operation

resource:

resource "instana_application_config" "application_myapp" {
  label          = "MYAPP"
  scope          = "INCLUDE_IMMEDIATE_DOWNSTREAM_DATABASE_AND_MESSAGING"
  boundary_scope = "INBOUND"
  tag_filter     = "agent.tag.myapp-public NOT_EMPTY OR agent.tag.myapp-author NOT_EMPTY"
}

I'm a bit hesitant to remove the application_config completely in Instana, because there are some applications and teams depending on this.

What would you need to further debug this?

[gessnerfl]

@ppuschmann would it be possible to share the json response from the API of you Instana Instance?

GET <instana_host>/api/application-monitoring/settings/application/<application_id>
Authorization: apiToken <instana_api_token>

I'm interested in the tagFilterExpression element only.

[gessnerfl]

@ppuschmann I think I already found the root cause of the issue. See #111

[ppuschmann]

@gessnerfl here you are:

"tagFilterExpression": {
    "type": "EXPRESSION",
    "logicalOperator": "OR",
    "elements": [
        {
            "type": "TAG_FILTER",
            "name": "agent.tag",
            "stringValue": "myapp-public=",
            "numberValue": null,
            "booleanValue": null,
            "key": "myapp-public",
            "value": null,
            "operator": "NOT_EMPTY",
            "entity": "DESTINATION"
        },
        {
            "type": "TAG_FILTER",
            "name": "agent.tag",
            "stringValue": "myapp-author=",
            "numberValue": null,
            "booleanValue": null,
            "key": "myapp-author",
            "value": null,
            "operator": "NOT_EMPTY",
            "entity": "DESTINATION"
        }
    ]
}

[gessnerfl]

Thanks this fits to the fix implemented in #111. Will try to provide the release ASAP.

[gessnerfl]

@ppuschmann the fix is now released. Can you please retry with version 1.1.2

[ppuschmann]

Sorry, there's no success. Still the same error messages.

Import doesn't work (on the setup where I removed the entries from the statefile) and "usual" configuration (on a different server) did also not work.

[gessnerfl]

@ppuschmann Missed one more thing to address there. Will provide another patch later today.

[gessnerfl]

@ppuschmann version 1.1.3 is now available which should help you to delete the application configs. However it turned our that there is a conceptual issue with the tag filters which I need to fix in addition. Currently unary expressions are not properly supported with tag filters. I therefore need to adopt the parser.

[ppuschmann]

@gessnerfl v1.1.3 is working! The regular "apply" and also "import" are running fine again.

Thank you very much!

[gessnerfl]

@ppuschmann please note that the root cause is not solved and I will try to release today or tomorrow a breaking change how tags are used in the new tag_filters. The current implementation is based on a wrong assumption.

See also #114

[gessnerfl]

@ppuschmann release 1.1.4 is now available fixing #114. Please note the change in the syntax when using tags. The tag key needs to be separated from the entity path using colon :. Example: agent.tag:mytag NOT_EMPTY