Data Source azurerm_storage_table_entities does not return all properties from the table

imileusnic commented 9 months ago

Is there an existing issue for this?

[X] I have searched the existing issues

Community Note

Please vote on this issue by adding a :thumbsup: reaction to the original issue to help the community and maintainers prioritize this request
Please do not leave "+1" or "me too" comments, they generate extra noise for issue followers and do not help prioritize the request
If you are interested in working on this issue or have submitted a pull request, please leave a comment and review the contribution guide to help.

Terraform Version

1.6.6

AzureRM Provider Version

3.94.0

Affected Resource(s)/Data Source(s)

azurerm_storage_table_entities

Terraform Configuration Files

Storage Account (test9873storage)
===============
- Tables
  - rdepartments
    - partition_key: depkey, row-key: row1, name: dep01
    - partition_key: depkey, row-key: row2, name: dep02
  - projects
    - partition_key: projkey, row-key: row1, department: dep01, project: proj01
    - partition_key: projkey, row-key: row2, department: dep02, project: proj02

Config File
============
terraform {
  required_providers {
    azurerm = {
      source  = "hashicorp/azurerm"
      version = "3.94.0"
    }
  }
}

provider "azurerm" {
  skip_provider_registration = true
  features {}
}

data "azurerm_storage_table_entities" "departments" {
  table_name           = "rdepartments"
  storage_account_name = "teststorage"
  filter               = "PartitionKey eq 'depkey'"
}

data "azurerm_storage_table_entities" "projects" {
  table_name           = "rprojects"
  storage_account_name = "teststorage"
  filter               = "PartitionKey eq 'projkey'"
}

output "department_entities" {
  value = data.azurerm_storage_table_entities.departments
}

output "project_entities" {
  value = data.azurerm_storage_table_entities.projects
}

Debug Output/Panic Output

# removed the previous section of Debug file related to registering resources/data sources and retrieving a list of storage accounts (way too many in my subscription).

2024-03-01T09:02:50.901+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: Cache Miss - looking up the account key for storage account "teststorage"..: timestamp="2024-03-01T09:02:50.901+0100"
2024-03-01T09:02:50.901+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: AzureRM Request: 
POST /subscriptions/1b9d79eb-3a60-42ee-837c-71141e1f3755/resourceGroups/rg-adacta-cloud-cost-mgmt/providers/Microsoft.Storage/storageAccounts/teststorage/listKeys?%24expand=kerb&api-version=2021-09-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.6 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 storage/2021-09-01 HashiCorp Terraform/1.6.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.94.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 0
X-Ms-Correlation-Request-Id: 0ebfb613-886c-6cd6-9542-17b9cfe17df4
Accept-Encoding: gzip: timestamp="2024-03-01T09:02:50.901+0100"
2024-03-01T09:02:51.365+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: AzureRM Response for https://management.azure.com/subscriptions/1b9d79eb-3a60-42ee-837c-71141e1f3755/resourceGroups/rg-adacta-cloud-cost-mgmt/providers/Microsoft.Storage/storageAccounts/teststorage/listKeys?%24expand=kerb&api-version=2021-09-01: 
HTTP/2.0 200 OK
Content-Length: 380
Cache-Control: no-cache
Content-Type: application/json
Date: Fri, 01 Mar 2024 08:03:03 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 0ebfb613-886c-6cd6-9542-17b9cfe17df4
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 11999
X-Ms-Request-Id: 20354044-4c8c-4ddf-bf0d-2fc780314ffc
X-Ms-Routing-Request-Id: GERMANYWESTCENTRAL:20240301T080303Z:9eb64878-c152-4f41-9fdd-216734fd8fe2

{"keys":[{"creationTime":"2024-02-07T20:48:16.4040434Z","keyName":"key1","value":"Opvob4wjh2Pw7PCzOdC7jOHvnUeaWnPc/V2rx+K2HOAiJ0SRn4gkqV8Z7oskg1FWiFIA09b+bx5Q+AStQJE+5w==","permissions":"FULL"},{"creationTime":"2024-02-07T20:48:16.4040434Z","keyName":"key2","value":"QWvjQn4ujWvMuAGgKO+PKqxWNlSnRahssj5KeIG/pqbS41RdsNaSHU0BQATIq1npJFS7VU16eMmn+AStr/Z/tA==","permissions":"FULL"}]}: timestamp="2024-03-01T09:02:51.365+0100"
2024-03-01T09:02:51.366+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: Cache Miss - looking up the account key for storage account "teststorage"..: timestamp="2024-03-01T09:02:51.366+0100"
2024-03-01T09:02:51.366+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: AzureRM Request: 
POST /subscriptions/1b9d79eb-3a60-42ee-837c-71141e1f3755/resourceGroups/rg-adacta-cloud-cost-mgmt/providers/Microsoft.Storage/storageAccounts/teststorage/listKeys?%24expand=kerb&api-version=2021-09-01 HTTP/1.1
Host: management.azure.com
User-Agent: Go/go1.21.6 (amd64-linux) go-autorest/v14.2.1 Azure-SDK-For-Go/v66.0.0 storage/2021-09-01 HashiCorp Terraform/1.6.6 (+https://www.terraform.io) Terraform Plugin SDK/2.10.1 terraform-provider-azurerm/3.94.0 pid-222c6c49-1b0a-5959-a213-6608f9eb8820
Content-Length: 0
X-Ms-Correlation-Request-Id: 0ebfb613-886c-6cd6-9542-17b9cfe17df4
Accept-Encoding: gzip: timestamp="2024-03-01T09:02:51.366+0100"
[0m[1mdata.azurerm_storage_table_entities.departments: Read complete after 3s [id=https://teststorage.table.core.windows.net/rdepartments(ff6c9437dadcd557af6d01127f6045eebc7ef272)][0m
2024-03-01T09:02:51.820+0100 [DEBUG] provider.terraform-provider-azurerm_v3.94.0_x5: AzureRM Response for https://management.azure.com/subscriptions/1b9d79eb-3a60-42ee-837c-71141e1f3755/resourceGroups/rg-adacta-cloud-cost-mgmt/providers/Microsoft.Storage/storageAccounts/teststorage/listKeys?%24expand=kerb&api-version=2021-09-01: 
HTTP/2.0 200 OK
Content-Length: 380
Cache-Control: no-cache
Content-Type: application/json
Date: Fri, 01 Mar 2024 08:03:03 GMT
Expires: -1
Pragma: no-cache
Server: Microsoft-Azure-Storage-Resource-Provider/1.0,Microsoft-HTTPAPI/2.0 Microsoft-HTTPAPI/2.0
Strict-Transport-Security: max-age=31536000; includeSubDomains
X-Content-Type-Options: nosniff
X-Ms-Correlation-Request-Id: 0ebfb613-886c-6cd6-9542-17b9cfe17df4
X-Ms-Ratelimit-Remaining-Subscription-Resource-Requests: 11998
X-Ms-Request-Id: 82c98bcd-b5ea-4b03-b218-feb56c99f052
X-Ms-Routing-Request-Id: GERMANYWESTCENTRAL:20240301T080303Z:b1eb510f-09f9-455d-9ccb-abb9f5dae4cb

{"keys":[{"creationTime":"2024-02-07T20:48:16.4040434Z","keyName":"key1","value":"Opvob4wjh2Pw7PCzOdC7jOHvnUeaWnPc/V2rx+K2HOAiJ0SRn4gkqV8Z7oskg1FWiFIA09b+bx5Q+AStQJE+5w==","permissions":"FULL"},{"creationTime":"2024-02-07T20:48:16.4040434Z","keyName":"key2","value":"QWvjQn4ujWvMuAGgKO+PKqxWNlSnRahssj5KeIG/pqbS41RdsNaSHU0BQATIq1npJFS7VU16eMmn+AStr/Z/tA==","permissions":"FULL"}]}: timestamp="2024-03-01T09:02:51.819+0100"
[0m[1mdata.azurerm_storage_table_entities.projects: Read complete after 3s [id=https://teststorage.table.core.windows.net/rprojects(a646ccc27611b263f55bf4514ccaf606c826d095)][0m
2024-03-01T09:02:51.898+0100 [DEBUG] provider.stdio: received EOF, stopping recv loop: err="rpc error: code = Unavailable desc = error reading from server: EOF"
2024-03-01T09:02:51.901+0100 [DEBUG] provider: plugin process exited: path=.terraform/providers/registry.terraform.io/hashicorp/azurerm/3.94.0/linux_amd64/terraform-provider-azurerm_v3.94.0_x5 pid=121998
2024-03-01T09:02:51.908+0100 [DEBUG] provider: plugin exited
2024-03-01T09:02:51.908+0100 [DEBUG] building apply graph to check for errors
2024-03-01T09:02:51.908+0100 [DEBUG] ProviderTransformer: "data.azurerm_storage_table_entities.departments (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/azurerm"]
2024-03-01T09:02:51.908+0100 [DEBUG] ProviderTransformer: "data.azurerm_storage_table_entities.projects (expand)" (*terraform.nodeExpandApplyableResource) needs provider["registry.terraform.io/hashicorp/azurerm"]
2024-03-01T09:02:51.908+0100 [DEBUG] ReferenceTransformer: "data.azurerm_storage_table_entities.departments (expand)" references: []
2024-03-01T09:02:51.909+0100 [DEBUG] ReferenceTransformer: "data.azurerm_storage_table_entities.projects (expand)" references: []
2024-03-01T09:02:51.909+0100 [DEBUG] ReferenceTransformer: "output.project_entities (expand)" references: [data.azurerm_storage_table_entities.projects (expand)]
2024-03-01T09:02:51.909+0100 [DEBUG] ReferenceTransformer: "output.department_entities (expand)" references: [data.azurerm_storage_table_entities.departments (expand)]
2024-03-01T09:02:51.909+0100 [DEBUG] ReferenceTransformer: "provider[\"registry.terraform.io/hashicorp/azurerm\"]" references: []
2024-03-01T09:02:51.909+0100 [DEBUG] pruneUnusedNodes: data.azurerm_storage_table_entities.departments (expand) is no longer needed, removing
2024-03-01T09:02:51.909+0100 [DEBUG] pruneUnusedNodes: data.azurerm_storage_table_entities.projects (expand) is no longer needed, removing
2024-03-01T09:02:51.909+0100 [DEBUG] pruneUnusedNodes: provider["registry.terraform.io/hashicorp/azurerm"] is no longer needed, removing
2024-03-01T09:02:51.909+0100 [INFO]  backend/local: plan operation completed

Changes to Outputs:
  [32m+[0m[0m department_entities = {
      [32m+[0m[0m filter               = "PartitionKey eq 'depkey'"
      [32m+[0m[0m id                   = "https://teststorage.table.core.windows.net/rdepartments(ff6c9437dadcd557af6d01127f6045eebc7ef272)"
      [32m+[0m[0m items                = [
          [32m+[0m[0m {
              [32m+[0m[0m partition_key = "depkey"
              [32m+[0m[0m properties    = {
                  [32m+[0m[0m name = "dep01"
                }
              [32m+[0m[0m row_key       = "row01"
            },
          [32m+[0m[0m {
              [32m+[0m[0m partition_key = "depkey"
              [32m+[0m[0m properties    = {
                  [32m+[0m[0m name = "dep02"
                }
              [32m+[0m[0m row_key       = "row02"
            },
        ]
      [32m+[0m[0m select               = []
      [32m+[0m[0m storage_account_name = "teststorage"
      [32m+[0m[0m table_name           = "rdepartments"
      [32m+[0m[0m timeouts             = [90mnull[0m[0m
    }
  [32m+[0m[0m project_entities    = {
      [32m+[0m[0m filter               = "PartitionKey eq 'projkey'"
      [32m+[0m[0m id                   = "https://teststorage.table.core.windows.net/rprojects(a646ccc27611b263f55bf4514ccaf606c826d095)"
      [32m+[0m[0m items                = [
          [32m+[0m[0m {
              [32m+[0m[0m partition_key = "projkey"
              [32m+[0m[0m properties    = {
                  [32m+[0m[0m project = "proj01"
                }
              [32m+[0m[0m row_key       = "row01"
            },
          [32m+[0m[0m {
              [32m+[0m[0m partition_key = "projkey"
              [32m+[0m[0m properties    = {
                  [32m+[0m[0m project = "proj02"
                }
              [32m+[0m[0m row_key       = "row02"
            },
        ]
      [32m+[0m[0m select               = [
          [32m+[0m[0m "department",
          [32m+[0m[0m "project",
          [32m+[0m[0m "RowKey",
          [32m+[0m[0m "PartitionKey",
        ]
      [32m+[0m[0m storage_account_name = "teststorage"
      [32m+[0m[0m table_name           = "rprojects"
      [32m+[0m[0m timeouts             = [90mnull[0m[0m
    }

You can apply this plan to save these new output values to the Terraform
state, without changing any real infrastructure.
[90m
─────────────────────────────────────────────────────────────────────────────[0m

Note: You didn't use the -out option to save this plan, so Terraform can't
guarantee to take exactly these actions if you run "terraform apply" now.

Expected Behaviour

Expected is that output "project_entities" has in each item "properties" property with "project" and "department" property.

Expected is that the documentation page for the data source has the correct name of the data source.

Expected is that unit tests check also the number of properties that are returned with and without "select".

Actual Behaviour

"project_entities" output has only one property in "properties", sometimes "department" and sometimes "project".

If I use "azurerm_storage_table_entity" data source to retrieve one entity from "projects" table, I will get in "entities" property both properties ("department" and "project"). So, I conclude that storage account and table with data is functioning correctly.

Steps to Reproduce

terraform init
terraform plan
check output values as explained in actual behavior.

Important Factoids

No response

References

No response

imileusnic commented 9 months ago

Hi @mybayern1974 , I apologize for disturbing you, but I would appreciate it if you could explain to me how come that this is an enhancement and not a bug. If this is an enhancement that means that the data source should not return all properties of all entities that are satisfying the filter condition. Please could you explain to me which properties of the filtered entities this data source should return?

mybayern1974 commented 8 months ago

@imileusnic , thank you for calling out. By re-reading the issue description, I'm convinced, at least apparently, this looks like a bug rather than enhancement. I'm re-labeling it. Please expect either me or my colleague would chime in later to share more findings of this issue.

imileusnic commented 8 months ago

@mybayern1974 , thank you very much for help.

github-actions[bot] commented 7 months ago

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

hashicorp / terraform-provider-azurerm