[Bug]: Not getting schemas and tables from Athena datasource created through "aws_quicksight_data_source"

[parth-soni-hpe]

Terraform Core Version

v1.0.0

AWS Provider Version

4.64.0

Affected Resource(s)

aws_quicksight_data_source

My code:

resource "aws_quicksight_data_source" "default" {
  data_source_id = "test-datasource-id"
  name           = "My Cool Datasource"

  parameters {
    athena {
      work_group  = "primary"
    }
  }

  permission {
    actions = [
      "quicksight:DescribeDataSource",
      "quicksight:DescribeDataSourcePermissions",
      "quicksight:PassDataSource",
      "quicksight:UpdateDataSource",
      "quicksight:DeleteDataSource",
      "quicksight:UpdateDataSourcePermissions"

    ]
    principal = "my quicksight user/group arn"
  }

I tried both user and group arns. Datasource created using this does not have permission to fetch schemas and tables. I am getting empty from schemas get API. {"schemas":[]}, {"dataSources":[]} If I create the same data source with aws cli it is working, it is even working with local-exec null resource terraform option.

Things I have already verified:

1. My user I have all required lake formation permissions.
2. quick insight Service principal has all required permissions.
3. I am using same AWS creds for both aws cli and terraform provider tf.

Expected Behavior

It should create quicksight data source and datasource should have access to fetch schemas and run custom queries.

Actual Behavior

Datasource created through the terraform function is unable to fetch schemas, I am getting a blank json array. Custom query also not working and getting permission denied error.

Relevant Error/Panic Output Snippet

No response

Terraform Configuration Files

resource "aws_quicksight_data_source" "default" {
  data_source_id = "test-datasource-id"
  name           = "My Cool Datasource"

  parameters {
    athena {
      work_group  = "primary"
    }
  }

  permission {
    actions = [
      "quicksight:DescribeDataSource",
      "quicksight:DescribeDataSourcePermissions",
      "quicksight:PassDataSource",
      "quicksight:UpdateDataSource",
      "quicksight:DeleteDataSource",
      "quicksight:UpdateDataSourcePermissions"

    ]
    principal = "<QUICKSIGHT_USER/GROUP_ARN>"
  }

BACKEND tf:

terraform {
  backend "s3" {
    config = {
      encrypt = true
    }
  }
  required_version = ">= 0.12.0"
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 4.64.0"
    }
  }
}

Steps to Reproduce

1. Write terraform function to create a datasource and run against the aws account with IAM user having the privilege to setup quicksight and create data source.

Debug Output

region: us-west-2 timestamp:
1694088151580 requestId:
abb13dee- sourceErrorCode:
100071 sourceErrorMessage: [Simba]AthenaJDBC An error has been thrown from the AWS Athena client. Insufficient permissions to execute the query. Insufficient Lake Formation permission(s) on worldcities [Execution ID: ] sourceErrorState:
HY000 sourceException:
java.sql.SQLException sourceType: ATHENA

Panic Output

No response

Important Factoids

No response

References

No response

Would you like to implement a fix?

None

[github-actions[bot]]

Community Note

Voting for Prioritization

• Please vote on this issue by adding a 👍 reaction to the original post to help the community and maintainers prioritize this request.
• Please see our prioritization guide for information on how we prioritize.
• Please do not leave "+1" or other comments that do not add relevant new information or questions, they generate extra noise for issue followers and do not help prioritize the request.

Volunteering to Work on This Issue

• If you are interested in working on this issue, please leave a comment.
• If this would be your first contribution, please review the contribution guide.

[justinretzolk]

Hey @parth-soni-hpe 👋 Thank you for taking the time to raise this! I'm not certain that this is the issue, however, there are areas within AWS where tasks are automated when resources are created in other ways, but not when created via the API, as Terraform does. It may be that there is an additional step that needs to be taken, but that would be outside of the scope of this repository (we use Issues in this repository to track feature enhancements and bugs in the AWS Provider). So that we can maintain focus on that, we instead ask that broader questions like this are raised using one of the Community Resources, such as the AWS Provider forum. With that in mind, I'm going to close this issue.

[adam-rummer-hpe]

@justinretzolk I'm trying to get my head around why this isn't a bug? Could you or somebody else please offer some guidance on how one should use this resource to achieve our aim if it's working as intended? Do we need to pair it with another resource?

[github-actions[bot]]

I'm going to lock this issue because it has been closed for 30 days ⏳. This helps our maintainers find and focus on the active issues. If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

[justinretzolk]

Hey @adam-rummer-hpe 👋 Apologies for the delay here -- I've been largely out of office for the last few weeks, but wanted to make sure to answer you, despite the bot's auto-lock 🙂. When reviewing this particular issue, what struck me was the lack of any errors, outside of the apparent permissions issue once trying to enter the query. This leads me to believe that this is a situation where AWS automatically handles some step in the setup process when creating the resources via the UI, which it doesn't automatically handle when creating the resources via the API (or, in this case, Terraform). This is a fairly common pattern in AWS, and something we'd consider a configuration issue rather than a bug.

Usually AWS calls these scenarios our in their documentation, so you may be able to find some hints there as far as any additional resources that might need to be created. Otherwise, it may be worthwhile to inspect the resources created via the console vs. Terraform to see if there's anything obviously missing there.

All of that said, if you find something that indicates that the provider isn't sending the appropriate data, or is missing some configuration option, we'd be happy to look into that as a bug and/or feature request.

[adam-rummer-hpe]

Thanks so much for getting back to us @justinretzolk, really appreciate you taking the time :)

Will look into this further. We're currently working around using a null resource to call out to the CLI, which does work. Would you still consider that a configuration problem even though the CLI uses the API?

[justinretzolk]

Hey @adam-rummer-hpe 👋 Thanks for calling that out the usage of the CLI again -- looking the original issue description over again, I'd overlooked that bit, and that definitely could make a difference here. I'm going to re-open the issue so we can keep talking about this a bit more. Sorry about that.

Are you able to capture the payload being sent from the AWS CLI and then provide that and trace logging from Terraform so that we could compare them? That might give us a clue as to whether something is missing.

[parth-soni-hpe]

Hey @adam-rummer-hpe 👋 Thanks for calling that out the usage of the CLI again -- looking the original issue description over again, I'd overlooked that bit, and that definitely could make a difference here. I'm going to re-open the issue so we can keep talking about this a bit more. Sorry about that.

Are you able to capture the payload being sent from the AWS CLI and then provide that and trace logging from Terraform so that we could compare them? That might give us a clue as to whether something is missing.

This is what we are providing while using AWS CLI:

aws quicksight create-data-source  --aws-account-id 097795826936 --data-source-id test-data-sourc
e --name test-data-source --type ATHENA --data-source-parameters '{
          "AthenaParameters": {
            "WorkGroup": "primary"
          }
         }'  --permissions '{  "Principal" : "QUICKSIGHT_GROUP_ARN", "Actions" : [
    "quicksight:DescribeDataSource",
     "quicksight:DescribeDataSourcePermissions",
          "quicksight:PassDataSource",
          "quicksight:UpdateDataSource",
          "quicksight:DeleteDataSource",
          "quicksight:UpdateDataSourcePermissions"
        ] }'```

And this is when we are using terraform :
`aws quicksight create-data-source --aws-account-id 097795826936 --data-source-id test-data-source --name test-data-source --type ATHENA --data-source-parameters '{
  "AthenaParameters": {
    "WorkGroup": "primary"
  }
}' --permissions '{
  "Principal": "QUICKSIGHT_GROUP_ARN",
  "Actions": [
    "quicksight:DescribeDataSource",
    "quicksight:DescribeDataSourcePermissions",
    "quicksight:PassDataSource",
    "quicksight:UpdateDataSource",
    "quicksight:DeleteDataSource",
    "quicksight:UpdateDataSourcePermissions"
  ]
}'`

I don't notice a significant difference in the payload used for both cases and the AWS Role we are utilizing when running both AWS CLI and Terraform is the same, so there isn't a permission issue. However, I'm unsure why the issue we're experiencing is only with the data source created through Terraform.