AWS::QuickSight::Template - AWS::QuickSight::*

pgarbe commented 4 years ago

1. Title

AWS::QuickSight::*

2. Scope of request

There'se no QuickSight resources available, although the scope of the API has recently been extended.

3. Expected behavior

Create DataSets, Analysis, Dashboards and Reportings via CloudFormation.

5. Helpful Links to speed up research and evaluation

https://docs.aws.amazon.com/quicksight/latest/APIReference/API_Operations.html

6. Category (required)

Analytics (Athena, EMR, Glue,...)

AndrewCi commented 4 years ago

Is there any sort of timeline or update on when additional CloudFormation support for QuickSight will be available?

humanzz commented 4 years ago

Hi there... any updates on this?

matthewcarlisle commented 4 years ago

Bueller?

rianwouters commented 4 years ago

It's quite unclear to me why AWS keeps on building specific resources. If you would only standardize a bit on the format of the HTTP API of the services, a custom resource could easily be build to have cloudformation support readily available as soon as the Service`s HTTP API / SDK support is available. See started this already almost 2 years ago, but it was especially difficult because there is hardly any standardisation in the SDK interfaces. See https://github.com/rianwouters/cfn-customresources

benzaam commented 4 years ago

Hi ! Any update on this? This would come handy, certainly with creating templates. This is currently a pain to do with the AWS CLI...

moltar commented 3 years ago

@rohits-spec do you have any insight as to how soon this is coming? We are planning our deployment of QS right now, and we need to know whether to use work arounds or if CF will be supported any day now?

Will this take:

[ ] week
[ ] month
[ ] quarter

?

Thanks! ❤️

aidansteele commented 3 years ago

Quicksight now has CloudFormation support for:

AWS::QuickSight::Analysis
AWS::QuickSight::Dashboard
AWS::QuickSight::Template
AWS::QuickSight::Theme

I haven't actually tried them yet to see how complete they are, but they arrived yesterday!

moltar commented 3 years ago

Ah cool.

I was hoping there'd be support for actually deploying the QS service itself.

It's annoying to do manual setup in staging, dev and testing.

Especially when all other parts are already automated with CDK.

rohits-spec commented 3 years ago

Closing this issue as Quicksight onboarded CloudFormation on 1/14.

otakusid commented 3 years ago

Any chances that samples with CloudFormation resources configuration will be added to the documentation? It will be cool to have an example of an Analysis setup with some visualizations configuration

pgarbe commented 3 years ago

@rohits-spec I'd not close it. It's not usable at the moment.

RemjeyLiu commented 3 years ago

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization
Namespace
Users
Groups
Group Memberships
Data Sources
Data Sets
Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

rohits-spec commented 3 years ago

Reopening this issue due to the concerns raised here by community members. I will work with Quicksight team and provide an update here.

austinlasseter commented 3 years ago

I agree with the commenters above. We need robust CloudFormation support for QuickSight. Re-building the same dashboards from scratch in dev, test and prod using nothing but the GUI... really kinda sucks.

danieljamesscott commented 3 years ago

Probably not quite the right place to ask for this for it, but API support for configuring VPC Connections to QuickSight would be nice. https://docs.aws.amazon.com/quicksight/latest/user/vpc-creating-a-connection-in-quicksight.html

Then adding CloudFormation support for it too would be wonderful!

moltar commented 3 years ago

Ha, @danieljamesscott was just looking for the exact same thing today. I guess there is no way, eh?

danieljamesscott commented 3 years ago

Ha, @danieljamesscott was just looking for the exact same thing today. I guess there is no way, eh?

Not that I've found. 😢

gtfuhr commented 3 years ago

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization

Namespace

Users

Groups

Group Memberships

Data Sources

Data Sets

Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

Is there any estimate for the deployment of the DataSets, DataSources, and Permissions QuickSight/CloudFormation features?

Lanayx commented 3 years ago

Can anybody provide any examples of using these new CloudFormation or CDK constructs? The documentation is pretty useless

mmuller88 commented 3 years ago

Here we go. Will do a blog post soon and create those CDK custom constructs dataset.txt datasource.txt

accrisoftteam commented 3 years ago

I recently attempted to use AWS::QuickSight::DataSource and ran into an Internal Error when attempting to use RdsParameters. We have worked around the issue thanks to AWS Support (use MySqlParameters instead) but wanted to mention it here as Support said it is reported as non-functional.

mmuller88 commented 3 years ago

Here we go. Post: https://martinmueller.dev/qs-constructs-eng/ GitHub Repo: https://github.com/mmuller88/cdk-quicksight-constructs

mmuller88 commented 3 years ago

Lol looks like AWS added DataSource and DataSet in the meantime D: ? https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-dataset.html https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-datasource.html

so this issue can be closed

rohits-spec commented 3 years ago

Quicksight has onboarded the datasource and dataset resources https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/AWS_QuickSight.html .

nathanagez commented 3 years ago

Hi @rohits-spec, I noticed that multiple fields are marked as not required (Required: No) in the documentation while they are, for instance with AWS::QuickSight::DataSource the DataSetId and AwsAccountId fields are required.

Also, when deploying my CloudFormation template I'm getting this error:

AWSTemplateFormatVersion: 2010-09-09
Parameters:
  AthenaWorkGroup:
    Type: String
    Default: amplifysandbox-AthenaWorkGroup-dev
  env:
    Type: String
    Default: debug
  appName:
    Type: String
    Default: testNathan
Resources:
  QuickSightDataSource:
    Type: 'AWS::QuickSight::DataSource'
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSourceId: cdkdatasource
      DataSourceParameters:
        AthenaParameters:
          WorkGroup: !Ref AthenaWorkGroup
      Permissions:
        - Actions:
            - 'quicksight:DescribeDataSource'
            - 'quicksight:DescribeDataSourcePermissions'
            - 'quicksight:PassDataSource'
            - 'quicksight:UpdateDataSource'
            - 'quicksight:DeleteDataSource'
            - 'quicksight:UpdateDataSourcePermissions'
          Principal: !Sub 
            - 'arn:aws:quicksight:us-west-2:${Account}:user/default/nathan'
            - Account: !Ref 'AWS::AccountId'
      Name: cdkdatasource
      Type: ATHENA

mjdean1994 commented 3 years ago

@rohits-spec it seems to me like the features required to use QuickSight at an Enterprise level have not yet been enabled for CloudFormation.

The thing that immediately comes to mind is VPC Connection.

The DataSource object supports passing a VpcConnectionArn here: https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-datasource.html

But there is no way to create that resource through CloudFormation. That would mean having to create something manually through the UI.

At least for my organization, this makes it impossible to use QuickSight without direct UI access, which is very frowned upon in our production environments.

I would encourage the team to keep this issue open until QuickSight is fully supported through CloudFormation.

padaszewski commented 3 years ago

@rohits-spec The DataSource resource does not have the data source parameter for AWS IoT Analytics, which is the most important one for me atm, and also some others.

Dazza65 commented 3 years ago

I can create a DataSource and a DataSet with CF but they aren't visible in the UI although I can list and describe them with the CLI.

The details that are returned through a describe-data-set or describe-data-source are exactly the same when compared to a resource that is created through the UI.

Dazza65 commented 3 years ago

I can create a DataSource and a DataSet with CF but they aren't visible in the UI although I can list and describe them with the CLI.

The details that are returned through a describe-data-set or describe-data-source are exactly the same when compared to a resource that is created through the UI.

Actually I managed to fix this by adding a Permissions property to both the DataSource and DataSet as it seems the QuickSight UI only displays objects created by your QuickSight user by default. BTW, I'm only using the Standard Edition.

justinhauer commented 3 years ago

I am trying to create this but getting stuck on what the datasourceId should be....

justinhauer commented 3 years ago

@Dazza65 would you mind showing a generalized example?

lukelafountaine commented 3 years ago

In my brief experience trying to use QuickSight with CloudFormation, I've already seen a number of problems:

As mentioned above by @nathanagez, the documentation is out of sync with reality. The Required field frequently does not reflect its true value (e.g. here and here)
There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?
I was expecting a way to describe the details of an analysis rather than just defining its related data sets (i.e. adding sheets, adding visuals to those sheets, etc.). Not being able to describe these things will require me to edit them through the UI in production, which somewhat defeats the purpose of using IAC. Like @mjdean1994, my organization really frowns on having to click through a UI to configure things

@rohits-spec I'd also suggest that this ticket is reopened, because the CloudFormation support does not seem usable in its current state.

CC @jthomerson

Dazza65 commented 3 years ago

@Dazza65 would you mind showing a generalized example?

@justinhauer - you may have solved this by now but the datasource ID can be anything you want :)

Here's an extract from my template.yml from my TriggerQS_lambda repo:

QSDataSource: Type: AWS::QuickSight::DataSource Properties: AwsAccountId: !Ref AWS::AccountId DataSourceId: !Join ['-', [!Ref AWS::AccountId, '20210625', '01']] DataSourceParameters: S3Parameters: ManifestFileLocation: Bucket: devopskpi-346327484579-etloutput Key: quicksight.json Name: "DevOpsKPILTTD" Type: S3 Permissions:

Actions:
- quicksight:DescribeDataSource
- quicksight:DescribeDataSourcePermissions
- quicksight:PassDataSource Principal: !Sub
- 'arn:aws:quicksight:ap-southeast-2:${Account}:user/default/harrisd'
- Account: !Ref 'AWS::AccountId'

AndrewCi commented 3 years ago

In my brief experience trying to use QuickSight with CloudFormation, I've already seen a number of problems:

As mentioned above by @nathanagez, the documentation is out of sync with reality. The Required field frequently does not reflect its true value (e.g. here and here)

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I was expecting a way to describe the details of an analysis rather than just defining its related data sets (i.e. adding sheets, adding visuals to those sheets, etc.). Not being able to describe these things will require me to edit them through the UI in production, which somewhat defeats the purpose of using IAC. Like @mjdean1994, my organization really frowns on having to click through a UI to configure things

@rohits-spec I'd also suggest that this ticket is reopened, because the CloudFormation support does not seem usable in its current state.

CC @jthomerson

Agreed this issue should be re-opened.

gtfuhr commented 3 years ago

This issue should be re-opened. o/

derrik-fleming commented 3 years ago

Also, requesting to re-open.

mukaddim98 commented 3 years ago

I'm really new to cloudformation and I was trying to build a quicksight dataset, analysis, and dashboard. The documentation has templates and structures of how to create the cloudformation templates, but no real examples. Examples for each of those three templates would be really helpful.

nathanagez commented 3 years ago

Another request feature should be to support account creation through CloudFormation ? Is it possible @rohits-spec ?

Also, just for information, AWS support gave me some information, if it can help:

Thank you for your patience,

Recently the internal service team replied and for that here are the key points:

We have fixed the changes that should show proper error message in Cloudformation, instead of 'null'.
Additionally, when using a custom workgroup there are some additional steps that should be considered, please follow along:

QuickSight uses a default bucket aws-athena-query-results-[REGION]-[ACCOUNT_ID] for primary workgroup and have required permissions on it by default when you enable Athena permission for QuickSight. You have this bucket in your account 'aws-athena-query-results-us-west-2-xxxx', therefore you were able to create data source with primary.

For non-primary workgroup, we require an additional S3 bucket location, and for that additional bucket a new Policy must exist attached to the role 'aws-quicksight-s3-consumers-role-v0'[3..4].

The policy that allows that is the following:

=> CustomAWSQuicksightAthenaAccess

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "s3:GetBucketLocation",
                "s3:GetObject",
                "s3:ListBucket",
                "s3:ListBucketMultipartUploads",
                "s3:ListMultipartUploadParts",
                "s3:AbortMultipartUpload",
                "s3:CreateBucket",
                "s3:PutObject"
            ],
            "Resource": [
                "arn:aws:s3:::<name of the bucket used for the non-primary workgroup>*"
            ]
        }
    ]
}

Add the policy for the existing role: arn:aws:iam::ACCOUND_ID:role/service-role/aws-quicksight-s3-consumers-role-v0

Once that exists, you will be able to create the resource as non-primary workgroup. I personally tested the above in my own account and it worked.

I hope the above helped, also if you have additional queries regarding it please let us know. We will be happy to assist till everything is clear.

References:

[1] AWS::QuickSight::DataSource - https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-quicksight-datasource.html [2] https://docs.aws.amazon.com/athena/latest/ug/workgroups-settings-override.html [3] Insufficient Permissions When Using Athena with Amazon QuickSight - https://docs.aws.amazon.com/quicksight/latest/user/troubleshoot-athena-insufficient-permissions.html [4] Workgroup or Output Errors When Using Amazon Athena with Amazon QuickSight - https://docs.aws.amazon.com/quicksight/latest/user/troubleshoot-athena-workgroup.html

marktheskies commented 3 years ago

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I am also running into this issue and agree that this should be re-opened.

rohits-spec commented 3 years ago

Reopening this issue for the further analysis.

luchees commented 3 years ago

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I am also running into this issue and agree that this should be re-opened.

It's not possible creating this with API/CLI nor CFN and CDK. Seems to be not only CFN who has a circular dependency.

amukasyan commented 3 years ago

Does anyone here know the proper way to format the data_source_parameter for an S3 connection? I've tried {'s3_parameters': {'manifest_file_location': {'bucket': 'bucket_name', 'key': 'manifest.csv'}}} as well as {'S3Parameters': {'ManifestFileLocation': {'Bucket': 'bucket_name', 'Key': 'manifest.csv'}}} with no success and those seem to be the 2 options that documentation points toward for json format. I am writing in python using the AWS CDK.

bhumika-18 commented 3 years ago

In my brief experience trying to use QuickSight with CloudFormation, I've already seen a number of problems:

As mentioned above by @nathanagez, the documentation is out of sync with reality. The Required field frequently does not reflect its true value (e.g. here and here)

There seems to be a circular dependency between the AWS::QuickSight::Analysis and AWS::QuickSight::Template resources. Seemingly, you can't create an analysis without referencing a template. But you can't create a template without referencing an analysis or another template. How are you supposed to create the first template?

I was expecting a way to describe the details of an analysis rather than just defining its related data sets (i.e. adding sheets, adding visuals to those sheets, etc.). Not being able to describe these things will require me to edit them through the UI in production, which somewhat defeats the purpose of using IAC. Like @mjdean1994, my organization really frowns on having to click through a UI to configure things

@rohits-spec I'd also suggest that this ticket is reopened, because the CloudFormation support does not seem usable in its current state. CC @jthomerson

Agreed this issue should be re-opened.

Hey, i can help you out with the circular dependency part.

create a Template using an "Manually created Analysis" as a Source entity.
Then Create one more template and analysis using Step1 template as Source entity.
i am creating template in Step2 to create Dashboards using that template.

References : https://devops.learnquicksight.online/quicksight-via-cloudformation.html you can download the cloudformation sample from here and they are using a publicly created template's arn, so you can describe to get the code of that.

Thankyou

karthikns16 commented 3 years ago

Also, would be cool if the following resources can be managed by CloudFormation too:

Account Customization

Namespace

Users

Groups

Group Memberships

Data Sources

Data Sets

Permissions (For Themes, Data Sources, Data Sets, Analyses, Templates, and Dashboards)

Please give this capability ASAP.

otakusid commented 2 years ago

no no no, there is still no way to describe details of analysis and there are no samples of resources usage in documentation

amukasyan commented 2 years ago

I strongly disagree with closing. Unless documentation has been greatly updated and all above concerns are addressed in a blog post.

luchees commented 2 years ago

Is there any timeline on fixing the circular dependency between Analysis and Template?

alerof commented 2 years ago

Has anybody created AWS::QuickSight::DataSet for ATHENA DataSource ?

I'm stuck with "Model validation failed (#: extraneous key [DataSourceArn] is not permitted)" (RequestToken: 61292db1-xxx, HandlerErrorCode: InvalidRequest)

I have

  QSDataSource:
    Type: AWS::QuickSight::DataSource
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSourceId: 'QSDataSource-XXX'
      Permissions:
        - Actions:
            - 'quicksight:DescribeDataSource'
            - 'quicksight:DescribeDataSourcePermissions'
            - 'quicksight:PassDataSource'
            - 'quicksight:UpdateDataSource'
            - 'quicksight:DeleteDataSource'
            - 'quicksight:UpdateDataSourcePermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      Name: !Sub 'ApiGw-${Environment}'
      SslProperties:
        DisableSsl: false
      Type: ATHENA

  QSDataSet:
    DependsOn: QSDataSource
    Type: AWS::QuickSight::DataSet
    Properties:
      AwsAccountId: !Ref 'AWS::AccountId'
      DataSetId: 'QSDataSet-XXX'
      ImportMode: SPICE
      Name: ApiGwShort
      Permissions:
        - Actions:
            - 'quicksight:CreateDataSet'
            - 'quicksight:DeleteDataSet'
            - 'quicksight:DescribeDataSet'
            - 'quicksight:DescribeDataSetPermissions'
            - 'quicksight:PassDataSet'
            - 'quicksight:UpdateDataSet'
            - 'quicksight:UpdateDataSetPermissions'
          Principal: !Sub
            - 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:user/default/myname'
            - Account: !Ref 'AWS::AccountId'
      PhysicalTableMap:
        RelationalTable:
          DataSourceArn: !Sub 'arn:aws:quicksight:eu-central-1:${AWS::AccountId}:datasource/QSDataSource-XXX'
          InputColumns:
            - servicename
            - dt
            - responsesize
          Name: apigw_short
          Schema: !Sub 'logs-db-${Environment}'

moltar commented 2 years ago

How do you create a new DataSource? It is asking for dataSourceId, and if I provide one, it errors out with:

Resource of type 'AWS::QuickSight::DataSource' with identifier 'Foo' was not found.

If I don't provide it, then it errors out with:

1 validation error detected: Value null at 'dataSourceId' failed to satisfy constraint: Member must not be null

So what's the correct approach?

I've tried the following values:

AB
b5172c44-86fa-49b2-bd8f-9679260dc378 (Random UUID)
null

moltar commented 2 years ago

Huh, so strange. The error:

Resource of type 'AWS::QuickSight::DataSource' with identifier 'My-DataSource-Name' was not found.

Is returned when specifying VpcConnectionProperties.VpcConnectionArn value that does not exist.

So weird and misleading.

aws-cloudformation / cloudformation-coverage-roadmap