aws-cloudformation / cloudformation-coverage-roadmap

The AWS CloudFormation Public Coverage Roadmap
https://aws.amazon.com/cloudformation/
Creative Commons Attribution Share Alike 4.0 International
1.11k stars 57 forks source link

Glue Crawler RecrawlPolicy doesn't work with SQS queue #1849

Open willsurridge opened 1 year ago

willsurridge commented 1 year ago

Name of the resource

AWS::Glue::Crawler

Resource Name

No response

Issue Description

Created a Glue crawler and an SQS queue to push S3 event notifications with associated permissions in a single template.

    const crawler = new aws_glue.CfnCrawler(this, 'analyticsCrawler', {
      role: crawlerRole.roleArn,
      targets: {
        s3Targets: [
          {
            path: `s3://${this.props.dataBucket.bucketName}/`,
            sampleSize: 10,
            eventQueueArn: crawlerQueue.queueArn,
          },
        ],
      },
      databaseName: database.databaseName,
      schedule: { scheduleExpression: 'cron(15 17 ? * FRI *)' },
      recrawlPolicy: { recrawlBehavior: 'CRAWL_EVENT_MODE' },
      schemaChangePolicy: {
        deleteBehavior: 'DEPRECATE_IN_DATABASE',
        updateBehavior: 'UPDATE_IN_DATABASE',
      },
    });

Expected Behavior

Both the Queue and Crawler should deploy and the Crawler should use the RecrawlPolicy to only crawl the relevant files.

Observed Behavior

Error message:

    "eventTime": "2023-11-08T16:52:48Z",
    "eventSource": "glue.amazonaws.com",
    "eventName": "CreateCrawler",
    "awsRegion": "eu-west-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "InvalidInputException",
    "errorMessage": "SQS queue arn:aws:sqs:eu-west-1:<account>:<recource> does not exist or the role provided does not have access to it.",
    "requestParameters": {
        "role": "arn:aws:iam::<account>:role/<resource>",
        "schedule": "cron(15 17 ? * FRI *)",

Test Cases

Deploying in a single template fails Deploying the queue first and then the crawler works fine, but deploying them at the same time fails. Adding a dependency to the crawler on the queue also doesn't work

Other Details

Spoken to AWS technical support (Case ID 14257963701) who said there was an internal ticket, but suggested raising a bug here too

Rizxcviii commented 1 month ago

It would be nice to know when we're actually getting an ETA for this, does Coming Soon mean that it's in the deployment pipeline, or some testing is being done?