aws-cloudformation / cloudformation-coverage-roadmap

The AWS CloudFormation Public Coverage Roadmap
https://aws.amazon.com/cloudformation/
Creative Commons Attribution Share Alike 4.0 International
1.11k stars 54 forks source link

Glue Crawler RecrawlPolicy doesn't work with SQS queue #1849

Open willsurridge opened 9 months ago

willsurridge commented 9 months ago

Name of the resource

AWS::Glue::Crawler

Resource Name

No response

Issue Description

Created a Glue crawler and an SQS queue to push S3 event notifications with associated permissions in a single template.

    const crawler = new aws_glue.CfnCrawler(this, 'analyticsCrawler', {
      role: crawlerRole.roleArn,
      targets: {
        s3Targets: [
          {
            path: `s3://${this.props.dataBucket.bucketName}/`,
            sampleSize: 10,
            eventQueueArn: crawlerQueue.queueArn,
          },
        ],
      },
      databaseName: database.databaseName,
      schedule: { scheduleExpression: 'cron(15 17 ? * FRI *)' },
      recrawlPolicy: { recrawlBehavior: 'CRAWL_EVENT_MODE' },
      schemaChangePolicy: {
        deleteBehavior: 'DEPRECATE_IN_DATABASE',
        updateBehavior: 'UPDATE_IN_DATABASE',
      },
    });

Expected Behavior

Both the Queue and Crawler should deploy and the Crawler should use the RecrawlPolicy to only crawl the relevant files.

Observed Behavior

Error message:

    "eventTime": "2023-11-08T16:52:48Z",
    "eventSource": "glue.amazonaws.com",
    "eventName": "CreateCrawler",
    "awsRegion": "eu-west-1",
    "sourceIPAddress": "cloudformation.amazonaws.com",
    "userAgent": "cloudformation.amazonaws.com",
    "errorCode": "InvalidInputException",
    "errorMessage": "SQS queue arn:aws:sqs:eu-west-1:<account>:<recource> does not exist or the role provided does not have access to it.",
    "requestParameters": {
        "role": "arn:aws:iam::<account>:role/<resource>",
        "schedule": "cron(15 17 ? * FRI *)",

Test Cases

Deploying in a single template fails Deploying the queue first and then the crawler works fine, but deploying them at the same time fails. Adding a dependency to the crawler on the queue also doesn't work

Other Details

Spoken to AWS technical support (Case ID 14257963701) who said there was an internal ticket, but suggested raising a bug here too