Ingestion fails on bucket objects with acl set to `public-read-write`

[akingscote]

Im deploying my infrastructure with terraform:

resource "aws_s3_bucket_object" "xxxxxxxxxxxxxx" {
  bucket   = aws_s3_bucket.xxxxxxxx.bucket
  key      = "xxxxxxxxxxxxxx"
  source   = "xxxxxxxxxxxxxx"
  acl = "public-read-write"
}

Switching to just public-read, the ingestion dosent have any problems.

Ingestion fails with the following:

╭─────────────────────────────── Traceback (most recent call last) ────────────────────────────────╮
│ /opt/awspx/cli.py:402 in main                                                                    │
│                                                                                                  │
│   399 │   │   console.start()                                                                    │
│   400 │                                                                                          │
│   401 │   try:                                                                                   │
│ ❱ 402 │   │   args.func(args)                                                                    │
│   403 │                                                                                          │
│   404 │   except (KeyboardInterrupt, SystemExit):                                                │
│   405 │   │   console.stop()                                                                     │
│                                                                                                  │
│ /opt/awspx/cli.py:150 in handle_ingest                                                           │
│                                                                                                  │
│   147 │   except ClientError as e:                                                               │
│   148 │   │   console.critical(e)                                                                │
│   149 │                                                                                          │
│ ❱ 150 │   ingestor = IngestionManager(session=session, console=console, services=args.services   │
│   151 │   │   │   │   │   │   │   │   db=args.database, quick=args.quick, skip_actions=args.sk   │
│   152 │   │   │   │   │   │   │   │   only_types=args.only_types, skip_types=args.skip_types,    │
│   153 │   │   │   │   │   │   │   │   only_arns=args.only_arns, skip_arns=args.skip_arns)        │
│                                                                                                  │
│ /opt/awspx/lib/aws/ingestor.py:82 in __init__                                                    │
│                                                                                                  │
│     79 │   │   self.load_transitives()                                                           │
│     80 │   │                                                                                     │
│     81 │   │   if not skip_actions:                                                              │
│ ❱   82 │   │   │   self.load_actions()                                                           │
│     83 │   │                                                                                     │
│     84 │   │   self.zip = self.save(db)                                                          │
│     85                                                                                           │
│                                                                                                  │
│ /opt/awspx/lib/aws/ingestor.py:275 in load_actions                                               │
│                                                                                                  │
│    272 │   │   │   │                                                                             │
│    273 │   │   │   │   acl = BucketACL(resource, resources) \                                    │
│    274 │   │   │   │   │   if resource.label() == "AWS::S3::Bucket" \                            │
│ ❱  275 │   │   │   │   │   else ObjectACL(resource, resources)                                   │
│    276 │   │   │   │                                                                             │
│    277 │   │   │   │   self.update(acl.principals())                                             │
│    278 │   │   │   │   self.update(acl.actions())                                                │
│                                                                                                  │
│ /opt/awspx/lib/aws/policy.py:645 in __init__                                                     │
│                                                                                                  │
│   642 │   │   │   │   │   │   statement["Principal"] = {"AWS": grantee["URI"]}                   │
│   643 │   │   │   │                                                                              │
│   644 │   │   │   │   # Handle Actions                                                           │
│ ❱ 645 │   │   │   │   statement["Action"] = self.AccessControlList[permission]                   │
│   646 │   │   │   │                                                                              │
│   647 │   │   │   │   # Handle Resources (Bucket and Objects in Bucket)                          │
│   648 │   │   │   │   statement["Resource"] = [resource.id(), resource.id() + "/*"]              │
╰──────────────────────────────────────────────────────────────────────────────────────────────────╯
TypeError: unhashable type: 'list'

Thanks

[beatro0t]

Thanks for reporting the issue @akingscote! If possible, would you mind posting the actual ACL of the affected bucket so we can recreate the issue, removing anything potentially

[akingscote]

So it was a real basic infrastructure, i was just playing around with the tool. I wanted to see what happened when i had a bucket with the following:

resource "aws_s3_bucket" "xxxx" {
  bucket = format("%s-xxxx", var.company)
  acl    = "private"
}

Then with object(s) with a different acls. Like i said, public-read seems to be ok, but just not public-read-write.

The tool is fantastic btw! Really cool - hopefully I find some time to add support for more services.

[beatro0t]

Thats great to hear and would be fantastic!

I've tried to replicate the issue using the following terraform template, which include a public-read-write object acl:

provider "aws" {
  region  = "eu-west-1"
  profile = "default"
}

resource "aws_s3_bucket" "b" {
  bucket = "issue-48"
  acl    = "private"
}

resource "aws_s3_bucket_object" "object" {
  bucket = "issue-48"
  key    = "object.txt"
  source = "/tmp/object.txt"
  acl = "public-read-write"
}

However, this did not produce any errors while ingesting. Can you confirm the offending Object ACL resembles the following:

{
  "Owner": {
    "DisplayName": "**********",
    "ID": "****************************************************************"
  },
  "Grants": [
    {
      "Grantee": {
        "DisplayName": "**********",
        "ID": "****************************************************************",
        "Type": "CanonicalUser"
      },
      "Permission": "FULL_CONTROL"
    },
    {
      "Grantee": {
        "Type": "Group",
        "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
      },
      "Permission": "READ"
    },
    {
      "Grantee": {
        "Type": "Group",
        "URI": "http://acs.amazonaws.com/groups/global/AllUsers"
      },
      "Permission": "WRITE"
    }
  ]
}

[akingscote]

Thanks for looking into this, that looks about right!

I suspect i know the problem, which I probably should have mentioned from the beginning... I had made some modifications (hacks) to get the tool to run against localstack - so it must be a problem there. The terraform you posted is basically the same as what i was using. I hope you didnt spend too much time looking into this - thanks.

[beatro0t]

Fantastic, glad we got to the bottom of it and absolutely no problem at all!