Support JSON files for S3 Control `JobManifestSpec`

[tim-finnigan]

Discussion here: https://github.com/aws/aws-cli/discussions/7941

[tim-finnigan]

P90584535

[tim-finnigan]

We heard back from a member of the S3 team who noted that they do support supplying the manifest.json file for inventory reports. See https://docs.aws.amazon.com/AmazonS3/latest/userguide/batch-ops-create-job.html

Amazon S3 Inventory report – Must be a CSV-formatted Amazon S3 Inventory report. You must specify the manifest.json file that is associated with the inventory report. For more information about inventory reports, see Amazon S3 Inventory. If the inventory report includes version IDs, S3 Batch Operations operates on the specific object versions.

The CSV in the name of the S3InventoryReport_CSV_20161130 object is just to indicate that the underlying format of the Inventory Report must be CSV, not ORC or parquet.

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.

[damolaobaleke]

@tim-finnigan @trevorrowe
How is that done then ? because yes they do support it but only from the GUI or console but not from the sdk or api. Could they show an example on how to pass a manifest.json file in the sdk it ? without getting the error

Reasons for failure
Reading the manifest is forbidden: 403 Forbidden

https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_JobManifestSpec.html

The documentation literally shows only a manifest.csv being supplied . even the example code is a csv file

Java

JobManifest manifest = new JobManifest()
                    .withSpec(new JobManifestSpec()
                            .withFormat("S3BatchOperations_CSV_20180820")
                            .withFields(new String[]{
                                    "Bucket", "Key"
                            }))
                    .withLocation(new JobManifestLocation()
                            .withObjectArn("arn:aws:s3:::my_manifests/manifest.csv")
                            .withETag("60e460c9d1046e73f7dde5043ac3ae85"));

python boto3

Manifest={
            'Spec': {
                'Format': 'S3BatchOperations_CSV_20180820',
                'Fields': ['Bucket','Key']
            },
            'Location': {
                'ObjectArn':f"arn:aws:s3:::{manifest_path}",
                'ETag': e_tag.replace('"', '').strip()
            }
},

What would be the valid value to specify json as the format because even the name here S3InventoryReport_CSV_20161130 which is said to be the underling format for the inventory report is completely different from the valid name value S3BatchOperations_CSV_20180820 used to specify the manifest format

S3InventoryReport_CSV_20161130 This isn't a valid value for the manifest format even. it should be an S3BatchOperations

[tim-finnigan]

Thanks for following up — the manifest_path in your example above can reference a JSON file. For example: https://stackoverflow.com/a/63515562. As previously mentioned the CSV in the name of S3InventoryReport_CSV_20161130 is just to indicate that the underlying format of the Inventory Report must be CSV, not ORC or parquet.

If you have suggestions for improving the API documentation we recommend reaching out directly through the Provide feedback links at the bottom of the API documentation page (https://docs.aws.amazon.com/AmazonS3/latest/API/API_control_CreateJob.html).

[github-actions[bot]]

This issue is now closed.

Comments on closed issues are hard for our team to see. If you need more assistance, please either tag a team member or open a new issue that references this one. If you wish to keep having a conversation with other community members under this issue feel free to do so.