Unable to write to output file - Failed to write data (access denied)

[rodrigohashimoto]

Hi,

I´m getting the error msg from mediaconvert when running the test. " Unable to write to output file [s3://outgestpool/TRAILER/Default/Thumbnails/TRAILER.0000000.jpg]: [Failed to write data: Access Denied] Error code 1401. " I checked the mediaconvert role which seems to be ok. To double check it I also run a separate convert job to write to the same bucket and using the same role and it worked.

Could someone help, pls?

Kind Regards, rodrigo

[gordonmckenzie]

There could be a few things here. Please try/check the following: -

1) Confirm that the output bucket that you are using to store the converted files ready for streaming is set to "Block all public access" OFF or ON.

2) If you are blocking public access (for example to use CloudFront signed urls), then please remove the following code from the job.json file: -

"DestinationSettings": { "S3Settings": { "AccessControl": { "CannedAcl": "PUBLIC_READ" } } } N.B. check to ensure that the JSON file is valid by linting it, as MediaConversion will not be invoked because the Lambda will fail (the error will be in CloudWatch).

3) In my setup, I am blocking public access to use CloudFront signed urls, therefore I also created a policy on the bucket to allow PutObject from MediaConverter (N.B. I am fully aware this may not actually be necessary)

{ "Version": "2008-10-17", "Id": "PolicyForCloudFrontPrivateContent", "Statement": [ { "Sid": "1", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::cloudfront:user/CloudFront Origin Access Identity XXXXXXXXXXX" }, "Action": "s3:GetObject", "Resource": "arn:aws:s3:::s3<YOUR OUTPUT BUCKET>/*" }, { "Sid": "Stmt1598211804529", "Effect": "Allow", "Principal": { "AWS": "arn:aws:iam::<UNIQUE ID>:role/MediaConvertRole" }, "Action": "s3:PutObject", "Resource": "arn:aws:s3:::<YOUR OUTPUT BUCKET>/*" } ] }

[rodrigohashimoto]

Thanks for your help Gordon,

just tried both on and off "block all public access" , also tried removing the "DestinationSettings" like you recommended. Still having same "access denied " error on media convert jobs. The execution results are below, do you have any other ideas you can share? many thanks.

Response: "statusCode": 200 }

Request ID: "4deac982-95e9-4736-b83c-5d650b9c7cf4"

Function logs: roup"}], "AdAvailOffset": 0} [INFO] 2020-08-25T10:16:18.192Z 4deac982-95e9-4736-b83c-5d650b9c7cf4 outputGroup['OutputGroupSettings']['Type'] == FILE_GROUP_SETTINGS [INFO] 2020-08-25T10:16:18.192Z 4deac982-95e9-4736-b83c-5d650b9c7cf4 templateDestinationKey == /MP4/ [INFO] 2020-08-25T10:16:18.192Z 4deac982-95e9-4736-b83c-5d650b9c7cf4 outputGroup['OutputGroupSettings']['Type'] == FILE_GROUP_SETTINGS [INFO] 2020-08-25T10:16:18.192Z 4deac982-95e9-4736-b83c-5d650b9c7cf4 templateDestinationKey == /Thumbnails/ [INFO] 2020-08-25T10:16:18.193Z 4deac982-95e9-4736-b83c-5d650b9c7cf4 {"Inputs": [{"AudioSelectors": {"Audio Selector 1": {"DefaultSelection": "DEFAULT", "ProgramSelection": 1, "Offset": 0}}, "FilterEnable": "AUTO", "TimecodeSource": "EMBEDDED", "PsiControl": "USE_PSI", "FilterStrength": 0, "DenoiseFilter": "DISABLED", "FileInput": "s3://rodeolabz-us-west-2/vodconsole/TRAILER.mp4", "VideoSelector": {"ColorSpace": "FOLLOW"}, "DeblockFilter": "DISABLED"}], "OutputGroups": [{"OutputGroupSettings": {"Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": {"Destination": "s3://outgest/TRAILER/Default/MP4/"}}, "Outputs": [{"VideoDescription": {"CodecSettings": {"Codec": "H_264", "H264Settings": {"NumberReferenceFrames": 3, "SpatialAdaptiveQuantization": "ENABLED", "TemporalAdaptiveQuantization": "ENABLED", "Syntax": "DEFAULT", "MinIInterval": 0, "UnregisteredSeiTimecode": "DISABLED", "FramerateControl": "INITIALIZE_FROM_SOURCE", "Telecine": "NONE", "Bitrate": 3000000, "FlickerAdaptiveQuantization": "DISABLED", "EntropyEncoding": "CABAC", "GopSizeUnits": "FRAMES", "GopBReference": "DISABLED", "AdaptiveQuantization": "HIGH", "InterlaceMode": "PROGRESSIVE", "QualityTuningLevel": "SINGLE_PASS", "Softness": 0, "CodecProfile": "MAIN", "RepeatPps": "DISABLED", "FieldEncoding": "PAFF", "SlowPal": "DISABLED", "NumberBFramesBetweenReferenceFrames": 2, "GopSize": 90, "ParControl": "INITIALIZE_FROM_SOURCE", "Slices": 1, "FramerateConversionAlgorithm": "DUPLICATE_DROP", "RateControlMode": "CBR", "SceneChangeDetect": "ENABLED", "CodecLevel": "AUTO", "GopClosedCadence": 1}}, "DropFrameTimecode": "ENABLED", "Sharpness": 50, "AntiAlias": "ENABLED", "Height": 720, "Width": 1280, "ScalingBehavior": "DEFAULT", "RespondToAfd": "NONE", "ColorMetadata": "INSERT", "AfdSignaling": "NONE", "TimecodeInsertion": "DISABLED"}, "AudioDescriptions": [{"CodecSettings": {"Codec": "AAC", "AacSettings": {"CodecProfile": "LC", "Specification": "MPEG4", "RateControlMode": "CBR", "AudioDescriptionBroadcasterMix": "NORMAL", "SampleRate": 48000, "Bitrate": 96000, "CodingMode": "CODING_MODE_2_0", "RawFormat": "NONE"}}, "LanguageCodeControl": "FOLLOW_INPUT", "AudioTypeControl": "FOLLOW_INPUT"}], "ContainerSettings": {"Mp4Settings": {"MoovPlacement": "PROGRESSIVE_DOWNLOAD", "CslgAtom": "INCLUDE", "FreeSpaceBox": "EXCLUDE"}, "Container": "MP4"}}], "CustomName": "MP4", "Name": "File Group"}, {"OutputGroupSettings": {"Type": "FILE_GROUP_SETTINGS", "FileGroupSettings": {"Destination": "s3://outgest/TRAILER/Default/Thumbnails/"}}, "Outputs": [{"VideoDescription": {"CodecSettings": {"FrameCaptureSettings": {"MaxCaptures": 500, "FramerateDenominator": 5, "FramerateNumerator": 1, "Quality": 80}, "Codec": "FRAME_CAPTURE"}, "DropFrameTimecode": "ENABLED", "Sharpness": 50, "AntiAlias": "ENABLED", "Height": 720, "Width": 1280, "ScalingBehavior": "DEFAULT", "RespondToAfd": "NONE", "ColorMetadata": "INSERT", "AfdSignaling": "NONE", "TimecodeInsertion": "DISABLED"}, "ContainerSettings": {"Container": "RAW"}}], "CustomName": "Thumbnails", "Name": "File Group"}], "AdAvailOffset": 0} /var/runtime/urllib3/connectionpool.py:986: InsecureRequestWarning: Unverified HTTPS request is being made to host 'eelovgiob.mediaconvert.sa-east-1.amazonaws.com'. Adding certificate verification is strongly advised. See: https://urllib3.readthedocs.io/en/latest/advanced-usage.html#ssl-warnings InsecureRequestWarning, END RequestId: 4deac982-95e9-4736-b83c-5d650b9c7cf4 REPORT RequestId: 4deac982-95e9-4736-b83c-5d650b9c7cf4 Duration: 1760.86 ms Billed Duration: 1800 ms Memory Size: 128 MB Max Memory Used: 96 MB

[gordonmckenzie]

Sorry you are still having problems Rodrigo. From the AWS documentation error code 1401 is a permissions problem: -

Amazon S3 denied access to a file or bucket. Check the bucket policies on your input and output locations. Also check that the job has the right IAM role specified and that the role has the necessary permissions.

I would do the following: -

1. Double check that your Lambda function has the necessary S3 PutObject permissions in its IAM
2. Check that the MediaConvert role again has the correct permissions in IAM, including S3 permission
3. Explicitly include the role of the MediaConvert in the top level of the job.json file
4. Try adding the bucket policy I included above
5. Double check your destinations in the job.json file e.g. s3://outgest/TRAILER/Default/Thumbnails/ whereas mine looks like this s3://</MEDIABUCKET/>/Thumbnails/

N.B. I have to escape MEDIABUCKET with forward slashes otherwise the markdown strips MEDIABUCKET away.

Hope this helps the problem.

[rodrigohashimoto]

thanks Gordon, tried all above except the suggestion n.3, I was not able to include the mediaconvertrole in the job.json file correclty, and consequently running the test fails with statuscode: 500. any other ideas?

[aburkleaux-amazon]

Hi,

Can you provide some more information:

1. Are you using the CloudFormation template or using the step-by-step console tutorial?
2. Have you modified the workflow in any way, including changing the job settings?

This does seem like it may be an IAM issue. There are two roles involved for the WatchFolder workflow :

MediaConvertRole - gives MediaConvert permissions needed to access your S3 buckets VODLambdaRole - gives the VODLambdaConvert Lambda permissions to call MediaConvert APIs, list the content of the input bucket and pass the MediaConvertRole to the MediaConvert service.

Please check VODLambdaRole to make sure it is granting "iam:PassRole" for the ARN for MediaConvertRole.

Alex

[rodrigohashimoto]

Hi Alex, appreciate the help.

1. I´m following the step-by-step tutorial. I only used different names for the lambda and mediaconvert roles and inline policy that I created, could this be an issue?

2. No changes in the job.json

Double checked mediaconvert and lambda roles, they both have S3FullAccess, and lambda role has also AWSLambdaBasicExecutionRole policy and an inline policy granting "iam:PassRole" for the ARN for MediaConvertRole.

rodrigo

[aburkleaux-amazon]

There is a CloudFormation template that automatically deploys the stack in the tutorial. I just ran it in us-east-1 and it is working. You could try deploying the stack and then compare between roles and other setup you did manually following the tutorial.

Instructions are here:

https://github.com/aws-samples/aws-media-services-vod-automation/blob/master/MediaConvert-WorkflowWatchFolderAndNotification/README.md

Alex

[cMurschall]

I had the exact same problem. I had blocked all access for the output bucket aka vod-mediabucket. After unsetting Block all public access" it worked.

I was planing to use cloudfront to distribute the outputs, but apparently its not needed.

[rodrigohashimoto]

Unsetting Block didn´t work.

Also deploying a CloudFormation failed with these msgs:

2020-08-28 06:01:36 UTC-0300	emc-watchfolder	ROLLBACK_IN_PROGRESS	The following resource(s) failed to create: [NotifyEventRule, MediaBucketPolicy, LambdaRole]. . Rollback requested by user.
2020-08-28 06:01:35 UTC-0300	LambdaRole	CREATE_FAILED	Resource creation cancelled
2020-08-28 06:01:35 UTC-0300	NotifyEventRule	CREATE_FAILED	Resource creation cancelled
2020-08-28 06:01:34 UTC-0300	MediaBucketPolicy	CREATE_FAILED	API: s3:PutBucketPolicy Access Denied

Any ideas?

[aburkleaux-amazon]

Hi,

If you are running this and an IAM user (rather than the account root user), make sure you have adequate permission to modify bucket policies in the S3 buckets you create. This article shows how to check that:

https://aws.amazon.com/premiumsupport/knowledge-center/s3-access-denied-bucket-policy/

Alex

[rodrigohashimoto]

Hi ALex, I tried with both user with administrator privileges and with root user too. Both had same failure in CloudFormation. :(

[aburkleaux-amazon]

Thanks for your perseverance :-)

I was able to reproduce this error by setting the S3 Block public access (account settings) to Block all public access On.

The workflow is relying on MediaConvert to set the ACLs for its outputs to Public Read so that the videos can be played back directly from the S3 Object URL. This is set using the following output attributes in the job.json file;

"DestinationSettings": {
              "S3Settings": {
                "AccessControl": {
                  "CannedAcl": "PUBLIC_READ"
                }
              }

In order for this to work, you need to configure S3 as follows:

• Block public access (account settings) to Block all public access Off
• Block public access to buckets and objects granted through new access control lists (ACLs) Off
• Block public access to buckets and objects granted through any access control lists (ACLs) Off

I will add this to the tutorial instructions. Let me know if you are able to verify it is working for you.

Alternatives:

1. Keep S3 Block public access (account settings) to Block all public access On, remove the S3Settings attributes from job.json and download the videos to play them.
2. Setup as in 1 and create a CloudFront Distribution and CloudFront origin access identity to access the MediaConvert output via HTTPS. See https://github.com/awslabs/video-on-demand-on-aws for an example of a stack using CloudFront for video playback.

[rodrigohashimoto]

thank YOU for your perseverance in helping. My "Block Public Account Settings" looks odd.

I was logged in as root, tried editing and save it but still not showing the 5 separate Block options. :(

[rodrigohashimoto]

Hi Alex, sorry, my public access (account settings) was not looking odd like I described above, it was my antivirus preventing the page from displaying properly. :/

Also, I was able to launch and run the cloudformation template and it worked fine.

I will investigate by comparing the stack created roles and permissions against my step-by-step setup to find what´s going on. Many thanks to All.

[rodrigohashimoto]

Hi Alex,

just to update the issue.

It worked by configuring the S3 Block Public Access (account settings) as you mentioned.

And alternative 1, also worked by keeping S3 Block public access (account settings) to Block all public access on and removing the S3Settings attributes from job.json.

Thank you very much Rodrigo

[aburkleaux-amazon]

Hi Rodrigo,

Thanks for the update. I'm glad it is working for you now. I will update the tutorial with the S3 configuration requirements.

Alex