search

awslabs / amplify-video

An open source Category Plugin for the AWS Amplify-CLI that makes it easy to deploy live and file based streaming video services and integrate them into your Amplify applications.
https://www.npmjs.com/package/amplify-category-video
Apache License 2.0
267 stars 56 forks source link

S3 endpoint issues / Access denied errors - IAM with full access #59

Closed bijanmmarkes closed 4 years ago

bijanmmarkes commented 4 years ago

Describe the bug I've tried all combinations of amplify add video ranging from yes/no for create a distribution, autostart: yes/no, all types of methods. In the end, all combinations of enabling/disabling options end with error:

CREATE_IN_PROGRESS rMP AWS::CloudFormation::Stack Wed Dec 18 2019 03:56:10 GMT-0800 (Pacific Standard Time)
CREATE_FAILED      rMP AWS::CloudFormation::Stack Wed Dec 18 2019 03:56:11 GMT-0800 (Pacific Standard Time) S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html
⠋ Updating resources in the cloud. This may take a few minutes...

CREATE_FAILED mylivestream-prd-20191218035347-videomylivestream-CB2G4388G54G AWS::CloudFormation::Stack Wed Dec 18 2019 03:56:12 GMT-0800 (Pacific Standard Time) The following resource(s) failed to create: [rMP].
⠇ Updating resources in the cloud. This may take a few minutes...

CREATE_FAILED               videomylivestream           AWS::CloudFormation::Stack Wed Dec 18 2019 03:56:17 GMT-0800 (Pacific Standard Time) Embedded stack arn:aws:cloudformation:us-west-2:541764378143:stack/mylivestream-prd-20191218035347-videomylivestream-CB2G4388G54G/5f64ac40-218d-11ea-98e0-0acc43cbe642 was not successfully created: The following resource(s) failed to create: [rMP].

When I add create distribution I receive even more errors, which I'm assuming are all linked to the first failure of S3: S3 error: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint.

To Reproduce Steps to reproduce the behavior:

  1. amplify init
  2. amplify add video
  3. amplify push
  4. See error

Expected behavior I expect it to push resources to the cloud, what am I missing? I followed exact instructions from the README.MD.

Screenshots If applicable, add screenshots to help explain your problem.

Desktop (please complete the following information):

Additional context Template file:

{
  "AWSTemplateFormatVersion": "2010-09-09",

  "Description": "mylivestream",

  "Metadata": {
    "AWS::CloudFormation::Interface": {
      "ParameterGroups": [
        {
          "Label": { "default": "Provision: Source Files Configuration" },
          "Parameters": [
            "pS3",
            "pSourceFolder",
            "pPackageName"
          ]
        },
        {
          "Label": { "default": "Shared Settings: Encoding and Packaging Configuration" },
          "Parameters": [
            "pGopSizeInSec",
            "pGopPerSegment",
            "pSegmentPerPlaylist"
          ]
        },
        {
          "Label": { "default": "MediaLive: Channel Configuration" },
          "Parameters": [
            "pInputSecurityGroup",
            "pIngestType",
            "pEncodingProfile",
            "pGopSizeInSec",
            "pGopPerSegment",
            "pStartChannel"
          ]
        },
        {
          "Label": { "default": "MediaPackage: Channel Configuration" },
          "Parameters": [
            "pEndpoints",
            "pStartOverWindow"
          ]
        },
        {
          "Label": { "default": "MediaStore: Container Configuration" },
          "Parameters": [
            "pEnableMediaStore"
          ]
        },
        {
          "Label": { "default": "CloudFront: Distribution Configuration" },
          "Parameters": [
            "pEnableDistribution",
            "pPriceClass",
            "pS3CF",
            "pS3PrefixCF"
          ]
        }
      ],
      "ParameterLabels": {
        "env":{
          "default": "Env variable for Amplify"
        },
        "pProjectName": {
          "default": "Your project name"
        },
        "pS3": {
          "default": "S3 Bucket Name"
        },

        "pSourceFolder": {
          "default": "Source Folder"
        },

        "pPackageName": {
          "default": "Lambda Package Name"
        },

        "pGopSizeInSec": {
          "default": "GOP Size"
        },

        "pGopPerSegment": {
          "default": "GOP(s) Per Segment"
        },

        "pSegmentPerPlaylist": {
          "default": "Segment(s) Per Playlist"
        },

        "pInputSecurityGroup": {
          "default": "Input Security Group"
        },

        "pIngestType": {
          "default": "Ingest Type"
        },

        "pEncodingProfile": {
          "default": "Encoding Profile"
        },

        "pStartChannel": {
          "default": "Auto-start"
        },

        "pEndpoints": {
          "default": "Packaging Type(s)"
        },

        "pStartOverWindow": {
          "default": "Content Window (in seconds)"
        },

        "pEnableMediaStore": {
          "default": "Enable MediaStore?"
        },

        "pEnableDistribution": {
          "default": "Create distribution?"
        },

        "pPriceClass": {
          "default": "Price Class"
        },

        "pS3CF": {
          "default": "S3 Bucket for logs"
        },

        "pS3PrefixCF": {
          "default": "S3 Prefix (Folder) for logs"
        }
      }
    }
  },

  "Resources": {

    "rMP": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "TemplateURL": {
          "Fn::Sub": "https://s3.amazonaws.com/${pS3}/${pSourceFolder}/mediapackage.template"
        },
        "Parameters": {
          "pS3": {
            "Ref": "pS3"
          },

          "pSourceFolder": {
            "Ref": "pSourceFolder"
          },

          "pPackageName": {
            "Ref": "pPackageName"
          },

          "pProvisionLambdaHandler": "orchestration.MediaPackageChannel",

          "pChannelId": {
            "Fn::If": [
              "HasEnvironmentParameter",
              {
                "Fn::Join": [
                  "-",
                  [
                    {
                      "Ref": "pProjectName"
                    },
                    {
                      "Ref": "env"
                    }
                  ]
                ]
              },
              {
                "Ref": "pProjectName"
              }
            ]
          },

          "pIngestType": "hls",

          "pEndpoints": {
            "Fn::Join": [ ",", { "Ref": "pEndpoints" } ]
          },

          "pStartOverWindow": {
            "Ref": "pStartOverWindow"
          },

          "pGopSizeInSec": {
            "Ref": "pGopSizeInSec"
          },

          "pGopPerSegment": {
            "Ref": "pGopPerSegment"
          },

          "pSegmentPerPlaylist": {
            "Ref": "pSegmentPerPlaylist"
          }
        }
      }
    },

    "rML": {
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "TemplateURL": {
          "Fn::Sub": "https://s3.amazonaws.com/${pS3}/${pSourceFolder}/medialive.template"
        },
        "Parameters": {
          "pS3": {
            "Ref": "pS3"
          },

          "pSourceFolder": {
            "Ref": "pSourceFolder"
          },

          "pPackageName": {
            "Ref": "pPackageName"
          },

          "pProvisionLambdaHandler": "orchestration.MediaLiveChannel",

          "pChannelId": {
            "Fn::If": [
              "HasEnvironmentParameter",
              {
                "Fn::Join": [
                  "-",
                  [
                    {
                      "Ref": "pProjectName"
                    },
                    {
                      "Ref": "env"
                    }
                  ]
                ]
              },
              {
                "Ref": "pProjectName"
              }
            ]
          },

          "pInputSecurityGroup": {
            "Fn::Join": [ ",", { "Ref": "pInputSecurityGroup" } ]
          },

          "pIngestType": {
            "Ref": "pIngestType"
          },

          "pEncodingProfile": {
            "Ref": "pEncodingProfile"
          },

          "pGopSizeInSec": {
            "Ref": "pGopSizeInSec"
          },

          "pGopPerSegment": {
            "Ref": "pGopPerSegment"
          },

          "pSegmentPerPlaylist": {
            "Ref": "pSegmentPerPlaylist"
          },

          "pStartChannel": {
            "Ref": "pStartChannel"
          },

          "pEndpointUrls": {
            "Fn::GetAtt": [ "rMP", "Outputs.oIngestUrls" ]
          },

          "pUsernames": {
            "Fn::GetAtt": [ "rMP", "Outputs.oUsers" ]
          },

          "pParameterStoreKeys": {
            "Fn::GetAtt": [ "rMP", "Outputs.oParameterStoreKeys" ]
          },

          "pMediaStoreEndpoint": {
            "Fn::If": [
              "cEnableMediaStore",
              { "Fn::GetAtt": [ "rMS", "Outputs.oContainerEndpoint" ] },
              ""
            ]
          }
        }
      }
    },

    "rMS": {
      "Condition": "cEnableMediaStore",
      "Type": "AWS::CloudFormation::Stack",
      "Properties": {
        "TemplateURL": {
          "Fn::Sub": "https://s3.amazonaws.com/${pS3}/${pSourceFolder}/mediastore.template"
        },
        "Parameters": {
          "pS3": {
            "Ref": "pS3"
          },

          "pSourceFolder": {
            "Ref": "pSourceFolder"
          },

          "pPackageName": {
            "Ref": "pPackageName"
          },

          "pProvisionLambdaHandler": "orchestration.MediaStoreContainer",

          "pContainerName": {
            "Fn::If": [
              "HasEnvironmentParameter",
              {
                "Fn::Join": [
                  "-",
                  [
                    {
                      "Ref": "pProjectName"
                    },
                    {
                      "Ref": "env"
                    }
                  ]
                ]
              },
              {
                "Ref": "pProjectName"
              }
            ]
          }
        }
      }
    }
  },

  "Parameters": {
    "env": {
      "Type": "String",
      "Description": "The environment name. e.g. Dev, Test, or Production",
      "Default": "NONE"
    },
    "pProjectName": {
      "Type": "String",
      "Description": "store template and lambda package",
      "Default": "mylivestream"
    },
    "pS3": {
      "Type": "String",
      "Description": "store template and lambda package",
      "AllowedPattern" : "[a-zA-Z][a-zA-Z0-9-_]*",
      "Default": "mylivestream-prd-20191218035347-deployment"
    },

    "pSourceFolder": {
      "Type": "String",
      "Description": "store template and lambda package",
      "Default": "livestream-helpers"
    },

    "pPackageName": {
      "Type": "String",
      "Description": "lambda package zip file",
      "Default": "psdemo-js-live-workflow_v0.4.0.zip"
    },

    "pInputSecurityGroup": {
      "Type": "CommaDelimitedList",
      "Description": "comma delimited CIDR list. ie., 10.1.0.0/16,10.2.0.0/16",
      "Default": "0.0.0.0/0"
    },

    "pIngestType": {
      "Type": "String",
      "Description": "medialive ingest type",
      "AllowedValues": [ "RTP_PUSH", "UDP_PUSH", "RTMP_PUSH" ],
      "Default": "RTMP_PUSH"
    },

    "pEncodingProfile": {
      "Type": "String",
      "Description": "FULL (6 renditions), MOBILE (3 renditions), HD (single 1080p), SD (single 432p)",
      "Default": "MOBILE",
      "AllowedValues": [ "FULL", "MOBILE", "HD", "SD" ]
    },

    "pGopSizeInSec": {
      "Type": "Number",
      "Description": "specify GOP size in seconds. Use 1s to force IP-frame encode for low-latency.",
      "Default": "1",
      "MinValue": "1"
    },

    "pGopPerSegment": {
      "Type": "Number",
      "Description": "specify number of GOPs per segment. Use 1 for low-latency.",
      "Default": "2",
      "MinValue": "1"
    },

    "pStartChannel": {
      "Type": "String",
      "Description": "start the channel upon creation",
      "Default": "NO",
      "AllowedValues": [ "YES", "NO" ]
    },

    "pEndpoints": {
      "Type": "CommaDelimitedList",
      "Description": "output streaming standards. MediaPackage supports HLS, DASH, MSS, and/or CMAF",
      "Default": "CMAF",
      "AllowedValues": [
        "HLS", "HLS,DASH", "HLS,MSS", "HLS,DASH,MSS", "HLS,DASH,MSS,CMAF",
        "DASH", "DASH,MSS", "DASH,MSS,CMAF",
        "MSS", "MSS,CMAF",
        "CMAF"
      ]
    },

    "pStartOverWindow": {
      "Type": "Number",
      "Description": "specify catch-up TV window, max. 3 days (259200)",
      "Default": "86400",
      "MinValue": "60",
      "MaxValue": "259200"
    },

    "pSegmentPerPlaylist": {
      "Type": "Number",
      "Description": "specify number of segments per playlist/manifest, minimum 1 (recommended 3 and above)",
      "Default": "3",
      "MinValue": "1"
    },

    "pEnableMediaStore": {
      "Type": "String",
      "Description": "Caution: By enabling MediaStore, it clones the MediaLive output channels that will ingest to MediaStore origin.",
      "Default": "NO",
      "AllowedValues": [ "YES", "NO" ]
    },

    "pEnableDistribution": {
      "Type": "String",
      "Description": "",
      "Default": "NO",
      "AllowedValues": [ "YES", "NO" ]
    },

    "pPriceClass": {
      "Type": "String",
      "Description": "see http://docs.aws.amazon.com/AmazonCloudFront/latest/DeveloperGuide/PriceClass.html for details",
      "AllowedValues": [ "PriceClass_100", "PriceClass_200", "PriceClass_All" ],
      "Default": "PriceClass_100"
    },

    "pS3CF": {
      "Type": "String",
      "Description": "specify where CloudFront will deliver access logs to. Leave it blank to use the same bucket of this workflow.",
      "Default": ""
    },

    "pS3PrefixCF": {
      "Type": "String",
      "Description": "specify the prefix (folder) where CloudFront will deliver access logs to. Leave it as is.",
      "Default": ""
    }
  },

  "Conditions": {
    "cS3CF": {
      "Fn::Not": [
        {
          "Fn::Equals": [ { "Ref": "pS3CF" }, "" ]
        }
      ]
    },
    "cEnableMediaStore": {
      "Fn::Equals": [ { "Ref": "pEnableMediaStore" }, "YES" ]
    },

    "cEnableDistribution": {
      "Fn::Equals": [ { "Ref": "pEnableDistribution" }, "YES" ]
    },
    "HasEnvironmentParameter": {
      "Fn::Not": [
          {
              "Fn::Equals": [
                  {
                      "Ref": "env"
                  },
                  "NONE"
              ]
          }
      ]
    }
  },

  "Outputs": {
    "oMediaLiveChannelName": {
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oChannelName" ] },
      "Description": "MediaLive Channel Name"
    },

    "oMediaLiveChannelId": {
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oChannelId" ] },
      "Description": "MediaLive Channel Id"
    },

    "oMediaLivePrimaryIngestUrl": {
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oPrimaryIngestUrl" ] },
      "Description": "(P) MediaLive Ingest Url"
    },

    "oMediaLiveBackupIngestUrl": {
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oBackupIngestUrl" ] },
      "Description": "(B) MediaLive Ingest Url"
    },

    "oPrimaryHlsEgress": {
      "Value": {

          "Fn::GetAtt": [ "rMP", "Outputs.oHlsEndpoint" ]

      },
      "Description": "(P) HLS Egress Url"
    },

    "oPrimaryDashEgress": {
      "Value": {

          "Fn::GetAtt": [ "rMP", "Outputs.oDashEndpoint" ]

      },
      "Description": "(P) DASH Egress Url"
    },

    "oPrimaryMssEgress": {
      "Value": {

          "Fn::GetAtt": [ "rMP", "Outputs.oMssEndpoint" ]

      },
      "Description": "(P) MSS Egress Url"
    },

    "oPrimaryCmafEgress": {
      "Value": {

          "Fn::GetAtt": [ "rMP", "Outputs.oCmafEndpoint" ]

      },
      "Description": "(P) CMAF Egress Url"
    },

    "oMediaStoreContainerName": {
      "Condition": "cEnableMediaStore",
      "Value": { "Fn::GetAtt": [ "rMS", "Outputs.oContainerName" ] },
      "Description": "MediaStore Container Name"
    },

    "oMediaStoreContainerArn": {
      "Condition": "cEnableMediaStore",
      "Value": { "Fn::GetAtt": [ "rMS", "Outputs.oContainerArn" ] },
      "Description": "MediaStore Container Arn"
    },

    "oPrimaryMediaStoreEgressUrl": {
      "Condition": "cEnableMediaStore",
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oPrimaryMediaStoreEgressUrl" ] },
      "Description": "(P) MediaStore Egress Url"
    },

    "oBackupMediaStoreEgressUrl": {
      "Condition": "cEnableMediaStore",
      "Value": { "Fn::GetAtt": [ "rML", "Outputs.oBackupMediaStoreEgressUrl" ] },
      "Description": "(B) MediaStore Egress Url"
    }
  }
}
bijanmmarkes commented 4 years ago

I've resolved the above issue by updating the s3 endpoint format to the following:

"TemplateURL": {
   "Fn::Sub": "https://${pS3}.s3-us-west-2.amazonaws.com/${pSourceFolder}/mediastore.template"
}

It seems the previous endpoint format may have been deprecated, perhaps it should be updated with a PR. @wizage

But now I'm getting this error: 

CREATE_FAILED   rMP  AWS::CloudFormation::Stack Wed Dec 18 2019 04:36:15 GMT-0800 (Pacific Standard Time) S3 error: Access Denied For more information check http://docs.aws.amazon.com/AmazonS3/latest/API/ErrorResponses.html

Is this package still being developed or should it be working? I'm confused. Shouldn't amplify/the video plugin take care of all of this stuff?

The AWS profile I'm using has administrator and S3 administrator access.

wizage commented 4 years ago

This shouldn't be issue as we are following the same methodology use across the amplify CLI:

https://github.com/aws-amplify/amplify-cli/blob/54d4d64e03dc246e42ba3e2d19b1789d2dbeaddc/packages/amplify-provider-awscloudformation/lib/push-resources.js#L368

This seems to be like a fluke that has happened with other resources that have been created using the CLI: https://github.com/aws-amplify/amplify-cli/issues/2732 https://github.com/aws-amplify/amplify-cli/issues/2619

Can you try to run amplify delete and then try to create a video resource again.

bijanmmarkes commented 4 years ago

Thanks, @wizage, that seems to have worked, but I don't see how... I believe that depending on the options chosen during the video add, it throws errors.

From the looks of it, this is not a mobile SDK? I don't see a way to incorporate this code into my project, it seems like all the streaming occurs from the CLI?

wizage commented 4 years ago

It is an error with how S3 propagates. The regional S3 is available immediately but the non regional one (the first error), takes up to 24 hours iirc.

As far as SDK that is correct. This is not at all a SDK or library. We are working on creating library components for livestreaming and video player for it later. For now though it just creates the entire backend and you have to write your own code to integrate with the outputs.

bijanmmarkes commented 4 years ago

Thank you for all your help @wizage. Looking forward to the future progress of this. Glad someone is working on it. 👍 I guess I'll be using WebDav and custom code for now.