need to transfer infrastructure to OpenStreetMap US

[d3netxer]

Description

The goal is to transfer the backend infrastructure to OpenStreetMap US.

[d3netxer]

Requesting to OpenStreetMap US (@iandees) to implement the following:

Create the following two buckets in US-East:

• teachosm-project-pics-deployed
• teachosm-project-content-original-deployed

These two buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the following will be entered and saved for the CORS configuration:

<?xml version="1.0" encoding="UTF-8"?>
<CORSConfiguration xmlns="http://s3.amazonaws.com/doc/2006-03-01/">
<CORSRule>
    <AllowedOrigin>*</AllowedOrigin>
    <AllowedMethod>GET</AllowedMethod>
    <AllowedMethod>PUT</AllowedMethod>
    <AllowedMethod>POST</AllowedMethod>
    <AllowedMethod>HEAD</AllowedMethod>
    <AllowedHeader>*</AllowedHeader>
</CORSRule>
</CORSConfiguration>

Also create the following two buckets in US-East:

• teachosm-project-content-deployed
• teachosm-project-posts-deployed
• teachosm-libreoffice-deployed

These three buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the default CORS configuration can be left as is.

[iandees]

I created the following buckets:

• teachosm-project-pics-deployed
• teachosm-project-content-original-deployed
• teachosm-project-content-deployed
• teachosm-project-posts-deployed
• teachosm-libreoffice-deployed

And disabled block all public access. The first two have CORS configurations added.

Remaining when we're farther along:

• [ ] Adjust AllowedOrigin on the CORS configuration so that it only loads on OSM US-hosted sites.
• [ ] Put domain and/or CDN in front of the S3 buckets and don't allow public access to the S3 buckets

[d3netxer]

I tested deploying today, but need some additional adjustments from @iandees :

1) use this advanced permissions template for reference (https://serverless-stack.com/chapters/customize-the-serverless-iam-policy.html) and for the cloudformation policies adjust the resource to this: arn:aws:cloudformation:us-east-1:509553493608:stack/imageUploadDemo-dev/*

2) create the following two new deployment buckets in us-east-1:

• convert-post-request-to-yaml-deploy

• serverlessS3Upload-deploy

These two buckets will have ‘Block all public access’ turned off, in the Access control list for the ‘everyone’ group all boxes will be unchecked, and the default CORS configuration can be left as is.

[iandees]

1. I added arn:aws:cloudformation:us-east-1:509553493608:stack/imageUploadDemo-dev/* to the list of cloudformation resources in the existing policy on your account.
2. Added S3 bucket convert-post-request-to-yaml-deploy. Bucket serverlessS3Upload-deploy is not possible because bucket names can only have lower case letters.

[d3netxer]

ok, for item 2, please create this bucket instead: teachosm-serverless-s3upload-deploy

Also, I am having trouble with the permissions for the convert-post-request-to-yaml-deploy bucket. The Serverless Error is saying: "Could not locate deployment bucket. Error: Access Denied".

I will get with you offline about this.

[iandees]

As a reminder, your account has access to do anything to buckets with the prefix teachosm-geosurge-. You should be able to create buckets, list them, etc. as long as they have that prefix.

In the meantime, I created the teachosm-serverless-s3upload-deploy bucket and added convert-post-request-to-yaml-deploy to the list of buckets you can write to.

[d3netxer]

Thanks, noted about access using the prefix.

I just keep on moving forward using the teachosm-serverless-s3upload-deploy and "convert-post-request-to-yaml-deploy" buckets.

I ran the deploy command again and I was allowed to move forward. However, a new error now appears that says my user is not allowed to perform apigateway:PATCH on resource.

I think this can be resolved with another edit to the permissions. In the existing permissions, there should be a block like this:

{
      "Effect": "Allow",
      "Action": [
        "apigateway:GET",
        "apigateway:POST",
        "apigateway:PUT",
        "apigateway:DELETE"
      ],
      "Resource": [
        "arn:aws:apigateway:<region>::/restapis"
      ]
    },
    {
      "Effect": "Allow",
      "Action": [
        "apigateway:GET",
        "apigateway:POST",
        "apigateway:PUT",
        "apigateway:DELETE"
      ],
      "Resource": [
        "arn:aws:apigateway:<region>::/restapis/*"
      ]
    },

Please make sure that the is 'us-east-1'. Also add the "apigateway:PATCH" to the Actions.

[iandees]

I added PATCH to the list of Actions.

[d3netxer]

Thanks, the convert-post-request-to-yaml lambda function deployed successfully.

I am now getting a new error for the serverlessS3Upload function related to permissions for the cloudformation:DescribeStacks. Can you make adjustments to the permissions to the block that looks like this:

{
      "Effect": "Allow",
      "Action": [
        "cloudformation:Describe*",
        "cloudformation:List*",
        "cloudformation:Get*",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack",
        "cloudformation:DeleteStack"
      ],
      "Resource": "arn:aws:cloudformation:<region>:<account_no>:stack/<service_name>*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:ValidateTemplate"
      ],
      "Resource": "*"
    },

I'm assuming the correct region and account_no are already inserted here. For the service name, you can input 'image-upload-prod'.

[iandees]

image-upload-prod is now in the CloudFormation list.

[d3netxer]

Thanks, it got past this step now.

The next permissions error is related to an Error for IamRoleLambdaExecution - API: iam:GetRole on on resource: role image-upload-prod-us-east-1-lambdaRole. Can you modify this block of the permissions:

{
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:DetachRolePolicy",
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy",
        "iam:DeleteRolePolicy"
      ],
      "Resource": [
        "arn:aws:iam::<account_no>:role/<service_name>*-lambdaRole"
      ]
    },

and insert 'image-upload-prod' for the ?

[iandees]

Done!

[d3netxer]

Excellent. This worked, so these 2 lambdas are deployed: serverlessS3Upload and convert-post-request-to-yaml. There is a final one libreoffice-convert-to-pdf that I need to deploy, but I don't have a serverless script for this. My plan is begin trying next week to deploy using AWS CLI.

[d3netxer]

I have started trying to deploy the final function, using Serverless. Similar to the previous function, I got a cloudformation:DescribeStacks error. Can you make an adjustment in this block:

{
      "Effect": "Allow",
      "Action": [
        "cloudformation:Describe*",
        "cloudformation:List*",
        "cloudformation:Get*",
        "cloudformation:CreateStack",
        "cloudformation:UpdateStack",
        "cloudformation:DeleteStack"
      ],
      "Resource": "arn:aws:cloudformation:<region>:<account_no>:stack/<service_name>*/*"
    },
    {
      "Effect": "Allow",
      "Action": [
        "cloudformation:ValidateTemplate"
      ],
      "Resource": "*"
    },

You can add another resource by adding a comma followed by a second item. In the second item can you add 'geosurge-libreoffice-convert-to-pdf*' for the ?

Also, let's modify the AWS role permissions as well. Looking at this block:

{
      "Effect": "Allow",
      "Action": [
        "iam:GetRole",
        "iam:PassRole",
        "iam:CreateRole",
        "iam:DeleteRole",
        "iam:DetachRolePolicy",
        "iam:PutRolePolicy",
        "iam:AttachRolePolicy",
        "iam:DeleteRolePolicy"
      ],
      "Resource": [
        "arn:aws:iam::<account_no>:role/<service_name>*-lambdaRole"
      ]
    },

Can you add another resource item and again add 'geosurge-libreoffice-convert-to-pdf*' for the ?

Thanks

[iandees]

Done!

[d3netxer]

I'm getting this error, maybe to do with formatting. This link may help: https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements_resource.html

Error --------------------------------------------------

  The CloudFormation template is invalid: Template format error: At least one Resources member must be defined.

[d3netxer]

@iandees I spent sometime making improvements and testing a whole re-deployment of all the lambdas on another AWS account. I have a new IAM policy that I hope will take care of all the permission issues here: https://gist.github.com/d3netxer/b1d1a4012d6bf20b910c22d02ee43a80

Can you try replacing the policy with this one? You just need to replace the and wherever needed. Thanks

[iandees]

Change is made. It looks like you've been deploying to us-east-1, so I replaced <region> with that, too.

[d3netxer]

Excellent. All of the functions deployed successfully. A few modifications are left to complete the deployment on the live site.

One of the Lambda functions makes a pull request to the Github repo. For this I believe we need to post as a user who has admin access to the repo, along with a personal access token. @iandees you can either create a new Github user or use an existing one. The pull requests will be posted as that user.

• that user will need to have access to the repo: https://github.com/osmus/teachosm.org
• create a personal access token for the user (Go to settings --> developer settings --> personal access tokens, and create one with the repo and admin: repo_hook scope boxes checked)
• pass along the username and personal access token to me privately

[iandees]

This is done. The user is teachosm-bot. I shared the access token privately.

[d3netxer]

thanks. I am getting close, today I switched out the backends, including copying the files between s3 buckets. I need to look at the libreoffice converter a little bit more because it isn't converting pdfs to pdfs. It probably shouldn't do this, if the file doesn't need to be converted, it should just be copied from the contents upload folder to the contents folder instead; so adding an if statement should fix this issue most likely. Right now 2 project download links are broken because of this. I will try to continue fixing this and doing more tests within the next few days.