Question: What is a proper setting for rds serverless v2 public access for a copilot storage generated database

[phasetr]

I can create an rds serverless PostreSQL database in AWS console which can access publicly, but I do not understand a setting for aws copilot generated one.

I know a property PubliclyAccesible in CloudFormation spec, but I'd like to understand the security setting for various environments. In particular, I set publicly access setting open for test or staging environments, but the production environment setting more secure.

Thanks in advance.

[huanjani]

Hi @phasetr! You can use Mappings or Conditions! See https://aws.github.io/copilot-cli/docs/developing/addons/environment/#writing-the-conditions-and-the-mappings-sections. After you run copilot storage init and a yaml file is generated for you, tweak that file:

Mappings:
  PublicAccessibilityMap:
    test: 
      "publiclyAccessible": true
    prod: 
      "publiclyAccessible": false
[name]DBCluster:
  Metadata:
    'aws:copilot:description': 'The testing Aurora Serverless v2 database cluster'
  Type: 'AWS::RDS::DBCluster'
  Properties:
    PubliclyAccessible: !FindInMap [PublicAccessibilityMap, !Ref Env, publiclyAccessible]

...something along those lines! I hope that answers your questions!

[phasetr]

@huanjani Thank you for your comment. I'll try it!

[phasetr]

@huanjani I tried, but I have the following error:

- Updating the infrastructure for stack aspdotnet-staging-web                          [update rollback complete]  [91.7s]
  The following resource(s) failed to update: [AddonsStack].
  - An Addons CloudFormation Stack for your additional AWS resources                   [update rollback complete]  [79.6s]
    The following resource(s) failed to update: [webclusterDBCluster].
    - The webcluster Aurora Serverless v2 database cluster                             [delete complete]           [4.3s]
      Resource handler returned message: "PubliclyAccessible isn't supported
       for DB engine aurora-postgresql. (Service: Rds, Status Code: 400, Req
      uest ID: ed830f0f-c125-4d86-b78e-5b1be1881a22)" (RequestToken: c509d9d
      0-d62d-9ffe-4776-419e03b89505, HandlerErrorCode: InvalidRequest)
    - The webcluster Aurora Serverless v2 writer instance                              [update complete]           [2.3s]
  - An IAM role to control permissions for the containers in your service              [not started]
  - An App Runner service to run and manage your containers                            [not started]
✘ deploy service web to environment staging: deploy service: stack aspdotnet-staging-web did not complete successfully and exited with status UPDATE_ROLLBACK_COMPLETE

My full sample is here. This message means Aurora serverless v2 (Aurora-Postgresql?) cannot access publicly? However I can acceess publicly it from the AWS console setting. (I am an AWS newbie, so I still do not understand thoroughly the behavior and setting.) What is a proper setting?

[huanjani]

Hi, @phasetr.

Ahhh, yes, I just found this: https://docs.aws.amazon.com/AmazonRDS/latest/AuroraUserGuide/Aurora.CreateInstance.html#Aurora.CreateDBCluster.SettingsNotApplicableDBClusters; apparently that property can't be applied to Aurora clusters. 😤

I think you'll have to configure the security group to accept traffic over the internet.

In terms of varying accessibility among environments, you could use Conditions/Mappings in your addons template again, or you could have the associated workload placed in private/public subnets with the

network:
  vpc:
    placement:

field in your workload manifest, which can differ from env to env using the environments field.

I see that you're using a Request-Driven Web Service, which also has the https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#http-private field, but I don't think you want the service itself to be private, right?

[phasetr]

@huanjani Thank you!

I see that you're using a Request-Driven Web Service, which also has the https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#http-private field, but I don't think you want the service itself to be private, right?

Yes, I am going to create an public service.

I'm not entirely sure, but I remember seeing a message that I couldn't initialize it without the network setting when I run copilot storage init. Can I delete this setting?

[huanjani]

Yes, you can remove the whole network field (https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#network) in which case the Request-Driven Web Service won't be connected to a VPC. So this will work if your database is public.

[phasetr]

Hi, @huanjani.

I'm struggling and restart the setting. I have the follwoing error.

✘ execute svc deploy: deploy service web to environment staging: deploy service: check if changeset is empty: create change set copilot-f906daf0-6e57-4a1b-b725-53944a7b8ae6 for stack aspdotnet-staging-web: ValidationError: Template error: instance of Fn::GetAtt references undefined resource ServiceSecurityGroup status code: 400, request id: a31ddca8-058f-4d92-b67f-eb89ebfb9a2b: describe change set copilot-f906daf0-6e57-4a1b-b725-53944a7b8ae6 for stack aspdotnet-staging-web: ChangeSetNotFound: ChangeSet [copilot-f906daf0-6e57-4a1b-b725-53944a7b8ae6] does not exist status code: 404, request id: f17eb96b-f092-4ed4-b16f-03a82e038d0b

First I run the commands copilot app init, copilot svc init, copilot env init, and then copilot deploy. The service starts. Then I run the command copilot storage init and copilot deploy, then I have the above error message.

Here is a temporary setting, in particular I commented out the http setting in environments/staging/manifest.yml. The copilot setting files is created by copilot and I do not change them.

[huanjani]

Ah, so sorry @phasetr-- I misread your message above:

I'm not entirely sure, but I remember seeing a message that I couldn't initialize it without the network setting when I run copilot storage init. Can I delete this setting?

And you were right. In order to add your Aurora cluster, you will need VPC access. So uncomment the network.vpc.placement: private part of your RDWS manifest (https://aws.github.io/copilot-cli/docs/manifest/rd-web-service/#network-vpc-placement). This will add a NAT gateway in order for your service to have internet connectivity (and NAT gateways can be pricey). You can weight the costs of using an App Runner/RDWS workload with NAT Gateway vs. a Load Balanced Web Service without NAT Gateway.

But I do think the way to configure the access to your DB is through the security group. But just to confirm-- when you say you want the DB to be publicly accessible, do you mean open to the internet or just accessible by your workload/service? If the latter, then you will have that once they're in the same VPC.

[phasetr]

Thank you, @huanjani.

But I do think the way to configure the access to your DB is through the security group. But just to confirm-- when you say you want the DB to be publicly accessible, do you mean open to the internet or just accessible by your workload/service? If the latter, then you will have that once they're in the same VPC.

I mean open to the internet. My intention is to access the DB in the test/development (or staging) environments for simplicity, in particular, by local GUI tools. Of course I know it is not secure.

I hear that, in general, I should access the DBs by some intermediate EC2 for security. However I do not understand the setting for copilot (or other means, CDK or CloudFormation). So my reluctant second choice is open access to the DBs, since I need the developing/staging envirionments as soon as possible.

Here is another, some related question: If I choose the load balanced service, I can access ap containers using copilot svc exec, I think. Can I manage the DBs, e.g., migrations or some maintainance for development by copilot svc exec? My App Runner choice is not obligatory, so I can switch to the load balanced service.

[huanjani]

Hi, @phasetr.

We are now working on creating a publicly accessible Aurora db of our own, in order to confirm how you could do it. I will keep you posted on that.

In the meantime, yes-- you can use svc exec with LBWSs. But for migrating/managing the DB, exec may not be very helpful; was your job (#4579) not a good solution for you?

[phasetr]

Thank you, @huanjani.

But for migrating/managing the DB, exec may not be very helpful; was your job (#4579) not a good solution for you?

I also had in mind the fine-tuning of table items without an administration page. Hence copilot job is a little bit unhelpful for some tasks. Anyway I will manage by working hard.

I'll choose closed db setting for a while and I'm looking forward to a new feature! Thank you again @huanjani.

[huanjani]

One more thought: You can do some storage work with exec--

https://aws.github.io/copilot-cli/docs/developing/storage/#using-copilot-svc-exec

[github-actions[bot]]

This issue is stale because it has been open 60 days with no response activity, and is tagged with pending/question. Remove the stale label, add a comment, or this will be closed in 14 days.

[github-actions[bot]]

This issue is closed due to inactivity. Feel free to reopen the issue if you have any follow-ups!