Feedback: Multi-environments and Team workflows feature

[kaustavghosh06]

Summary

A new feature of the Amplify CLI has been released support multiple environments and team workflows. This feature was developed based on feedback from customers and project contributors. The feature is in BETA and will be merged into the main CLI after testing, validation, and feedback.

Details

Based on all of your feedback on the RFC – CI/CD workflows and Multiple Environments as a part of Amplify CLI GitHub issue, we’ve released a beta version of the CLI (npm install -g @aws-amplify/cli@multienv) to support multiple environments, team workflows and integrate with your CI/CD flows.

The new CLI has all of the same functionality as the previous versions but also supports a Git style workflow for switching between environments, teams, and even sharing projects. The internal structure of the CLI metadata has changed slightly though and a one-time migration of existing projects is necessary. We will soon be releasing an automatic migration feature into the build as well as a guide for manually migrating projects. Once the features are seen as well tested and stable, as well as the migration features we will merge into the main CLI. The community testing, feedback, and validation will help with this process.

When the automatic migration feature will be added, we still recommend backing up your existing local project before testing.

Please check out the detailed documentation for more information:

https://aws-amplify.github.io/docs/cli/multienv?sdk=js https://aws-amplify.github.io/docs/cli/multienv?sdk=ios https://aws-amplify.github.io/docs/cli/multienv?sdk=android

Also, as a part of our feedback mechanism, we’ve opened up this GitHub issue, to track feedback from everyone, iterate on this beta version and then merge it to master. Please leave any feedback on issues, bugs, or even successes so we can gather the feedback.

[abdelhakim-atmani]

First of all: thanks for the great job that you are doing.

I started to use the new cli on an existing project. I removed all the resources from the cloud and create a new environment by following the steps described in https://aws-amplify.github.io/docs/cli/multienv?sdk=js. On the init step the cli was hanging on the message "Successfully created initial AWS cloud resources for deployments.". Only the deployment bucket + Auth/Unauth roles have been created. I had to kill the amplify init profile to quit the execution. And unfortunately when I do a amplify status it displayed that there is no resources. I cannot see the resources I defined before the migration.

Current Environment: dev

| Category | Resource name | Operation | Provider plugin |
| -------- | ------------- | --------- | --------------- |

where before the migration I had the following:

Current Environment: undefined

| Category | Resource name   | Operation | Provider plugin   |
| -------- | --------------- | --------- | ----------------- |
| Auth     | cognito17950874 | Create    | awscloudformation |
| Function | sensors         | Create    | awscloudformation |
| Function | gateways        | Create    | awscloudformation |
| Function | sensorData      | Create    | awscloudformation |
| Api      | sensors         | Create    | awscloudformation |
| Api      | gateways        | Create    | awscloudformation |
| Api      | sensorData      | Create    | awscloudformation |
| Storage  | sensors         | Create    | awscloudformation |
| Storage  | gateways        | Create    | awscloudformation |
| Storage  | sensorData      | Create    | awscloudformation |

[kaustavghosh06]

@abdelhakim-atmani Thanks for the feedback. Which version of the CLI are you using?

[abdelhakim-atmani]

When I run amplify version it returns 0.2.1-multienv.2

[kaustavghosh06]

@abdelhakim-atmani And are you migrating your project (intitialized using an older version of the CLI), to a newer one? Can you please mention the list of steps so that we can reproduce this issue? Also, did the CLI ask you for an auto-migration?

[abdelhakim-atmani]

I created the project with the old version of the cli. I removed all the resources from AWS using the console. I updated the cli version to the new one I created a new branch git checkout dev Then I initialised the project amplify init Where it asked me if I want to migrate the project to the new version (I said yes) Migration done successfully It asked me if I want to use an existing environment (I said no) I specified the name of the new environment I choose my profile After that it created the deployment bucket and the Auth/Unauth roles.

And now when I run amplify status the list of resources is empty.

[abdelhakim-atmani]

And I have another question regarding the new cli. In the documentation you mention a command amplify env add to add an external CFN. Is there any example on how to use it and what is it supposed to do ? I did not find any documentation.

[kaustavghosh06]

@abdelhakim-atmani The Current Environment: undefined step is weird. When you migrate, it should have created an env called NONE -> on an amplify migrate command. Also, I don't think the command was hanging, but it was initializing a new environment with your resources. But since you killed the process, it wasn't able to initialize your environment correctly.

[abdelhakim-atmani]

The undefined step is because the first time I did not accept the auto migration. Once I did the migration it was showing NONE. Regarding the command which was hanging, I did not notice any movement from the CF console or on my laptop. I will re-run it and wait.

[kaustavghosh06]

@abdelhakim-atmani Let me know your console logs if you error out or hang this time. Thanks

[abdelhakim-atmani]

Great !!! It took more than 10 minutes but finally my environment is setup. During these 10 minutes I had only the message "Successfully created initial AWS cloud resources for deployments." displayed. I had no information about what is going on. Some logs on the console would be very cool. Thanks for your help.

Can I have more precision about the command amplify env add ?

[kaustavghosh06]

@abdelhakim-atmani Did it take you 10 minutes to just initialize the environment or to push it? For me an avergae it takes less than 5 seconds to just initialize a new environment (excluding the pushing of all the resources) with all the categories enabled. I'm curious to know which OS you're using? The amplify env add command is useful when you want to import a pre-existing stack/environment, initialized using the Amplify CLI, say in some other project. An example of the usage could be something like this:

#!/bin/bash
set -e
IFS='|'

AWSCLOUDFORMATIONCONFIG="{\
\"Region\": \"us-east-1\",\
\"DeploymentBucketName\": \"mytestproject67-20181106123241-deployment\",\
\"UnauthRoleName\": \"mytestproject67-20181106123241-unauthRole\",\
\"StackName\": \"mytestproject67-20181106123241\",\
\"StackId\": \"arn:aws:cloudformation:us-east-1:132393967379:stack/mytestproject67-20181106123241/1c03a3e0-e203-11e8-bea9-500c20ff1436\",\
\"AuthRoleName\": \"mytestproject67-20181106123241-authRole\",\
\"UnauthRoleArn\": \"arn:aws:iam::132393967379:role/mytestproject67-20181106123241-unauthRole\",\
\"AuthRoleArn\": \"arn:aws:iam::132393967379:role/mytestproject67-20181106123241-authRole\"\
}"
PROVIDER_CONFIG="{\
\"awscloudformation\":$AWSCLOUDFORMATIONCONFIG\
}"

AWS_CONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":true,\
\"profileName\":\"default\"\
}"

amplify env add \
--name myownv \
--config $PROVIDER_CONFIG \
--awsInfo $AWS_CONFIG \
--yes

So, in the case above, a new environment would be initialized in your current project using the configuration metadata provided as a part of the flags.

You can get this project metadata from another project using the amplify env get --name <env-name>

[abdelhakim-atmani]

Thanks for the precisions.

It took me 10 minutes just to initialise.Does it need to connect to internet ? My connection was very slow yesterday. I have a mac book pro (2017) and the os version installed is 10.13.6.

Is there any plan to be able to add custom CF to the project ? I put a comment on the following issue: https://github.com/aws-amplify/amplify-cli/issues/80

[abdelhakim-atmani]

Today I did a push and it is still running (more than 15 minutes). When I checked CF in the AWS console nothing is running and in my terminal I have the message: "Updating resources in the cloud. This may take a few minutes..." Any idea on how I can see what is the problem ? Is there any logs to have more details ?

[ajhool]

AFAIK, Amplify does not currently support the following use-case:

An app where multiple front-ends share the same backend. I have an Admin webapp and a customer-facing webapp. Ideally, these two projects would live in different git repos and I would able to do "amplify pull" to retrieve updates. Multienv seems to be approaching this functionality... is it all of the way there yet?

[kaustavghosh06]

@abdelhakim-atmani With this @multienv release you can totally share your backends between different projects. And we also have the amplify env pull command now with the @multienv support which makes it easier for the backends across multiple frontend projects to be in sync.

So here's probably what you could do. You would need to copy over your amplify directory from a project (say a JS project) to another frontend project (say android). (Note: Only copy over the files which are not present in the amplify section of your .gitignore file - I would recommend using a VCS to get this cloning process done). After you do so, run the amplify init command and select the appropriate environment you want to pull from. After this, you would want to run an amplify configure project command to tell amplify which frontend your project is using. Thats it!

To keep the backends in sync in the future, you would run an amplify env pull [--restore] to pull in the current backend configurations of the resources in the cloud for any of your environments.

[abdelhakim-atmani]

Thanks @kaustavghosh06 for the precisions.

I finally found why the push was taking so much time. One step in the push process is to zip the folder ${projet}/amplify/#current-cloud-backend and to upload it in a S3 bucket.

In my case this file is huge ( > 2 GiB). The reason of that is because this folder contains node_modules of all functions and all builds (zip files).

Check the file @aws-amplify/cli/node_modules/amplify-provider-awscloudformation/lib/push-resources.js there is a function called storeCurrentCloudBackend and this function is the one which zip and upload the file in S3.

Is this step really required ? If yes does it need to include this folders/files (node_modules and all builds) ?

[jbabe]

@kaustavghosh06

Not sure if it's on the roadmap but what about adding amplify pipeline add that would generate cloudformation templates to setup a CodePipeline like @teknogeek0 presented here?

[alexjurkiewicz]

This is great! Thanks for the new functionality.

The restrictions on environment name are quite harsh. I would like environment names that allow digits, dashes and can be longer (say up to 20 chars). This would let me create environments that match the names we use for those environments elsewhere.

[jbabe]

This is great! Thanks for the new functionality.

The restrictions on environment name are quite harsh. I would like environment names that allow digits, dashes and can be longer (say up to 20 chars). This would let me create environments that match the names we use for those environments elsewhere.

We'd love this too (realizing we'd probably be restricted to valid S3 bucket and DNS naming conventions). Reason is we're thinking a git post-checkout hook to create a new env matching the branch name would be nice to add to our repos to make the git branching/amplify env creation flow a little easier for developers to follow

[ajhool]

@kaustavghosh06 Per the use-case from earlier, I think that the easiest way to do it would be a separate repo for my typescript amplify code (an API package). Then, I would import the API package into my client packages (eg. admin web app, customer web app). It's nice that Amplify generates files and folders into the root dir of the project, but that becomes messy across multiple projects.

Is there a reason NOT to use amplify in a separate package/repo? Is there magic happening such that amplify needs to be located at the root of a typescript project?

[timoteialbu]

Do you guys have the following on the roadmap: Adding a cloudfront distro in front of the graphQL endpoint? We would like this because we want to protect our app using a WAF and we need a cloudfront in front of it to do so. We wanna do something similar to whats described in this article.

[jbabe]

Do you guys have the following on the roadmap: Adding a cloudfront distro in front of the graphQL endpoint? We would like this because we want to protect our app using a WAF and we need a cloudfront in front of it to do so. We wanna do something similar to whats described in this article.

I was mentioning this to @kneekey23 at a user group tonight and she suggested I reach out to the AppSync team; @itrestian are you able to share where this is on the AppSync roadmap?

Thanks again @kneekey23 for finding this thread for me!

[undefobj]

@timoteialbu @jbabe service related questions are best handled on the official AWS forums as this repo is specifically for the Amplify CLI. For an AppSync roadmap question or request please post here: https://forums.aws.amazon.com/forum.jspa?forumID=280&start=0

[jonmifsud]

@kaustavghosh06 we've recently switched some of our apps from the old awsmobile into the amplify multienv. First of all well done its a big step up in functionality.

I'm somewhat interested in the possibility of automating releases across multiple accounts. We're working on a SAAS model of a platform which includes sensitive client data. For this reason based on research and AWS Guidelines we are looking at a multi-account setup.

In essence we want our dev team to have access to a singular dev account; where all testing environments are set. However we then want to release to multiple environments at one go (assuming we don't have breaking changes); as well as the possibility to create new sub-accounts / environments at the same time. Does Amplify create CloudFormation templates that can be copied/used in this way?

[kaustavghosh06]

@jbabe Have you taken a look at the Amplify Console -> https://aws.amazon.com/amplify/console/? I think this should solve for your CI/CD needs which you would otherwise use Codepipeline for.

@alexjurkiewicz @jbabe There are various implications of having a wide variety of flexitbility in the environment name since we use this environment name as a part of the resource names for the various categories provisioned using the Amplify CLI. For eg: S3 does not allow uppercase letters, numbers etc as a part of the bucket name. Similarly Lex doesn't allow the usage of '-' as a part of the Bot name. Having said that we're brainstorming about how to best work around this limitation. I'll keep you guys updated as a part of this thread regarding what we come up with.

@ajhool Yes, the magic you're referring to does happen in the form of some code generation (https://aws-amplify.github.io/docs/cli/codegen?sdk=js) in your project if you have a GraphQL API added to your Amplify project and if you choose to enable codegen.

[kaustavghosh06]

@jonmifsud Yes, that's absolutely possible with the multienv version. You can associate different environments with different AWS profiles (which could be in same or different AWS accounts). You can then follow the flow mentioned out here -> https://aws-amplify.github.io/docs/cli/multienv?sdk=js to deploy to these environments tied to same or different AWS accounts.

[jonmifsud]

@kaustavgosh06 thanks for the super quick reply.

What we are looking to do is actually slightly more complex maybe.

We will have multiple dev environments and a test prod/staging environment. Once confirmed that we’re good to go we want to roll that out to say 5/10 different prod environments. This last step ideally we don’t want to do manually by a developer from amplify cli.

In my mind it would be something like update all environments we have based on the same production cloud formation template. (Which would be taken/generated from amplify-cli)

[rkuzsma]

I just initialized multienv on my project for the first time, following the documentation. Init worked, it successfully pushed a new dev configuration.

I then tried to add a new prod configuration by following the steps at the end of the documentation, specifically by running: amplify env add

The CLI threw an error:

/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/node_modules/gluegun/build/index.js:13
    throw up;
    ^
SyntaxError: Unexpected token u in JSON at position 0
    at JSON.parse (<anonymous>)
    at Command.run (/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/src/commands/env/add.js:9:25)
    at Runtime.<anonymous> (/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/node_modules/gluegun/build/runtime/run.js:90:58)

These 2 lines of @aws-amplify/cli/src/commands/env/add.js:

const config = JSON.parse(context.parameters.options.config);
const awsInfo = JSON.parse(context.parameters.options.awsInfo);

should be moved to just after line 20 to fix the error.

Other notes:

• The end of the documentation should be updated to specify that --name and --config are required parameters for the amplify env add command.
• The end of the documentation should format the – dashes as -- dashes to prevent copy/paste errors.
• Intuitively, I thought I should be able to say amplify env create to create a new environment. Instead, I think I need to re-run amplify init and carefully specify not to use the existing project.
• The documentation is a bit confusing to read. After the first pass through it, I mistakenly assumed that multienv was tightly coupled at the implementation level to git and to git branches, so I was initially turned off by using it. It was only when I re-read the last section, Some other helpful Environment related commands, that I second guessed my assumption. That section alone implied (to me at least) that pulling/adding/removing environments is decoupled from git.

[kaustavghosh06]

@jonmifsud Ideally I would recommend having a git-based workflow for managing deployments to different environments as recommended in https://aws-amplify.github.io/docs/cli/multienv?sdk=js. Having said that, you can still utilize the amplify push -env <env-name> command for your needs.

[kaustavghosh06]

@rkuzsma the amplify env add command should be used to add an already existing environment (with an existing cloudformation stack setup) to your project. To initialize a new environment you would currently need to go through the amplify init flow.

Having said that, I agree a command like amplify env create would be nice to have. I'll brainstorm with my team regarding this. Thanks for the feedback. Also, we're in the process of tightening the multi-environment documentation based on various feedbacks like yours.

[rkuzsma]

Thanks! I have the following relevant use cases that I currently shell script around, but that I hope multienv can solve in the future. It looks really close to what I need:

1. I would like to configure and push a different Cognito auth for dev environment and prod environment, and have both distinct environment configurations saved into the same git branch, e.g. master branch.
2. I want a team member to be able to pull down the git master branch and switch between the dev and prod AWS environments by using the command line. (I am operating under the assumption that the team member has access to the requested AWS resources)
3. When switching environments, I want the auto-generated aws-exports.js file to be updated to reflect the Cognito auth values for the current environment, so that the app will work with minimal fuss.

[defel]

Hi,

I migrated to the new multienv build, here is my feedback:

What works great:

• show current profile and configuration,
• create new profiles,
• switching between profiles

What is missing for me:

• add (custom) environment variables to the current build, which will be used by yarn serve or yarn build
• get current profile in process.env for js-app on yarn serve or yarn build
• get the current profile with amplify-cli to use it in scripts

Reason: I have deployed on different stages/aws-accounts and have a lambda-backed api on these. I need to set this url on all stages (and local while developing) and atm I dont see a possibility to do this.

[defel]

Can someone reproduce the same problem?

Everytime I switch the profile, the following key is missing in generated aws-exports.json:

{ "aws_appsync_authenticationType": "AMAZON_COGNITO_USER_POOLS" }

I have to add it manually everytime after switching.

[ajhool]

@defel, I also have that issue and IIRC it has been fixed in the master branch but this branch is behind that commit. I'm having trouble finding what commit in the master branch has the fix or the other issues that have talked about that error. Doesn't it throw a "missing authorization token" error when making an API call? I know there was another issue about this I'm just not able to find it right now...

[kaustavghosh06]

@defel I was able to root cause the issue. I’ll provide a fix for it in the multienv branch soon.

[defel]

@ajhool correct, Request to graphql endpoint error: Request to graphql endpoint has missing authorization header: Graphql was the error I get in the browser later

@kaustavghosh06 thanks for fixing! :raised_hands:

[yashutanna]

Hey guys

Thanks for the awesome tool.

I have the following setup and I think i may be struggling while a solution exists out there. Ive tried the googled - but this is bleeding edge stuff

I have 2 git branches(thereby also 2 amplify environments) - Develop and Master, with git flow - we also create feature branches. all of which will use the develop amplify environment

I also want to have a cognito user pool per environment.

Im trying to sort out my CI/CD process for this green field project.

I've pushed the backend changes already - but need to get my app deployed using Amplify console. Since the aws-exports file is ignored by git (rightly so), when I push to develop branch, Amplify console picks up a new build but fails when I require aws-exports as a config object for Amplify.config().

Any feedback is highly appreciated!.

Thanks

[kaustavghosh06]

@yashutanna I think you’re missing the backend build section in the build config of your app in the Amplify console. Checkout https://github.com/aws-amplify/amplify-cli/issues/754#issuecomment-457279407 as to how to go how about adding it to your build spec.

[alexjurkiewicz]

Trying to test this and receiving the following error during init:

$ npx amplify init
Note: It is recommended to run this command from the root of your app directory
You're initializing your project with a beta version of the CLI which supports multiple environments of your project
? Enter a name for the project myapp
? Enter a name for the environment test
? Choose your default editor: Atom Editor
? Choose the type of app that you're building (Use arrow keys)

node_modules/gluegun/build/index.js:13
    throw up;
    ^

TypeError: Cannot read property 'value' of undefined
    at ListPrompt.getCurrentValue (node_modules/@aws-amplify/cli/node_modules/inquirer/lib/prompts/list.js:121:53)
    at MapSubscriber._next (node_modules/rxjs/internal/operators/map.js:49:35)
    at MapSubscriber.Subscriber.next (node_modules/rxjs/internal/Subscriber.js:66:18)
    at TakeSubscriber._next (node_modules/rxjs/internal/operators/take.js:54:30)
    at TakeSubscriber.Subscriber.next (node_modules/rxjs/internal/Subscriber.js:66:18)
    at Interface.handler (node_modules/rxjs/internal/observable/fromEvent.js:22:28)
    at emitOne (events.js:121:20)
    at Interface.emit (events.js:211:7)
    at Interface._onLine (readline.js:280:10)
    at Interface._line (readline.js:629:8)

$ grep amplify package.json
    "@aws-amplify/cli": "^0.2.1-multienv.42",

[michaelcuneo]

/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/node_modules/gluegun/build/index.js:13
    throw up;
    ^
SyntaxError: Unexpected token u in JSON at position 0
    at JSON.parse (<anonymous>)
    at Command.run (/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/src/commands/env/add.js:9:25)
    at Runtime.<anonymous> (/Users/REDACTED/.nvm/versions/node/v10.14.1/lib/node_modules/@aws-amplify/cli/node_modules/gluegun/build/runtime/run.js:90:58)

I';m getting this issue in my code, and I can't see a solution here anywhere that actually fixed the issue. I can't amplify serve my project.

[kaustavghosh06]

@michaelcuneo Which version of the CLI are you using? And you’re get the above mentioned error when you’re running the ‘amplify env add’ command?

[michaelcuneo]

I am running 1.1.7, I'm getting the error just doing an amplify serve.

Created the project at work, pulled the project at home in my office, npm install, then amplify serve.

I have also tried the project with amplify serve --profile work to ensure it's running with my, aws_iam work profile.

[kaustavghosh06]

What does ‘amplify status’ output?

[michaelcuneo]

Current Environment: nhc

Category	Resource name	Operation	Provider plugin

[ajhool]

I opened up an issue that is generally related to Multi-environments and Team workflows, here ( #604 ) that would make it easier ( In my opinion ) to manage multiple Amplify environments. I find that the amplify workflow is a little too coupled with the git workflow and that it would be nice to completely isolate Amplify into it's own repo/package system. Isolation almost always makes collaboration / multiple environments more simple

[michaelcuneo]

I don’t even know if i want to use multiple environments, like, as in staged dev/prod... I just want to run it in two places at once. I’d have assumed that would have been way simpler. I run it at work, and want to run the exact same identical project at home, with no env or modifications or anything, but it won’t work.

[ajhool]

Sorry, that previous comment was just a general response to this RFC, not a response to your issue.

I don't use serve but have not had the type of issue that you're describing, it has always been quite simple to run amplify

[kaustavghosh06]

@michaelcuneo Looks like your the amplify version that you're using in your nvm version is an older one (I remember seeing a similar issue previously based on your stack trace). Could you please open up another issue on GitHub so that we can track this issue if you're still seeing this?

[acrabb]

Not sure if this is the right place for this, but I've gotten this error Unexpected token export (referring to my aws-exports file) when using the CLI to try to add a new environment.

But team-provider-info.json had the new env added and when running amplify push everything seemed to work anyway so ¯_(ツ)_/¯

[ajhool]

@acrabb are you using typescript while aws-exports is a .js file? I saw that error until I renamed aws-exports.js to aws-exports.ts while using typescript