Amplify Continuous Deployment Problem: Monorepo settings and can't find aws-exports.js

geoseong commented 4 years ago

Hi.

I'm working monorepo project.

github repo: kendra-button
- I've tried with temp branch and feature-frontend-api branch.

I've tried so many ways to deploy via Hosting with Amplify Console (Managed hosting with custom domains, Continuous deployment).

the major problem for me is,

amplify's build image can't pull aws-export.js file when my amplify project is placed in other folder
even when amplify project is placed in root, sometimes can't find aws-exports.js(this is other project's case)
I followed monorepo configuration guide, amplify yml doesn't recognize what I've configured

Please let me know why this situation happens and how to deploy it without errors. I wasted about more than 3 days resolving this problem.

here is my situations below.

1st attempt: normal

Project architecture

tried way 1:
tried way 2:

/amplify.yml in my repository

version: 0.1
frontend:
  phases:
    build:
      commands:
        - cd frontend/kendra-button-front
        - yarn
        - yarn build
  artifacts:
    # IMPORTANT - Please verify your build output directory
    baseDirectory: frontend/kendra-button-front/src/out
    files:
      - '**/*'
  cache:
    paths: []

Errors in "Build" step in Amplify Console

2nd attempt: try to pull amplify's environment files manually

Project architecture

same architecture as 1st attempt

/amplify.yml in my repository

I'd add Environment Variables in Amplify Console
the reason why I put the long amplify pull command is
- try to pull amplify's environment files manually
- especially to get aws-exports.js!
- It's working in my macos with this long amplify pull command

version: 0.1
frontend:
  phases:
    # preBuild:
    #   commands:
    build:
      commands:
        - AMPLIFY="{\
          \"projectName\":\"$PROJECT_NAME\",\
          \"appId\":\"$AWS_APP_ID\",\
          \"envName\":\"$ENV_NAME\",\
          \"defaultEditor\":\"code\"\
          }"
        - AWSCLOUDFORMATIONCONFIG="{
          \"configLevel\":\"project\",
          \"useProfile\":false,
          \"profileName\":\"default\",
          \"accessKeyId\":\"$ACCESS_KEY_ID\",
          \"secretAccessKey\":\"$SECRET_ACCESS_KEY\",
          \"region\":\"us-west-2\"
          }"
        - PROVIDERS={
          \"awscloudformation\":$AWSCLOUDFORMATIONCONFIG
          }
        - amplify pull --amplify $AMPLIFY --providers $PROVIDERS --yes
        - find . -name "aws-exports.js"
        - cd frontend/kendra-button-front
        - yarn
        - yarn build
  artifacts:
    # IMPORTANT - Please verify your build output directory
    baseDirectory: frontend/kendra-button-front/src/out
    files:
      - '**/*'
  cache:
    paths: []

Errors in "Build" step in Amplify Console

I tried more than 20 times to solve this problem

3st attempt: "Monorepo settings"

refer to this document

Project architecture

/amplify.yml in my repository

version: 1
applications:
  - appRoot: frontend/kendra-button-front
    frontend:
      phases:
        build:
          - find . -name "aws-exports.js"
          - yarn
          - yarn build
      artifacts:
        files:
          - '**/*'
        baseDirectory: frontend/kendra-button-front/src/out

Errors in "Build" step in Amplify Console

rocketmaniac commented 4 years ago

Thanks, @geoseong! your debug line, - find . -name "aws-exports.js", helped me to figure out the path i needed in my code, i.e.:

import config from '../aws-exports.js'

Your input, plus removing aws-exports.js from .gitignore helped to get my app up and running again in the short-term. I'm pretty sure committing aws-exports.js is an anti-pattern, but it keeps me moving forward on my prototype for now, until I can figure out a better solution!

geoseong commented 4 years ago

@swaminator I also asked it to Discord to @brene , he said you are an expert on that. please help to get the solution how it works and how to solve the problem 😓

geoseong commented 4 years ago

@rocketmaniac like you said, committing aws-exports.js is anti-pattern and I think there is no reason to use amplify build with that way.

so I will use amplify-cli-action with S3 distribution until the response...

rocketmaniac commented 4 years ago

@geoseong did amplify-cli-action work for you? If so, can you please share a sanitized version of your flow?

geoseong commented 4 years ago

@rocketmaniac I was planning to deploy to s3 bucket in amplify-cli-action and configure CloudFront and custom domain manually. but I got a better solution to make Continuous Deployment in Amplify Console work.

first, make the shell script to execute amplify pull

(be careful with environment naming start with AWS. is the AWS's system environment)

I got a hint from #496 . thanks @vchaddha

#!/usr/bin/env bash

AMPLIFY="'{\
\"projectName\":\"${PROJECT_NAME}\",\
\"appId\":\"${AWS_APP_ID}\",\
\"envName\":\"${ENV_NAME}\",\
\"defaultEditor\":\"code\"\
}'"
AWSCLOUDFORMATIONCONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":false,\
\"profileName\":\"default\",\
\"accessKeyId\":\"${ACCESS_KEY_ID}\",\
\"secretAccessKey\":\"${SECRET_ACCESS_KEY}\",\
\"region\":\"${REGION}\"\
}"
PROVIDERS="'{\
\"awscloudformation\":${AWSCLOUDFORMATIONCONFIG}\
}'"

cmd="amplify pull --amplify ${AMPLIFY} --providers ${PROVIDERS} --yes"
echo $cmd
eval $cmd

and add to execute that sh file command lines in amplify.yml.

for me, like this below.

version: 0.1
frontend:
  phases:
    preBuild:
      commands:
        - cd frontend/kendra-button-front
        - chmod +x amplify-pull.sh
        - ./amplify-pull.sh
    build:
      commands:
        - find . -name "aws-exports.js"
        - yarn
        - yarn build
  artifacts:
    baseDirectory: frontend/kendra-button-front/src/out
    files:
      - '**/*'
  cache:
    paths: []

dbhagen commented 4 years ago

Ok, so this is my current solve for it. It relies on on pre-post git hooks (using Yorkie: https://github.com/yyx990803/yorkie) First, setup an EXPORT_KEY environment variable (I have it in my .zshrc file on my mac, export EXPORT_KEY=<some random key>). Hang on to this variable, you'll need it later.

In your package.json file, add these scripts:

{
  "scripts": {
    "unlock-amplify": "echo $EXPORT_KEY | gpg --batch -d --passphrase-fd 0 --yes aws-exports.js.gpg > aws-exports.js",
  },
  "gitHooks": {
    "post-commit": "[[ ! -z \"${EXPORT_KEY}\" ]] && [ -f .commit ] && rm -f .commit aws-exports.js.gpg && echo $EXPORT_KEY | gpg --batch --yes --passphrase-fd 0 -c aws-exports.js && git add aws-exports.js.gpg && git commit --amend -C HEAD --no-verify",
    "pre-commit": "touch .commit",
    "post-merge": "[[ ! -z \"${EXPORT_KEY\" ]] && echo \"$EXPORT_KEY\" | gpg --batch -d --passphrase-fd 0 --yes aws-exports.js.gpg > aws-exports.js"
  }
}

What you'll see is that it will gpg encrypt your aws-exports.js during a commit as aws-exports.js.gpg. This gets committed to your repo.

The latest build image seemed to be able to fire the post-merge script above on checking out the code for building, so you shouldn't need this next step, but I used to do this. In your Amplify.yml file, add this to your prebuild:

frontend:
  phases:
    preBuild:
      commands:
        - yarn unlock-amplify
or if you use NPM
        - npm run unlock-amplify

This allows you to decrypt the aws-exports.js.gpg back into a aws-exports.js file needed for building.

rocketmaniac commented 4 years ago

@geoseong , @dbhagen thanks for your solutions!

I ended up going with the amplify pull approach so as not to introduce any more dependancies to the build, and it works great.

It turns out the approach is somewhat documented here:

https://docs.amplify.aws/cli/usage/headless#amplify-pull-parameters

but @geoseong i like your doc better as it illustrates the use of the env vars ;-)

Since the amplify sample projects all make use of aws-exports in the frontend code, it would save everyone a lot of time and frustration if the docs or build templates were updated to cover the requisite preBuild step for amplify pull so that the sample code can actually build properly OOTB.

andrewbtp commented 4 years ago

For anybody using @geoseong 's approach or script, I had to switch profile to true to make it work.

geoseong commented 4 years ago

@andrewbtp do you mean \"useProfile\":true in AWSCLOUDFORMATIONCONFIG in shell script? if it is right, I'll take a note

andrewbtp commented 4 years ago

@geoseong That's it!

kldeb commented 4 years ago

Does anyone know the parameter to only generate the aws-exports file and not add the amplify directory? I'm testing locally since I can't figure out what it isn't working in the amplify console and it just deployed a stale backend on me, not sure why.

geoseong commented 4 years ago

I found another solution using backend and changing the version to 1 in amplify.yml. and create customized amplifyPush shell script. here is issue that I commented

need to wait until public solution comes...

kldeb commented 4 years ago

@geoseong thanks, that's quite the workaround. I added aws-exports.js to the repo for now. were you able to get a an aws-exports.js file generated using your amplify pull script? it didn't work for me.

geoseong commented 4 years ago

@kldeb yes. I was able to get aws-exports.js file using amplify pull cli. it's strange, need to know more specific that you'd tried 😅

kldeb commented 4 years ago

@geoseong here's my amplify.yml file:

version: 1
frontend:
  phases:
    preBuild:
      commands:
        - npm install
        - chmod +x amplify-pull.sh
        - ./amplify-pull.sh
    build:
      commands:
        - find . -name "aws-exports.js"
        - npm run build
  artifacts:
    baseDirectory: build
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

and here's my amplify-pull.sh file:

#!/bin/bash
set -e
IFS='|'

AMPLIFY="{\
\"projectName\":\"my-project-name\",\
\"appId\":\"$APP_ID\",\
\"envName\":\"$USER_BRANCH\",\
\"defaultEditor\":\"code\"\
}"

REACTCONFIG="{\
\"SourceDir\":\"src\",\
\"DistributionDir\":\"build\",\
\"BuildCommand\":\"npm run-script build\",\
\"StartCommand\":\"npm run-script start\"\
}"

AWSCLOUDFORMATIONCONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":false,\
\"profileName\":\"default\",\
\"accessKeyId\":\"${ACCESS_KEY_ID}\",\
\"secretAccessKey\":\"${SECRET_ACCESS_KEY}\",\
\"region\":\"us-east-1\"\
}"

FRONTEND="{\
\"frontend\":\"javascript\",\
\"framework\":\"react\",\
\"config\":$REACTCONFIG\
}"

PROVIDERS="{\
\"awscloudformation\":$AWSCLOUDFORMATIONCONFIG\
}"

cmd="amplify pull --amplify ${AMPLIFY} --frontend ${FRONTEND} --providers ${PROVIDERS} --yes"
echo $cmd
eval $cmd

geoseong commented 4 years ago

@kldeb did you test the shell script of your amplify-pull.sh in your local?

and remove the frontend parameter and try again.

kldeb commented 4 years ago

@geoseong yes, it worked locally but it added the amplify directory and files to the project. I don't see the option to turn that off in the headless docs.

I'll try removing the frontend parameter.

kennand commented 4 years ago

Clean way to do this is to set your Amplify project-config.json destination path to your mono repo's common folder and import from there.

litwicki commented 3 years ago

@geoseong are you still having any issues or can we close this issue?

siegerts commented 3 years ago

Closing for now since there seems to be multiple workarounds/patterns outlined above. If this is still an issue or confusing, I encourage opening a PR against the Amplify Documentation site for the headless patterns referenced above. It would be greatly appreciated 😄

https://github.com/aws-amplify/docs

geoseong commented 3 years ago

@siegerts I've already requested PR in https://github.com/aws-amplify/docs on 19 Jun 2020, and still in progress.

I understand the hard working atmosphere in this world, but the progress is so slow 😅

@litwicki I issued this 9 months ago and I'm not using Amplify right now so I almost forgot it, I'll try it again in the future

github-actions[bot] commented 1 year ago

This issue has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.

aws-amplify / amplify-hosting