SourceDir is getting ignored in the headless mode

[m98]

Before opening, please confirm:

• [X] I have installed the latest version of the Amplify CLI (see above), and confirmed that the issue still persists.
• [X] I have searched for duplicate or closed issues.
• [X] I have read the guide for submitting bug reports.
• [X] I have done my best to include a minimal, self-contained set of instructions for consistently reproducing the issue.
• [X] I have removed any sensitive information from my code snippets and submission.

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v16.13.1

Amplify CLI Version

7.6.26

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

Only used the CLI.

Amplify Categories

auth

Amplify Commands

pull

Describe the bug

I followed the Amplify headless mode documentation to create an amplify-pull.sh bash script to pull the app during the build time on the pipeline.

Here is the bash script:

#!/bin/bash
set -e
IFS='|'

AUTHCONFIG="{\
\"userPoolId\": \"$COGNITO_USER_POOL_ID\",\
\"webClientId\": \"$COGNITO_WEB_CLIENT_ID\",\
\"nativeClientId\": \"$COGNITO_WEB_CLIENT_ID\",\
}"

CATEGORIES="{\
\"auth\":$AUTHCONFIG\
}"

AWSCLOUDFORMATIONCONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":false,\
\"profileName\":\"default\",\
\"accessKeyId\":\"$AMPLIFY_ACCESS_KEY_ID\",\
\"secretAccessKey\":\"$AMPLIFY_SECRET_KEY\",\
\"region\":\"eu-west-1\"\
}"
AMPLIFY="{\
\"projectName\":\"$AMPLIFY_PROJECT_NAME\",\
\"appId\":\"$AMPLIFY_APP_ID\",\
\"envName\":\"$AMPLIFY_ENVIRONMENT\",\
\"defaultEditor\":\"code\"\
}"

# SourceDir is defined here, and is used in the FRONTEND config (a few lines bellow this)
REACTCONFIG="{\
\"SourceDir\":\"src/aws-data\",\
\"DistributionDir\":\"build\",\
\"BuildCommand\":\"npm run-script build\",\
\"StartCommand\":\"npm run-script start\"\
}"

FRONTEND="{\
\"frontend\":\"javascript\",\
\"framework\":\"react\",\
\"config\":$REACTCONFIG\
}"
PROVIDERS="{\
\"awscloudformation\":$AWSCLOUDFORMATIONCONFIG\
}"

amplify pull \
--amplify $AMPLIFY \
--frontend $FRONTEND \
--providers $PROVIDERS \
--categories $CATEGORIES \
--yes || true

Expected behavior

I expect running the above bash script put the aws-exports.js in the src/aws-data but it ignore the path and put the aws-exports.js in the src.

I tried other values for the SourceDir like aws-data, or even tried to manually create the directory in case that the CLI was not able to make it. But none of it helped!

Reproduction steps

1. Copy-paste the bash script above into an empty directory
2. put real values for the appId, accessKey, etc (now there are variables)
3. run sh pull.sh to pull the Amplify app
4. It's expected, but does not happen: The aws-exports.js should be in the src/aws-data

GraphQL schema(s)

No response

Log output

No response

Additional information

No response

[josefaidt]

Hey @m98 thanks for raising this! Unfortunately I was not able to reproduce this issue using the provided script. In this pipeline setting, does it have access to the source files from the repository? If so, project information like SourceDir is captured in a git-tracked file amplify/.config/project-config.json. With this we can run a headless amplify init command and use the existing settings by omitting the project config from our script.

#!/bin/bash
set -e
IFS='|'

AMPLIFY_APP_ID='xxx'
AMPLIFY_ENVIRONMENT='dev'
AWS_REGION='us-east-1'

AWSCLOUDFORMATIONCONFIG="{\
\"configLevel\":\"project\",\
\"useProfile\":true,\
\"profileName\":\"default\",\
\"region\":\"$AWS_REGION\"\
}"
AMPLIFY="{\
\"appId\":\"$AMPLIFY_APP_ID\",\
\"envName\":\"$AMPLIFY_ENVIRONMENT\",\
}"

PROVIDERS="{\
\"awscloudformation\":$AWSCLOUDFORMATIONCONFIG}"

amplify init \
  --amplify $AMPLIFY \
  --providers $PROVIDERS \
  --yes || true

[m98]

Yes, as far as I know, the bash script will have access to the source files in the repository. But sorry I could not understand how does it solve the issue?

In my case, there is a white-label app, and they have different titles or names based on which domain they are running from. For example, I build the React project once, but it will get deployed on example.com, and test.com, and based on different environment variables and the domain itself, the behavior and look of the website might be different.

What I'm trying to do now is to put the aws-exports into separate directories (instead of override) using this pull script. For example, it would look like:

src/
   aws-exports/
      example-project/
         aws-exports.js
      test-project/
         aws-exports.js

But as I explained in the issue, when I pass a value for the SourceDir, nothing happens, and ignore the path I set to it.

[josefaidt]

Hey @m98 thanks for clarifying! Can you provide an example of how you're generating these files in two different locations? Are these two separate executions of the provided script based on the Amplify environment?

[m98]

Hey @josefaidt! There is a bash script called amplify-pull.sh which is responsible to pull these two projects. It receive a parameter which you can set which project you want it to pull, then it automatically place it on the right directory.

For example, let say you want to pull project Y, you can do sh amplify-pull.sh project-y and it pull this Amplify application, and move the aws-exports to src/aws-exports/project-y.js

Then for the next project, first the script deletes the amplify directory at the root of the project, then pulls the other project.

The reason for deleting the amplify is that if it's there, you'll get an App ID mismatch from the Amplify CLI, and also we are not using the amplify directory (no backend is needed). Indeed I tried to pass --no in the last line of bash script to the amplify question which asks if you want to change the back-end on this project, but id ignores the --no, and waits for the answer on the terminal

The other way would be to use environment, each environment getting named after the project name. Like env-project-y and env-project-x but I'm not sure if that's the correct approach!

How would you handle this case?

---

Also about the SourceDir, it's still ignoring it! I could not make it work. I ended up using the mv scr/aws-exports.js src/aws-exports/$project.js But something to mention is that when I initialized the Amplify app, I set the SourceDir to src. But when I'm pulling it using the headless mode of the CLI, I'm passing src/aws-exports! I'm not sure if this is getting ignored because the value that is initialized is different from the value from the pull!

[josefaidt]

Hey @m98 :wave: upon revisiting this after looking into another headless script issue I was able to reproduce this using the following steps:

1. create a project with amplify init -y
2. create a git repository and push the new project to git
3. download the repository contents
4. unzip

5. use the following pull.sh script to pull the project

   #!/bin/bash
   set -e
   IFS='|'
   
   AMPLIFY_APP_ID='d1lob39t6eexky'
   AMPLIFY_ENVIRONMENT='dev'
   AWS_REGION='us-east-1'
   
   AWSCLOUDFORMATIONCONFIG="{\
   \"configLevel\":\"project\",\
   \"useProfile\":true,\
   \"profileName\":\"default\",\
   \"region\":\"$AWS_REGION\"\
   }"
   
   AMPLIFY="{\
   \"appId\":\"$AMPLIFY_APP_ID\",\
   \"envName\":\"$AMPLIFY_ENVIRONMENT\",\
   }"
   
   REACTCONFIG="{\
   \"SourceDir\":\"src/aws-data\",\
   }"
   
   FRONTEND="{\
   \"config\":$REACTCONFIG}"
   
   PROVIDERS="{\
   \"awscloudformation\":$AWSCLOUDFORMATIONCONFIG}"
   
   amplify pull \
     --amplify $AMPLIFY \
     --providers $PROVIDERS \
     --frontend $FRONTEND \
     --yes || true

6. observe exports file is generated at src/aws-exports.js rather than src/aws-data/aws-exports.js

In this case, I would assume all other values for SourceDir are overwritten in favor of the values provided by the script (i.e. ignored amplify/.config/project-info.json and ignored defaults src/). It is also worth noting changing pull to init or providing the entire FRONTEND block does not have any effect on the results.

Marking as a bug 🙂