search

aws-amplify / amplify-cli

The AWS Amplify CLI is a toolchain for simplifying serverless web and mobile development.
Apache License 2.0
2.8k stars 819 forks source link

Duplicated files generated every time running amplify pull/push #13843

Closed Dickensli closed 1 week ago

Dickensli commented 2 weeks ago

How did you install the Amplify CLI?

npm

If applicable, what version of Node.js are you using?

v22.4.0

Amplify CLI Version

12.12.4

What operating system are you using?

Mac

Did you make any manual changes to the cloud resources managed by Amplify? Please describe the changes made.

No manual changes made

Describe the bug

Every time I run amplify pull or amplify push, duplicate files are generated under the #current-cloud-backend directory. For example, my backend folder structure looks like this:

backend |── api |── awscloudformation/build │ └── root-cloudformation-stack.json |──auth │ └── build |── function |── amplify-meta.json |── backend-config.json |── tags.json

However, the #current-cloud-backend directory ends up with:

current-cloud-backend

|── api |── api 2 |──awscloudformation/build | └── root-cloudformation-stack.json | └── root-cloudformation-stack 2.json |── auth │ └── build │ └── build 2 |── auth 4 |── function |── amplify-meta.json |── amplify-meta 2.json |── backend-config.json |── backend-config 2.json |── tags.json └── tags 2.json

These number-appended folders are usually empty or outdated and vary with each run, e.g. I updated my api's defaultauthentication from API_KEY to AMAZON_COGNITO_USER_POOLS using amplify update api in local; then I ran amplify push, and two amplify-meta.jsons were generated. amplify-meta.json catch the change but amplify-meta 2.json does not.

Sometimes, it gets worse, and the api/ folder disappears. When I check Amplify Backend Studio, I see errors like "can't retrieve data since schema is missing." While I can fix this by running amplify push --force, the issue of duplicated files still persists.

One possible cause is: I have turned on CI/CD linked to github repo, usually when I have some backend change, I run amplify push, waiting for push to succeed, and then run git add, git commit, git push which triggers CI/CD runs. Could this introduce any conflict between manual amplify push vs. CI/CD push?

my amplify.yml

version: 1
appRoot: ./career-copilot-app/
backend:
  phases:
    build:
      Commands:
        - sudo pip3 install pipenv
        - sudo pip3 install venv
        - amplifyPush --simple
  artifacts:
    baseDirectory: amplify/backend
    files:
      - '**/*'
  cache:
    paths:
      - 'amplify/**/node_modules/**/*'
      - 'amplify/.serverless/**/*'
      - 'amplify/backend/**/build/**/*'
frontend:
  phases:
    preBuild:
      commands:
        - npm ci
        - npm audit fix
    build:
      commands:
        - npm run build
  artifacts:
    baseDirectory: dist/career-copilot-app/browser
    files:
      - '**/*'
  cache:
    paths:
      - node_modules/**/*

Expected behavior

no duplicated files/folders should be generated

Reproduction steps

  1. make some local changes in api schema or lambda function src
  2. amplify push (duplicated files appear)
  3. manually delete all duplicated files under #current-cloud-backend
  4. waiting for push to succeed
  5. git add
  6. git commit
  7. git push
  8. waiting for CI/CD to finish
  9. amplify pull
  10. another a few duplicated files appear

Project Identifier

083a411c049aa404823988034e0b09f6

Log output

``` # Put your logs below this line ```

Additional information

No response

Before submitting, please confirm:

ykethan commented 1 week ago

Hey @Dickensli, thank you for reaching out. Is the project currently being used in a folder that sync with cloud storage like iCloud or OneDrive? From the reproduction steps this appears to be similar to https://github.com/aws-amplify/amplify-cli/issues/13335

Dickensli commented 1 week ago

Thanks for the quick turnaround! No, I don't have any backup or sync with external storage. This should be more serious issue than seeing duplicated files. Please see this section

Sometimes, it gets worse, and the api/ folder disappears. When I check Amplify Backend Studio, I see errors like "can't retrieve data since schema is missing." While I can fix this by running amplify push --force, the issue of duplicated files still persists.

Moreover, sometimes what I observed is

  1. I created a new function/some_lambda_function by running amplify add function
  2. I ran amplify push
  3. Then after it finished, immediately, I ran amplify pull
  4. then cli created function/some_lambda_function and function/some_lambda_function 2 where

Overall, the result appears disorganized ...

ykethan commented 1 week ago

Hey @Dickensli, thank you for the information. Could you try the following

  1. download the #current-cloud-backend.zip archive, unzip, remove duplicate files, re-zip, and re-upload. While re-zipping do ensure the current-cloud-backend is not being upload as a nested folder.
  2. pull the backend into a different empty directory outside the existing project directory path using amplify pull --appId <app-id> --envName <env-name>
  3. make a small change such as adding a console log in a function
  4. then run amplify push Do let us know if this does still create duplicate files. On a push Amplify does not make a copy of the directory or files locally, the above enable us to dive deep to the issue.
Dickensli commented 1 week ago

Hi, I deduplicated all files and reupload #current-cloud-backend.zip; running amplify pull --debug --appId dyq20j2b80hyx --envName tuna I got following error

🛑 Failed to pull the backend.

Learn more at: https://docs.amplify.aws/cli/project/troubleshooting/

PullBackendFault: Failed to pull the backend.
    at attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend.js:55:15)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/commands/pull.js:64:9)
    at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
    at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
    at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

File at path: '/Users/dickensli/Desktop/new_amplify/amplify/#current-cloud-backend/amplify-meta.json' does not exist
Error: File at path: '/Users/dickensli/Desktop/new_amplify/amplify/#current-cloud-backend/amplify-meta.json' does not exist
    at JSONUtilities.readJson (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-cli-core/lib/jsonUtilities.js:44:19)
    at AmplifyToolkit.readJsonFile (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/extensions/amplify-helpers/read-json-file.js:6:45)
    at ensureAmplifyMeta (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/attach-backend.js:126:46)
    at Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/attach-backend.js:117:36)
    at async Object.attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/amplify-provider-awscloudformation/lib/index.js:196:3)
    at async queryProvider (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend-steps/a10-queryProvider.js:36:5)
    at async attachBackend (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/attach-backend.js:43:9)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/commands/pull.js:64:9)
    at async Object.executeAmplifyCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:194:9)
    at async executePluginModuleCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:139:5)
    at async executeCommand (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/execution-manager.js:37:9)
    at async Object.run (/snapshot/amplify-cli/build/node_modules/@aws-amplify/cli-internal/lib/index.js:121:5)

Session Identifier: 718e26cf-46c3-4064-a22b-6b99d0795434
Dickensli commented 1 week ago

Hey I finally got it right. The above error is because I accidentally included nested #current-cloud-backend to zip. You are right, my iCloud backup is the root cause. I did not realize iCloud backed up every file on my laptop... but Thanks! Really helpful!

github-actions[bot] commented 1 week ago

This issue is now closed. Comments on closed issues are hard for our team to see. If you need more assistance, please open a new issue that references this one.