k1LoW / serverless-s3-sync

A plugin to sync local directories and S3 prefixes for Serverless Framework :zap:
182 stars 68 forks source link

deleteRemoved: false is not working always #92

Open a-marcel opened 2 years ago

a-marcel commented 2 years ago

Hello,

i figured out that there is an issue, if one directory has the option deletedRemoved: true and all other are using deletedRemoved: false.

s3Sync:
    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/static-pages
      localDir: .serverless_nextjs/assets/static-pages/
      acl: private
      deleteRemoved: false

    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/static/
      localDir: .serverless_nextjs/assets/_next/static
      acl: private
      deleteRemoved: false

    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}/_next/data/
      localDir: .serverless_nextjs/assets/_next/data
      acl: private
      deleteRemoved: false

    - bucketName: belayus-${self:service}-${opt:stage, self:provider.stage}
      bucketPrefix: ${opt:stage, self:provider.stage}
      localDir: env-public-files/${opt:stage, self:provider.stage}/
      acl: private
      deleteRemoved: true

some files of the other configurations will be deleted even if they are configured with deleteRemoved: false.

After i set the last one to false. it works like expected.

I was trying to dig deeper but didn't found the exact issue in your code.

Thanks for checking.

coyoteecd commented 2 years ago

@a-marcel your issue is that the bucket prefix of the last entry includes the bucket entries synced by the previous 3 entries. The deleteRemoved flag is passed as-is to the S3 client, and it does what it's supposed to do (i.e. delete all files that start with bucketPrefix, that are not in the local dir).

The fact that all the uploading is done in parallel explains why the issue occurs for "some" of the files.