Bucket not found - Githubissues

PraveenAsh commented 4 years ago

In the latest version, The bucket name isnt being read unless globally installed. When rolled back to v1.7.2 works just fine.

It directly throws error msg stating,

S3 Sync: Synced metadata.
No Such Bucket -----------------------------------------

Please keep the backward support on. @k1LoW @nibynool

k1LoW commented 4 years ago

Hi @PraveenAsh .

Could you show me serverless.yml (s3Sync: ) ?
What is the latest version that works in your environment? ( 1.7.2 -> OK, 1.7.3 -> NG? )

PraveenAsh commented 4 years ago

serversless.yml :

 s3Sync:
    - bucketName: ${self:custom.siteName}
      bucketPrefix: ${self:custom.stage}
      localDir: static
      params: 
        - randomname.html:
            CacheControl: 'no-cache'

The custom variables exist but don't get picked. But the credentials exist in the environment with aws prefix.

The latest that failed was v1.10.2 but works fine in v1.7.2 OS : Catalina, MacOS

Nemo64 commented 4 years ago

I probably have the same issue. If I remove params it syncs assets fine but without metadata of course.

nibynool commented 4 years ago

@PraveenAsh it would be great to know what the earliest version is that fails and the latest version that works. It helps to narrow down where a code change has had an unexpected impact. At the moment the range you have provided is "somewhere between 1.7.2 and 1.10.2".

1.7.2 was released over 1 year ago, there have been 8 versions released since then, and 1.10.2 was released less than 3 weeks ago.

Are you creating the bucket in your resources? Can you verify the bucket is created on AWS, is the name of the bucket as expected?

nibynool commented 4 years ago

@Nemo64 it sounds like you have a different issue. Please create a new issue for it, but I would suggest checking out #41 as I think that could be the issue you are facing (there's a fix at the end of that issue)

nibynool commented 4 years ago

As a side note to this. I have no formal involvement in this project, but I am happy to assist where I can. If @k1LoW overrides anything I say then please honour his wishes above mine.

Nemo64 commented 4 years ago

@nibynool i'm not sure, i get the same error as @PraveenAsh :

S3 Sync: Syncing directories and S3 prefixes...
.
S3 Sync: Synced.
S3 Sync: Syncing metadata...
[{}]
.
S3 Sync: Synced metadata.

  No Such Bucket -----------------------------------------

  NoSuchBucket: The specified bucket does not exist
      at Request.extractError (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/services/s3.js:816:35)
      at Request.callListeners (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
      at Request.emit (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at Request.emit (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
      at Request.emit (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:683:14)
      at Request.transition (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:22:10)
      at AcceptorStateMachine.runTo (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:14:12)
      at /Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/state_machine.js:26:10
      at Request.<anonymous> (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:38:9)
      at Request.<anonymous> (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/request.js:685:12)
      at Request.callListeners (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:116:18)
      at callNextListener (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/sequential_executor.js:96:12)
      at IncomingMessage.onEnd (/Users/marco/.config/yarn/global/node_modules/aws-sdk/lib/event_listeners.js:307:13)
      at IncomingMessage.emit (events.js:317:22)
      at IncomingMessage.EventEmitter.emit (domain.js:483:12)
      at endReadableNT (_stream_readable.js:1215:12)
      at processTicksAndRejections (internal/process/task_queues.js:84:21)

And I have a very similar setup:

custom:
  s3Sync:
    - bucketName: '${self:service}-${self:provider.stage}-assets'
      localDir: 'public/build/'
      bucketPrefix: 'build/'
      params:
        - "**/????????????????????.*": # hashed webpack resources (which contain exactly 20 hex characters)
            CacheControl: 'public, max-age=31536000, immutable'

nibynool commented 4 years ago

Ahhh, thanks for the extra context @Nemo64. This should be resolved by PR #45 (there was a bug in my code that went into 1.10.2). https://github.com/k1LoW/serverless-s3-sync/issues/41#issuecomment-579708969 should allow you to work around the problem until it is merged and tagged. If it works can you please confirm in this thread so there is some additional verification it solves the issues experienced?

@PraveenAsh, with the additional context from @Nemo64, I think this may also solve your issue.

Nemo64 commented 4 years ago

Nope, doesn't resolve it. It has something to do with params. Just for fun, i used version 1.7.2, like @PraveenAsh said and (kinda) works. It doesn't update but on create it'll set metadata. Then I went though the versions and found that 1.10.0 is the first failing version, 1.9.2 works fine.

nibynool commented 4 years ago

The change between 1.9.2 and 1.10.0 was to add functionality to allow the update on change, so the 1.9.2 version should suffice for now.

I had some issues with 1.10.1 and added a fix for these, but in the process introduced a new bug and that became part of 1.10.2. I have created the PR #45 to fix these, and I believe the fix included the No Such Bucket issue.

~I'm not sure of your level of knowledge with npm and JS, so sorry if this is obvious to you, but to get my latest version (the one the PR is created from) you would need to manually override the reference in package.json to reference the nibynool/serverless-s3-sync repository and the specific commit. Just so I don't invest too much time in this can you please confirm that you tried this?~

Version 1.10.3 has just been released, this includes the changes I just mentioned. Please re-verify with the latest version (just for my peace of mind).

If this doesn't work, can you please try removing the bucketPrefix line from your config and remove the build/ from your localDir definition. If this works, then it gives me a great pointer to where the problem may be (I don't use the bucketPrefix in my config).

PraveenAsh commented 4 years ago

Deployment failing in v1.10.3 too. Back to square one. Just tested.

Same Error :

S3 Sync: Synced metadata.

  No Such Bucket -----------------------------------------

  NoSuchBucket: The specified bucket does not exist

nibynool commented 4 years ago

I think the issue is likely related to the bucketPrefix at this point. I will try to investigate further but may not get time for a few days. If anyone else has some time to look into it, it will be in the syncMetadata function in index.js likely around line 235.

nibynool commented 4 years ago

The current code expects the bucketPrefix to be prepended with a / and not have a trailing one.

Can you try changing the slash from the end of the prefix to the start of it. I will investigate adding better detection for leading and trailing slashes in the prefix if someone doesn't beat me to it.

Nemo64 commented 4 years ago

@nibynool a missing leading slash seems to be the issue 👍

nibynool commented 4 years ago

@PraveenAsh and @Nemo64, I have created PR #50 to fix this. Can you please update your package.json to contain "serverless-s3-sync": "github:nibynool/serverless-s3-sync#bucket-prefix-slashes" and let me know if that fixes the issue (I don't have a test project for this use case at this time).

If it doesn't fix the issue I will create a proper test case and try to resolve it. If it does fix the issue then we can ask @k1LoW to merge the PR (you will need to alter your package.json back to the release version once it has been tagged and released).

PraveenAsh commented 4 years ago

@nibynool works. I have verified it. Good to merge I guess.

Close the issue on review from @Nemo64. @nibynool @k1LoW thanks for the support.

Nemo64 commented 4 years ago

I can confirm, the bucket-prefix-slashes branch fixes it.

PraveenAsh commented 4 years ago

Bug fixed will most likely be available from 1.10.4 onwards(Should be coming out soon).

Closing issue. Reopen when such a time comes.

PraveenAsh commented 4 years ago

Merge the branch and give a new release.

@nibynool @k1LoW

judehunter commented 3 years ago

I just encountered this issue with the bucket prefix option. Can someone reopen this please?

k1LoW / serverless-s3-sync

Bucket not found #48