Localstack compability (local AWS emulator) Issue

juanpicado commented 5 years ago

My reason:

I want to create a docker-example with this plugin and localstack

https://github.com/localstack/localstack

Steps to reproduce:

Without credentials.

App Version:

All

Additional information:

I am aware this plugging was created in order to be used directly on AWS, but I need a way to use it as demonstration with localstack. Also will help to others to create PoC without be forced to have a AWS account.

I had to add my own credentials due this plugin expect will be already available in as env variables or be present in any other way behind the scenes.

AWS.config.update({
      accessKeyId: 'something',
      secretAccessKey: 'something',
      region: 'eu-west-2',
      logger: process.stdout
    });

If I try the use the plugin as it is I got

Attaching to localstack-s3, verdaccio-s3-plugin
localstack-s3    | 2019-02-10 07:24:47,199 CRIT Supervisor running as root (no user in config file)
localstack-s3    | 2019-02-10 07:24:47,201 INFO supervisord started with pid 1
localstack-s3    | 2019-02-10 07:24:48,203 INFO spawned: 'dashboard' with pid 8
localstack-s3    | 2019-02-10 07:24:48,206 INFO spawned: 'infra' with pid 9
localstack-s3    | (. .venv/bin/activate; bin/localstack web)
localstack-s3    | (. .venv/bin/activate; exec bin/localstack start)
verdaccio-s3-plugin |  warn --- config file  - /verdaccio/conf/config.yaml
verdaccio-s3-plugin |  warn -=- this is a plugin example
verdaccio-s3-plugin |  warn -=- this is a config plugin : { 's3-storage-example':
verdaccio-s3-plugin |    { bucket: 'my.unique.bucket.name',
verdaccio-s3-plugin |      keyPrefix: 'some-prefix',
verdaccio-s3-plugin |      region: 'eu-west-2',
verdaccio-s3-plugin |      endpoint: 'http://localstack-s3:5002' } }
verdaccio-s3-plugin |  warn --- Plugin successfully loaded: s3-storage-example
localstack-s3    | Starting local dev environment. CTRL-C to quit.
localstack-s3    | 2019-02-10T07:24:49:INFO:werkzeug:  * Running on http://0.0.0.0:8080/ (Press CTRL+C to quit)
localstack-s3    | 2019-02-10T07:24:49:INFO:werkzeug:  * Restarting with stat
localstack-s3    | 2019-02-10 07:24:49,863 INFO success: dashboard entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
localstack-s3    | 2019-02-10 07:24:49,863 INFO success: infra entered RUNNING state, process has stayed up for > than 1 seconds (startsecs)
localstack-s3    | 2019-02-10T07:24:49:WARNING:werkzeug:  * Debugger is active!
localstack-s3    | 2019-02-10T07:24:49:INFO:werkzeug:  * Debugger PIN: 198-835-854
localstack-s3    | Starting mock S3 (http port 5002)...
localstack-s3    | Ready.2019-02-10T07:24:55:INFO:localstack.utils.persistence: Restored 6 API calls from persistent file: /tmp/localstack/data/s3_api_calls.json
localstack-s3    | 
verdaccio-s3-plugin | [AWS s3 undefined 75.405s 0 retries] getObject({ Bucket: 'my.unique.bucket.name',
verdaccio-s3-plugin |   Key: 'some-prefix/verdaccio-s3-db.json' })
verdaccio-s3-plugin | (node:6) UnhandledPromiseRejectionWarning: Error: connect ECONNREFUSED 169.254.169.254:80
verdaccio-s3-plugin |     at TCPConnectWrap.afterConnect [as oncomplete] (net.js:1163:14)
verdaccio-s3-plugin | (node:6) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
verdaccio-s3-plugin | (node:6) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
verdaccio-s3-plugin exited with code 0
^CERROR: Aborting.

Verdaccio won't be able to continue, it get stuck until a timeout happens and I lose the process.

See -> 169.254.169.254:80

What I'm expecting here is a way to:

Send credentials in another way
Default config settings from outside
Be able to enable logging. Without that I don't see any activity between plugin and AWS

juanpicado commented 5 years ago

Here is my demo https://github.com/verdaccio/docker-examples/tree/master/amazon-s3-docker-example

I managed to bypass the credentials issue with environment variables, but I am having issues on creating the bucket

localstack-s3    | 2019-02-10T16:09:05:INFO:localstack.utils.persistence: Restored 125 API calls from persistent file: /tmp/localstack/data/s3_api_calls.json
localstack-s3    | Ready.
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:09:08] "GET / HTTP/1.1" 200 -
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:09:10] "PUT /test HTTP/1.1" 200 -
aws-resources_1  | make_bucket: test
verdaccio-s3-plugin |  warn --- config file  - /verdaccio/conf/config.yaml
amazon-s3-docker-example_aws-resources_1 exited with code 0
verdaccio-s3-plugin |  warn --- Plugin successfully loaded: s3-storage
verdaccio-s3-plugin | (node:6) UnhandledPromiseRejectionWarning: UnknownEndpoint: Inaccessible host: `test.localstack-s3'. This service may not be available in the `eu-west-2' region.
verdaccio-s3-plugin |     at Request.ENOTFOUND_ERROR (/usr/local/app/node_modules/aws-sdk/lib/event_listeners.js:486:46)
verdaccio-s3-plugin |     at Request.callListeners (/usr/local/app/node_modules/aws-sdk/lib/sequential_executor.js:106:20)
verdaccio-s3-plugin |     at Request.emit (/usr/local/app/node_modules/aws-sdk/lib/sequential_executor.js:78:10)
verdaccio-s3-plugin |     at Request.emit (/usr/local/app/node_modules/aws-sdk/lib/request.js:683:14)
verdaccio-s3-plugin |     at ClientRequest.error (/usr/local/app/node_modules/aws-sdk/lib/event_listeners.js:325:22)
verdaccio-s3-plugin |     at ClientRequest.<anonymous> (/usr/local/app/node_modules/aws-sdk/lib/http/node.js:93:19)
verdaccio-s3-plugin |     at ClientRequest.emit (events.js:182:13)
verdaccio-s3-plugin |     at ClientRequest.EventEmitter.emit (domain.js:442:20)
verdaccio-s3-plugin |     at Socket.socketErrorListener (_http_client.js:382:9)
verdaccio-s3-plugin |     at Socket.emit (events.js:182:13)
verdaccio-s3-plugin | (node:6) UnhandledPromiseRejectionWarning: Unhandled promise rejection. This error originated either by throwing inside of an async function without a catch block, or by rejecting a promise which was not handled with .catch(). (rejection id: 1)
verdaccio-s3-plugin | (node:6) [DEP0018] DeprecationWarning: Unhandled promise rejections are deprecated. In the future, promise rejections that are not handled will terminate the Node.js process with a non-zero exit code.
verdaccio-s3-plugin exited with code 0
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:09:23] "GET / HTTP/1.1" 200 -
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:09:23] "GET / HTTP/1.1" 200 -

but if run afterwards ...

➜ aws --endpoint-url=http://localhost:4572 s3 ls                                                               
2006-02-03 17:45:09 test

I made progress, but still ... issue after issue ...

Perhaps you see something I do not see.

juanpicado commented 5 years ago

Well, it seems bucket cannot use symbols between words, with dots works fine.

store:
  s3-storage:
    bucket: localstack.s3.plugin.test2
    keyPrefix: docker-test-prefix
    region: eu-west-2
    endpoint: http://localstack-s3:4572

Next issue ...

verdaccio-s3-plugin |  info <-- 172.21.0.1 requested 'GET /jquery'
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:31:09] "GET /localstack.s3.plugin.test2/docker-test-prefix/jquery/package.json HTTP/1.1" 200 -
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:31:09] "GET /localstack.s3.plugin.test2/docker-test-prefix/jquery/package.json HTTP/1.1" 200 -
verdaccio-s3-plugin |  debug-=- updating package jquery info
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:31:09] "PUT /localstack.s3.plugin.test2/docker-test-prefix/jquery/package.json HTTP/1.1" 200 -
verdaccio-s3-plugin |  http <-- 200, user: null(172.21.0.1), req: 'GET /jquery', bytes: 0/10255
verdaccio-s3-plugin |  http <-- 200, user: null(172.21.0.1), req: 'GET /jquery', bytes: 0/10255
verdaccio-s3-plugin |  info <-- 172.21.0.1 requested 'GET /jquery/-/jquery-3.3.1.tgz'
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:31:09] "GET /localstack.s3.plugin.test2/docker-test-prefix/jquery/jquery-3.3.1.tgz HTTP/1.1" 404 -
verdaccio-s3-plugin |  http <-- 404, user: null(172.21.0.1), req: 'GET /jquery/-/jquery-3.3.1.tgz', error: no such file available
verdaccio-s3-plugin |  http <-- 404, user: null(172.21.0.1), req: 'GET /jquery/-/jquery-3.3.1.tgz', error: no such file available
localstack-s3    | 127.0.0.1 - - [10/Feb/2019 16:31:10] "GET / HTTP/1.1" 200 -

Metadata is persisted, but no tarballs. Always 404.

➜ aws --endpoint-url=http://localhost:4572 s3 ls s3://localstack.s3.plugin.test2 --recursive --human-readable --summarize
2019-02-10 17:31:09  139.7 KiB docker-test-prefix/jquery/package.json
2019-02-10 17:32:40   14.6 KiB docker-test-prefix/loose-envify/package.json
2019-02-10 17:32:40   34.9 KiB docker-test-prefix/object-assign/package.json
2019-02-10 17:32:40   65.2 KiB docker-test-prefix/prop-types/package.json
2019-02-10 17:32:40  478.5 KiB docker-test-prefix/react/package.json
2019-02-10 17:32:40  131.3 KiB docker-test-prefix/scheduler/package.json
2019-02-10 17:29:56   87 Bytes docker-test-prefix/verdaccio-s3-db.json

juanpicado commented 5 years ago

This https://github.com/Remitly/verdaccio-s3-storage/pull/19 is causing my 404 error. Apply my patch finally I got full data in my bucket.

➜ aws --endpoint-url=http://localhost:4572 s3 ls s3://localstack.s3.plugin.test --recursive --human-readable --summarize
2019-02-11 22:03:09  405.6 KiB docker-test-prefix/jquery/jquery-3.3.1.tgz
2019-02-11 22:03:09  139.9 KiB docker-test-prefix/jquery/package.json
2019-02-11 22:11:49    6.4 KiB docker-test-prefix/js-tokens/js-tokens-4.0.0.tgz
2019-02-11 22:11:49   35.3 KiB docker-test-prefix/js-tokens/package.json
2019-02-11 22:11:48    2.8 KiB docker-test-prefix/loose-envify/loose-envify-1.4.0.tgz
2019-02-11 22:11:48   14.7 KiB docker-test-prefix/loose-envify/package.json
2019-02-11 22:11:48    2.6 KiB docker-test-prefix/object-assign/object-assign-4.1.1.tgz
2019-02-11 22:11:48   35.1 KiB docker-test-prefix/object-assign/package.json
2019-02-11 22:11:51   76.1 KiB docker-test-prefix/prop-types/package.json
2019-02-11 22:11:50   20.2 KiB docker-test-prefix/prop-types/prop-types-15.7.1.tgz
2019-02-11 22:11:50  145.8 KiB docker-test-prefix/react-is/package.json
2019-02-11 22:11:50    5.4 KiB docker-test-prefix/react-is/react-is-16.8.1.tgz
2019-02-11 22:11:50  480.0 KiB docker-test-prefix/react/package.json
2019-02-11 22:11:50   52.7 KiB docker-test-prefix/react/react-16.8.1.tgz
2019-02-11 22:11:51  131.7 KiB docker-test-prefix/scheduler/package.json
2019-02-11 22:11:51   12.4 KiB docker-test-prefix/scheduler/scheduler-0.13.1.tgz
2019-02-11 21:50:28  689.8 KiB docker-test-prefix/types-registry/package.json
2019-02-11 22:12:11   87 Bytes docker-test-prefix/verdaccio-s3-db.json
2019-02-11 22:12:23 1022.9 KiB docker-test-prefix/vue/package.json

Total Objects: 19
   Total Size: 3.2 MiB

Remitly / verdaccio-s3-storage