search

scality / cloudserver

Zenko CloudServer, an open-source Node.js implementation of the Amazon S3 protocol on the front-end and backend storage capabilities to multiple clouds, including Azure and Google.
https://www.zenko.io/cloudserver
Apache License 2.0
1.7k stars 240 forks source link

Create directories from environment variables #904

Open datamattsson opened 7 years ago

datamattsson commented 7 years ago

Description

I'm using the scality/s3server image and I find the behavior of the S3METADATAPATH and S3DATAPATH environment variables not following common patterns for container images.

Steps to reproduce the issue

$ docker run --rm -v export:/export -e S3DATAPATH=/export/localData -e S3METADATAPATH=/export/localMetaData scality/s3server

Actual result

Output:

npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info lifecycle s3@7.0.0~prestart: s3@7.0.0
npm info lifecycle s3@7.0.0~start: s3@7.0.0

> s3@7.0.0 start /usr/src/app
> npm-run-all --parallel start_dmd start_s3server

npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info lifecycle s3@7.0.0~prestart_dmd: s3@7.0.0
npm info lifecycle s3@7.0.0~prestart_s3server: s3@7.0.0
npm info lifecycle s3@7.0.0~start_dmd: s3@7.0.0
npm info lifecycle s3@7.0.0~start_s3server: s3@7.0.0

> s3@7.0.0 start_dmd /usr/src/app
> npm-run-all --parallel start_mdserver start_dataserver

> s3@7.0.0 start_s3server /usr/src/app
> node index.js

npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info it worked if it ends with ok
npm info using npm@3.10.10
npm info using node@v6.11.2
npm info lifecycle s3@7.0.0~prestart_mdserver: s3@7.0.0
npm info lifecycle s3@7.0.0~start_mdserver: s3@7.0.0

> s3@7.0.0 start_mdserver /usr/src/app
> node mdserver.js

npm info lifecycle s3@7.0.0~prestart_dataserver: s3@7.0.0
npm info lifecycle s3@7.0.0~start_dataserver: s3@7.0.0

> s3@7.0.0 start_dataserver /usr/src/app
> node dataserver.js

{"name":"S3","time":1505845693588,"level":"warn","message":"scality kms unavailable. Using file kms backend unless mem specified.","hostname":"fb0c354091d0","pid":49}
{"name":"MetadataFileClient","time":1505845693599,"level":"info","message":"connecting to metadata service at http://localhost:9990/MDFile/metadata","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693773,"workerId":1,"workerPid":102,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693779,"workerId":2,"workerPid":108,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693796,"workerId":3,"workerPid":109,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693822,"workerId":4,"workerPid":120,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693831,"workerId":5,"workerPid":126,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693843,"workerId":6,"workerPid":132,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693854,"workerId":7,"workerPid":133,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693880,"workerId":8,"workerPid":139,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693901,"workerId":9,"workerPid":150,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
{"name":"S3","time":1505845693914,"workerId":10,"workerPid":156,"level":"info","message":"new worker forked","hostname":"fb0c354091d0","pid":49}
fs.js:249
  binding.access(pathModule._makeLong(path), mode);
          ^

Error: ENOENT: no such file or directory, access '/export/localMetaData'
    at Error (native)
    at Object.fs.accessSync (fs.js:249:11)
    at MetadataFileServer.startServer (/usr/src/app/node_modules/arsenal/lib/storage/metadata/file/MetadataFileServer.js:124:12)
    at Object.<anonymous> (/usr/src/app/mdserver.js:17:14)
    at Module._compile (module.js:570:32)
    at Object.Module._extensions..js (module.js:579:10)
    at Module.load (module.js:487:32)
    at tryModuleLoad (module.js:446:12)
    at Function.Module._load (module.js:438:3)
    at Module.runMain (module.js:604:10)

npm info lifecycle s3@7.0.0~start_mdserver: Failed to exec start_mdserver script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "run" "start_mdserver"
npm ERR! node v6.11.2
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! s3@7.0.0 start_mdserver: `node mdserver.js`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the s3@7.0.0 start_mdserver script 'node mdserver.js'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     node mdserver.js
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls s3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
{"name":"DataFileStore","time":1505845694281,"error":{"errno":-2,"code":"ENOENT","syscall":"access","path":"/export/localData"},"level":"error","message":"Data path is not readable or writable","hostname":"fb0c354091d0","pid":92}
{"name":"S3","time":1505845694282,"error":{"errno":-2,"code":"ENOENT","syscall":"access","path":"/export/localData"},"level":"error","message":"Error initializing REST data server","hostname":"fb0c354091d0","pid":92}
npm info lifecycle s3@7.0.0~poststart_dataserver: s3@7.0.0
npm info ok 
ERROR: "start_mdserver" exited with 1.

npm info lifecycle s3@7.0.0~start_dmd: Failed to exec start_dmd script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/lib/node_modules/npm/bin/npm-cli.js" "run" "start_dmd"
npm ERR! node v6.11.2
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! s3@7.0.0 start_dmd: `npm-run-all --parallel start_mdserver start_dataserver`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the s3@7.0.0 start_dmd script 'npm-run-all --parallel start_mdserver start_dataserver'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm-run-all --parallel start_mdserver start_dataserver
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls s3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log
ERROR: "start_dmd" exited with 1.

npm info lifecycle s3@7.0.0~start: Failed to exec start script
npm ERR! Linux 4.9.41-moby
npm ERR! argv "/usr/local/bin/node" "/usr/local/bin/npm" "start"
npm ERR! node v6.11.2
npm ERR! npm  v3.10.10
npm ERR! code ELIFECYCLE
npm ERR! s3@7.0.0 start: `npm-run-all --parallel start_dmd start_s3server`
npm ERR! Exit status 1
npm ERR! 
npm ERR! Failed at the s3@7.0.0 start script 'npm-run-all --parallel start_dmd start_s3server'.
npm ERR! Make sure you have the latest version of node.js and npm installed.
npm ERR! If you do, this is most likely a problem with the s3 package,
npm ERR! not with npm itself.
npm ERR! Tell the author that this fails on your system:
npm ERR!     npm-run-all --parallel start_dmd start_s3server
npm ERR! You can get information on how to open an issue for this project with:
npm ERR!     npm bugs s3
npm ERR! Or if that isn't available, you can get their info via:
npm ERR!     npm owner ls s3
npm ERR! There is likely additional logging output above.

npm ERR! Please include the following file with any support request:
npm ERR!     /usr/src/app/npm-debug.log

Expected result

Docker entrypoint script creating the non-existing directories mentioned in the output:

Error: ENOENT: no such file or directory, access '/export/localMetaData'

Additional information: (Node.js version, Docker version, etc)

$ docker version
Client:
 Version:      17.06.2-ce
 API version:  1.30
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 20:12:06 2017
 OS/Arch:      darwin/amd64

Server:
 Version:      17.06.2-ce
 API version:  1.30 (minimum version 1.12)
 Go version:   go1.8.3
 Git commit:   cec0b72
 Built:        Tue Sep  5 19:59:19 2017
 OS/Arch:      linux/amd64
 Experimental: false

Feature Request

Create the directories specified in the environment variables if they are set and are not already created.

Use case

The common patterns found with other container images is to be smarter about these things and not force the user to customize the entrypoint.

LaurenSpiegel commented 7 years ago

Thanks for the input! Since these directories will contain persistent data, our thought was that if people did not want to go with the default localData/localMetadata, they would want to set up the directories themselves, but we can revisit that approach.

datamattsson commented 7 years ago

Thank you for your reply. Since localData and localMetaData are shipped empty, they should be created by the entrypoint script, taking the environment variables into account.

$ docker run --rm -it scality/s3server bash -c "ls -al local*"
localData:
total 12
drwxr-xr-x 2 root root 4096 Sep 21 20:38 .
drwxr-xr-x 1 root root 4096 Aug 15 19:52 ..
-rw-r--r-- 1 root root   71 Aug 15 19:51 .gitignore

localMetadata:
total 12
drwxr-xr-x 2 root root 4096 Sep 21 20:38 .
drwxr-xr-x 1 root root 4096 Aug 15 19:52 ..
-rw-r--r-- 1 root root   71 Aug 15 19:51 .gitignore