search

nginxinc / nginx-s3-gateway

NGINX S3 Caching Gateway
Apache License 2.0
496 stars 126 forks source link

Huawei OBS Support #199

Closed ehginanjar closed 7 months ago

ehginanjar commented 8 months ago

Hi,

Similar to this case, I want to ask whether this nginx-s3-gateway support the Huawei OBS? I build the OSS image, prepare the envs, then running it on Mac M1 Pro. I don't see any errors, so I assume credential check and everything work well. This is so far what I have done

  1. Clone the github repo.
  2. Build the image docker buildx build --platform linux/arm64 -f Dockerfile.oss -t nginx-plus-s3-gateway .
  3. Prepare the env 
    ALLOW_DIRECTORY_LIST=true
AWS_ACCESS_KEY_ID=s3accesskeyid
AWS_SECRET_ACCESS_KEY=s3secretaccesskey
DEBUG=true
S3_BUCKET_NAME=s3bucketname
S3_REGION=s3region
S3_SERVER_PORT=443 
S3_SERVER_PROTO=https
S3_SERVER=obs.s3region.vb.cloudname.com
S3_STYLE=path
AWS_SIGS_VERSION=2
PROVIDE_INDEX_PAGE=1
  4. This is the log 
    $ docker run -it --env-file ./s3-settings.env -p 80:80 nginx-plus-s3-gateway
/docker-entrypoint.sh: /docker-entrypoint.d/ is not empty, will attempt to perform configuration
/docker-entrypoint.sh: Looking for shell scripts in /docker-entrypoint.d/
/docker-entrypoint.sh: Launching /docker-entrypoint.d/00-check-for-required-env.sh
S3 Backend Environment
Access Key ID: s3accesskeyid
Origin: https://s3bucketname.obs.s3region.vb.cloudname.com:443
Region: s3region
Addressing Style: path
AWS Signatures Version: v2
DNS Resolvers:  someip
Directory Listing Enabled: true
Directory Listing Path Prefix:
Provide Index Pages Enabled: 1
Append slash for directory enabled:
Stripping the following headers from responses: x-amz-;
CORS Enabled: 0
/docker-entrypoint.sh: Launching /docker-entrypoint.d/10-listen-on-ipv6-by-default.sh
10-listen-on-ipv6-by-default.sh: info: Getting the checksum of /etc/nginx/conf.d/default.conf
10-listen-on-ipv6-by-default.sh: info: Enabled listen on IPv6 in /etc/nginx/conf.d/default.conf
/docker-entrypoint.sh: Ignoring /docker-entrypoint.d/15-local-resolvers.envsh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/20-envsubst-on-templates.sh
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_headers.conf.template to /etc/nginx/conf.d/gateway/v4_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_headers.conf.template to /etc/nginx/conf.d/gateway/v2_headers.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/cors.conf.template to /etc/nginx/conf.d/gateway/cors.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_location.conf.template to /etc/nginx/conf.d/gateway/s3_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v2_js_vars.conf.template to /etc/nginx/conf.d/gateway/v2_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/v4_js_vars.conf.template to /etc/nginx/conf.d/gateway/v4_js_vars.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3_server.conf.template to /etc/nginx/conf.d/gateway/s3_server.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/s3listing_location.conf.template to /etc/nginx/conf.d/gateway/s3listing_location.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/gateway/js_fetch_trusted_certificate.conf.template to /etc/nginx/conf.d/gateway/js_fetch_trusted_certificate.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/cache.conf.template to /etc/nginx/conf.d/cache.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/default.conf.template to /etc/nginx/conf.d/default.conf
20-envsubst-on-templates.sh: Running envsubst on /etc/nginx/templates/upstreams.conf.template to /etc/nginx/conf.d/upstreams.conf
/docker-entrypoint.sh: Launching /docker-entrypoint.d/22-enable_js_fetch_trusted_certificate.sh
/docker-entrypoint.sh: Launching /docker-entrypoint.d/30-tune-worker-processes.sh
/docker-entrypoint.sh: Configuration complete; ready for start up
2024/01/11 12:16:54 [notice] 1#1: using the "epoll" event method
2024/01/11 12:16:54 [notice] 1#1: nginx/1.25.3
2024/01/11 12:16:54 [notice] 1#1: built by gcc 12.2.0 (Debian 12.2.0-14)
2024/01/11 12:16:54 [notice] 1#1: OS: Linux 5.15.49-linuxkit-pr
2024/01/11 12:16:54 [notice] 1#1: getrlimit(RLIMIT_NOFILE): 1048576:1048576
2024/01/11 12:16:54 [notice] 1#1: start worker processes
2024/01/11 12:16:54 [notice] 1#1: start worker process 75
2024/01/11 12:16:54 [notice] 1#1: start cache manager process 76
2024/01/11 12:16:54 [notice] 1#1: start cache loader process 77
  5. Test some urls 
    
# CLIENT SIDE
$ curl http://localhost
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>

SERVER SIDE

/s3bucketname/] 172.17.0.1 - - [11/Jan/2024:12:23:45 +0000] "GET / HTTP/1.1" 404 146 "-" "curl/7.77.0" "-" 2024/01/11 12:23:45 [info] 74#74: *5 client 172.17.0.1 closed keepalive connection 2024/01/11 12:23:46 [notice] 76#76: http file cache: /var/cache/nginx/s3_proxy 0.000M, bsize: 4096 2024/01/11 12:23:46 [notice] 1#1: signal 17 (SIGCHLD) received from 76 2024/01/11 12:23:46 [notice] 1#1: cache loader process 76 exited with code 0 2024/01/11 12:23:46 [notice] 1#1: signal 29 (SIGIO) received



Does nginx-s3-gateway only work in an AWS environment for S3? In the documentation, however, it's mentioned that it works for all S3-compatible object storage systems, so I assume it includes [Huawei OBS](https://www.huaweicloud.com/intl/en-us/product/obs.html) as well.

Any help will be highly appreciated.
dekobon commented 8 months ago

Hi @ehginanjar, the nginx-s3-gateway works on many different S3 compatible object stores and not just AWS. In fact, the test cases for it use Minio. It appears that OBS on its documentation claims it supports the S3 interface. You may want to tweak some of the following settings to see if it works: Set AWS_SIGS_VERSION=4 Set S3_STYLE=virtual Then double check the values you gave for region and bucket.

Also, it may be good to run the test.sh script to make sure that the image you have built locally passes the tests.

ehginanjar commented 7 months ago

hi @dekobon, sorry I just get back to work on this matter. In my case, I just need to set S3_STYLE to virtual and it works. Thank you, appreciate it.