Question about config file parameters

sfc-gh-jsander commented 2 years ago

Sorry to misuse this Feature Request to ask a question. Trying to understand the target and bucket configuration items. Everything I have tried results in “Not Found” errors. I’ve tried a large number of configs, and the current one looks like this (it’s an edit of the sample taking out everything except the bare minimum to see if I can solve this problem):

$ cat /home/ubuntu/s3-proxy/conf/NOPE/config-example.yaml
# Log configuration
log:
  # Log level
  level: info
  # Log format
  format: text
  # Log file path
  # filePath:

# Targets map
targets:
  first-bucket:
    ## Mount point
    mount:
      path:
        - /
    ## Bucket configuration
    bucket:
      name: my-bucket
      prefix:
      region: us-east-2
      s3Endpoint: https://<REDACTED>.<REDACTED>.com
      disableSSL: false
      # s3ListMaxKeys: 1000
      credentials:
        accessKey:
          env: AWS_ACCESS_KEY_ID
        secretKey:
          env: AWS_SECRET_ACCESS_KEY

I’m starting the system like this:

$ sudo docker run -d --name s3-proxy-1 -e AWS_SECRET_ACCESS_KEY='<REDACTED>' -e AWS_ACCESS_KEY_ID='<REDACTED>' -p 8080:8080 -p 9090:9090 -v /home/ubuntu/s3-proxy/conf/NOPE:/proxy/conf oxynozeta/s3-proxy

The requests I’m making look like this from another host in the same network:

$ aws --endpoint-url http://10.0.0.224:8080 --profile <REDACTED> s3 ls my-bucket
An error occurred () when calling the ListObjectsV2 operation:<ENDS HERE>

And I get these errors in the logs:

2022-08-04T14:05:31.636209032Z time="2022-08-04T14:05:31Z" level=info msg="Starting s3-proxy version: v4.5.0 (git commit: 53d4768) built on 2022-03-30T19:42:02Z"
2022-08-04T14:05:31.636480759Z time="2022-08-04T14:05:31Z" level=info msg="Load S3 clients for all targets"
2022-08-04T14:05:31.639789233Z time="2022-08-04T14:05:31Z" level=info msg="Internal server listening on :9090"
2022-08-04T14:05:31.658982636Z time="2022-08-04T14:05:31Z" level=info msg="Server listening on :8080"
2022-08-04T14:05:43.716161002Z time="2022-08-04T14:05:43Z" level=info msg="no resource declared => skip authentication" client_ip="10.0.0.87:58670" http_method=GET http_proto=HTTP/1.1 http_scheme=http remote_addr="10.0.0.87:58670" req_id=83592f43e87d/rAqjygfsOK-000001 uri="http://10.0.0.224:8080/my-bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url" user_agent="aws-cli/2.7.15 Python/3.9.11 Linux/5.15.0-1015-aws exe/x86_64.ubuntu.22 prompt/off command/s3.ls"
2022-08-04T14:05:43.838108571Z time="2022-08-04T14:05:43Z" level=error msg="Not Found" client_ip="10.0.0.87:58670" error="Not Found" http_method=GET http_proto=HTTP/1.1 http_scheme=http remote_addr="10.0.0.87:58670" req_id=83592f43e87d/rAqjygfsOK-000001 stack="github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/response-handler.(*handler).NotFoundError,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/response-handler/error-handlers.go:161,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/bucket.(*requestContext).Get,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/bucket/requestContext.go:165,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/server.(*Server).generateRouter.func4.1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/server/server.go:356,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5.(*Mux).routeHTTP,github.com/go-chi/chi/v5@v5.0.7/mux.go:442,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-pr```
oxy/authx/authorization.Middleware.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/authx/authorization/main.go:38,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/authx/authentication.(*service).Middleware.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/authx/authentication/main.go:44,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/bucket.HTTPMiddleware.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/bucket/http-middleware.go:37,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/response-handler.HTTPMiddleware.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/response-handler/http-middleware.go:25,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5.(*Mux).ServeHTTP,github.com/go-chi/chi/v5@v5.0.7/mux.go:71,github.com/go-chi/chi/v5.(*Mux).Mount.func1,github.com/go-chi/chi/v5@v5.0.7/mux.go:314,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5.(*Mux).routeHTTP,github.com/go-chi/chi/v5@v5.0.7/mux.go:442,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5.(*Mux).ServeHTTP,github.com/go-chi/chi/v5@v5.0.7/mux.go:71,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/server.HostRouter.ServeHTTP,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/server/hostrouter.go:65,github.com/go-chi/chi/v5.(*Mux).Mount.func1,github.com/go-chi/chi/v5@v5.0.7/mux.go:314,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5.(*Mux).routeHTTP,github.com/go-chi/chi/v5@v5.0.7/mux.go:442,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5/middleware.Recoverer.func1,github.com/go-chi/chi/v5@v5.0.7/middleware/recoverer.go:38,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/metrics.(*prometheusClient).Instrument.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/metrics/prometheus.go:36,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/log.HTTPAddLoggerToContextMiddleware.func1.1,github.com/oxyno-zeta/s3-proxy/pkg/s3-proxy/log/http-middleware.go:26,net/http.HandlerFunc.ServeHTTP,net/http/server.go:2084,github.com/go-chi/chi/v5/middleware.RequestLogger.func1.1,github.com/go-chi/chi/v5@v5.0.7/middleware/logger.go:57" uri="http://10.0.0.224:8080/my-bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url" user_agent="aws-cli/2.7.15 Python/3.9.11 Linux/5.15.0-1015-aws exe/x86_64.ubuntu.22 prompt/off command/s3.ls"
2022-08-04T14:05:43.844387262Z time="2022-08-04T14:05:43Z" level=error msg="request complete" client_ip="10.0.0.87:58670" http_method=GET http_proto=HTTP/1.1 http_scheme=http remote_addr="10.0.0.87:58670" req_id=83592f43e87d/rAqjygfsOK-000001 resp_bytes_length=83 resp_elapsed_ms=129.027299 resp_status=404 uri="http://10.0.0.224:8080/my-bucket?list-type=2&prefix=&delimiter=%2F&encoding-type=url" user_agent="aws-cli/2.7.15 Python/3.9.11 Linux/5.15.0-1015-aws exe/x86_64.ubuntu.22 prompt/off command/s3.ls"

I can do the ls directly against the bucket from the config on the host where I’m running the proxy. So I know it’s reachable from that place with those credentials by that name (i.e. AWS_ACCESS_KEY_ID='<REDACTED>' AWS_SECRET_ACCESS_KEY=<REDACTED> aws s3 ls my-bucket --endpoint-url 'https://<REDACTED>.<REDACTED>.com/' - is working). I understand this is more than likely a config error on my part, and likely simply me not understanding the relationship between target config and bucket config. I’ve tried everything I can think of to supply different combinations of the values I have without making any progress. I have also tried to configure “normal” S3 buckets (i.e. not hosted on alternate endpoints and instead hosted natively by AWS S3 service), and those fail in exactly the same way. The only difference is I comment out the s3Endpoint in the config.

Thank you for any insight you may be able to offer in advance. And I apologize for the remedial nature of the question.

oxyno-zeta commented 2 years ago

Hello,

I think there is a misunderstood about the project in itself. I'm sorry but it isn't that kind of proxy. S3-Proxy isn't a proxy to the S3 API like a Minio Proxy or something like this. It is a web reverse proxy with S3 storage. That's why you can't use the AWS Cli on S3-Proxy.

Maybe I misunderstood your issue. Feel free to tell me :) .

Regards,

Oxyno-zeta

sfc-gh-jsander commented 2 years ago

Well THAT explains a lot. I'm very sorry to have wasted your time.

oxyno-zeta / s3-proxy

Question about config file parameters #307