ngx_http_aws_auth_canonical_request: more than one request buf, client

[slushpuppy]

Hello, I receive this error in my logs and stack trace, what measures should I take to start debugging and resolving this issue? $aws_token is null

location /s3/ {

            add_header X-debug-message "jwt debug: $basename:$aws_token:$aws_auth_date" always;

        proxy_set_header X-Amz-Date $aws_auth_date;
    proxy_set_header X-Amz-Content-SHA256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; # no body
    # proxy_set_header X-Amz-Security-Token 23HYTMLJluxQL...;
    proxy_set_header Authorization $aws_token;

         proxy_pass http://minio_s3;

[error] 42646#42646: *1 ngx_http_aws_auth_canonical_request: more than one request buf, client: 127.0.0.1, server: s3.public.localhost

[erankor]

can you provide more details? full nginx conf, the request you are making etc. i would guess from the error that you are sending a request with a body (e.g. PUT / POST) - this is not supported with header authentication, but it may work with a presigned URL (didn't try...). If the required request body sizes are not too big, it may be possible to add support for it, if proxy_request_buffering is enabled.

[slushpuppy]

thanks for the help. attached is my config file

map $request_uri $basename {
    ~(?<captured_basename>.+)(/.+?)$ '["$captured_basename"]';
}

aws_auth $aws_token {
    access_key xxxx;
    secret_key yyyy;
    service s3;
    region us-east-1;
}
server {
        listen 9001;
server_name s3.public.localhost;
   # Allow special characters in headers
   ignore_invalid_headers off;
   # Allow any size file to be uploaded.
   # Set to a value such as 1000m; to restrict file size to a specific value
   # Disable buffering
   proxy_buffering off;
   location /documents/ {
    proxy_set_header Authorization $aws_token;

         proxy_pass http://minio_s3;

    auth_jwt_key_file conf.d/jwtKey.json;
    auth_jwt "" token=$args;

            add_header X-debug-message "jwt debug: $basename:$aws_token:$aws_auth_date" always;
        #add_header X-debug-message "A static file was served: $args" always;
        proxy_set_header X-Amz-Date $aws_auth_date;
    proxy_set_header X-Amz-Content-SHA256 e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855; # no body
    # proxy_set_header X-Amz-Security-Token 23HYTMLJluxQL...;
        }
}

Object fetch GET request:

http://s3.public.localhost:9001/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/test.pdf?eyJ0eXAiOiJKV1QiLCJhbGciOiJIUzI1NiIsImtpZCI6InRlc3QxIn0.eyJpc3MiOiJodHRwczovL3Rlc3QxLmlzc3Vlci5leGFtcGxlLmNvbSIsInN1YiI6InRlc3QxLmlkZW50aWZpZXIiLCJhdWQiOiJ0ZXN0MS5hdWRpZW5jZS5leGFtcGxlLmNvbSIsImV4cCI6IDQxMzM4NjIwMDAsImlhdCI6IDE2NjI1MTIyODYsImVtYWlsIjoidGVzdDFAZXhhbXBsZS5jb20ifQ.2b2m62IaWeY971ofeZuk7CsaG1RhM3Vukp5xSYGt3ak

[erankor]

so just to clarify, you're saying you get this error on a GET request? can you paste the full error log line? i don't see anything special in this config that would explain it... the only thing that is maybe non-standard is the jwt auth, can you try to remove it and see if it changes anything? are you using the open source version or the nginx commercial version of jwt auth?

[slushpuppy]

thanks again for your help and patience. I commented out the jwt auth requirement accordingly

2024/07/13 23:01:37 [error] 93686#93686: *1 ngx_http_aws_auth_canonical_request: more than one request buf, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/test.pdf HTTP/1.1", host: "s3.public.localhost:9001"
2024/07/13 23:01:37 [error] 93686#93686: *1 ngx_http_aws_auth_canonical_request: more than one request buf while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/test.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/test.pdf", host: "s3.public.localhost:9001"

not sure if this would help- I am using wsl2 environment

[erankor]

that's very weird... nginx's proxy module builds the request on single buffer. would be interesting to look at the contents of the buffers, let's try to add this block right before the error is printed -

    if (u->request_bufs->next != NULL) {
        ngx_chain_t *cl;
        for (cl = u->request_bufs; cl != NULL; cl = cl->next) {
            ngx_str_t cur;

            cur.data = cl->buf->pos;
            cur.len = cl->buf->last - cl->buf->pos;

            ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
                "len=%uz, data=%V", cur.len, &cur);
        }

        ngx_log_error(NGX_LOG_ERR, r->connection->log, 0,
            "ngx_http_aws_auth_canonical_request: more than one request buf");
        return NGX_ERROR;
    }

it will print the content of the buffers to the error log, hopefully this would give some hint (there is a limit to the length of lines printed to the error log, i hope we'll still be able to understand...)

[slushpuppy]

2024/07/14 00:57:59 [error] 29062#29062: *1 len=330, data=GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.0
X-Amz-Date: 20240713T165759Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: minio_s3
Connection: close
Content-Length: 0
User-Agent: insomnia/2023.5.8
Accept: */*

, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 len=0, data=, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 ngx_http_aws_auth_canonical_request: more than one request buf, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 len=330, data=GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.0
X-Amz-Date: 20240713T165759Z
X-Amz-Content-SHA256: e3b0c44298fc1c149afbf4c8996fb92427ae41e4649b934ca495991b7852b855
Host: minio_s3
Connection: close
Content-Length: 0
User-Agent: insomnia/2023.5.8
Accept: */*

 while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 len=0, data= while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 len=0, data= while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 len=0, data= while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf", host: "s3.public.localhost:9001"
2024/07/14 00:57:59 [error] 29062#29062: *1 ngx_http_aws_auth_canonical_request: more than one request buf while reading response header from upstream, client: 127.0.0.1, server: s3.public.localhost, request: "GET /documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf HTTP/1.1", upstream: "http://127.0.0.1:9000/documents/bC9xYUhudDM4YXpzK1UrZVFzY0h0Ym1yLnJkVlVGQjZObWxOZks3VEdwemVRRjVzeXJocUJJamtr/temp1.pdf", host: "s3.public.localhost:9001"

nginx -v
nginx version: nginx/1.26.1

thanks again so much for helping

[erankor]

looks like there are some empty bufs after the main request buf, i don't really understand why, maybe you have some non-standard nginx module that causes it... but anyway, try #23 it should solve it.

[slushpuppy]

thank you so very much! confirmed fixed

[vaidyasr]

I have the same issue with my requirements. It happens only for PUT requests. How to fix it?.

[erankor]

requests with body (PUT/POST) are not supported, since the entire body has to be read (and hashed) before the request can be sent to the upstream server.