API 401 responses don't have a WWW-Authenticate header

[ianw]

Description

Hello,

This is related to another issue https://github.com/go-gitea/gitea/issues/24159 I opened about the api/v1/org call becoming authenticated with 1.19.2

As noted in that report, we have CI that tests our 1.18->1.19 transition. As part of this we use an Ansible URI call to get the org list

- name: Get list of orgs
  uri:
    url: 'https://localhost:3000/api/v1/orgs'
    method: GET

As noted in the other report, we've had to add user/password authentication to this for 1.19. However, it was surprising to find this didn't work when we simply added user: and password: fields to the call to do Basic auth. We found that setting the ansible force_basic_auth flag did make this work -- what this does is send the Basic authentication header in the first request, instead of responding to a 401. This uses Python urllib under the hood, so it seemed unlikely it would be an issue there.

So pulling on that thread some more, it seems that 401 responses from gitea don't include a WWW-Authenticate header to instruct the client on how it can authenticate. You can see in curl

$ curl -v  https://try.gitea.io/api/v1/orgs
...
> GET /api/v1/orgs HTTP/2
> Host: try.gitea.io
> user-agent: curl/7.85.0
> accept: */*
> 
< HTTP/2 401 
< alt-svc: h3=":443"; ma=2592000
< cache-control: max-age=0, private, must-revalidate, no-transform
< content-type: application/json;charset=utf-8
< date: Mon, 17 Apr 2023 02:48:25 GMT
< server: Caddy
< x-content-type-options: nosniff
< x-frame-options: SAMEORIGIN
< content-length: 73
< 
{"message":"token is required","url":"https://try.gitea.io/api/swagger"}

It's a real pain to replicate with an authenticated ssl call in the low-level urllib. The following program is essentially what Ansible is doing, and should make the API call, but it will fail to authenticate.

import http
import ssl
import urllib
import urllib.request

ssl._create_default_https_context = ssl._create_unverified_context

# https://stackoverflow.com/a/74416575
# python3.10 broke setting debuglevel=1 :(
https_old_init = urllib.request.HTTPSHandler.__init__

def https_new_init(self, debuglevel=None, context=None, check_hostname=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    https_old_init(self, debuglevel, context, check_hostname)

urllib.request.HTTPSHandler.__init__ = https_new_init

http_old_init = urllib.request.HTTPHandler.__init__

def http_new_init(self, debuglevel=None):
    debuglevel = debuglevel if debuglevel is not None else http.client.HTTPSConnection.debuglevel
    http_old_init(self, debuglevel)

urllib.request.HTTPHandler.__init__ = http_new_init
# end monkey patch

http.client.HTTPConnection.debuglevel = 1

url = 'https://try.gitea.io/api/v1/orgs'

auth_user = <user>
auth_passwd = <password>

passman = urllib.request.HTTPPasswordMgrWithDefaultRealm()
passman.add_password(None, url, auth_user, auth_passwd)
authhandler = urllib.request.HTTPBasicAuthHandler(passman)
opener = urllib.request.build_opener(authhandler, urllib.request.HTTPHandler(debuglevel=1))
urllib.request.install_opener(opener)

with urllib.request.urlopen(url) as response:
    print(response.read())

I think the problem here is without WWW-Authenticate response, urllib doesn't know to respond to the 401 call.

The same thing happens on our production 1.18.5 system; but it was the authentication discussed in with https://github.com/go-gitea/gitea/issues/24159 that made me look at this a bit further.

[1] https://docs.ansible.com/ansible/latest/collections/ansible/builtin/uri_module.html#parameter-force_basic_auth

Gitea Version

1.19.1

Can you reproduce the bug on the Gitea demo site?

Yes

Log Gist

No response

Screenshots

No response

Git Version

No response

Operating System

No response

How are you running Gitea?

We build from upstream and run in a container

Database

None

[yp05327]

ps: Get api/v1/orgs will not response 401 now, as backport #24259 has been merged 4 days ago. (after this issue created)

You are right, I found document here: https://developer.mozilla.org/en-US/docs/Web/HTTP/Status/401

This status code is sent with an HTTP WWW-Authenticate response header that contains information on how the client can request for the resource again after prompting the user for authentication credentials.

It seems that 401 response should have WWW-Authenticate header. But if search http.StatusUnauthorized, you will find that almost all of them have no WWW-Authenticate header.

[lunny]

WWW-Authenticate is for basic auth but not all 401 requires a basic auth?

[ianw]

WWW-Authenticate is for basic auth but not all 401 requires a basic auth?

https://www.rfc-editor.org/rfc/rfc7235#section-3.1 says

The 401 (Unauthorized) status code indicates that the request has not
been applied because it lacks valid authentication credentials for
the target resource.  The server generating a 401 response MUST send
a WWW-Authenticate header field (Section 4.1) containing at least one
challenge applicable to the target resource.

If the request included authentication credentials, then the 401
response indicates that authorization has been refused for those
credentials. 

As I read that, a blank request would return a WWW-Authenticate: Basic realm="gitea", Bearer as basic auth and bearer auth are supported. From what I read, it seems like if the authentication token is provided, but incorrect, you should still send back 401, but with a message indicating the provided auth was not accepted.

Really the first part is I guess the most important, because as noted the Ansible uri helper for example doesn't, by default do "preauth" where it sends the basic auth token with the initial request -- it seems to be waiting for the 401 response to tell it what to do.

[powerpaul17]

Is this still considered? Because I stumbled upon the problem when using git with http authentication. Apparently the git binary expects the WWW-Authenticate: Basic header before sending the Authorization header with credentials included in the remote url.

Update: Sorry, I had a configuration error with some middleware, everything is working as expected.