Authentication Broken on Insights QA Test Environment

[ironfroggy]

Authentication appears to be broken in some way on the QA test environment. With a valid user authenticated via 3scale, API requests are authenticated and passed to the API properly, but galaxy-api itself seems to respond with a 403 to the valid request.

This happens with requests made from the UI for a logged in user. It also happens with requests from the CLI trying to publish a collection.

When trying to publish a collection, it seems to both save the artifact (because the upload payload was allowed) but rejects actually creating the collection version or handing back a proper response, with galaxy-api determining the request is not authenticated.

Every failed 403 response has the same body: {"detail":"Invalid username/password."}

[alikins]

This isn't super useful, but for reference:

"Invalid username/password" and the 403 is a generic auth error/exception from django-rest-framework

https://github.com/encode/django-rest-framework/blob/master/rest_framework/authentication.py#L101

[alikins]

And in rest_framework.authentication.BasicAuthentication.authenticate_credientials, the "Invalid username/password" error is raised if django.contrib.auth.authenticate returns None

django.contrib.auth.authenticate https://github.com/django/django/blob/master/django/contrib/auth/__init__.py#L61

But it isn't particular clear to me how django.contrib.auth.authenticate could return None (but I need to verify against older versions).

[alikins]

Another thing that is unclear to me:

If the "Invalid username/password" stuff is coming from drf in galaxy-api or in pulp*. My suspicion is that it is coming from pulp and getting passed back. I feel like pulp can return that if the galaxy-api config isn't correct and isn't using the right user/pass for the galaxy_pulp REST requests to pulp_ansible

[awcrosby]

In case it is helpful, hitting /api/automation-hub/v3/_ui/namespaces/[any_namespace]/ in the browser results in these

pulp-api pulp: django.request:WARNING: Forbidden: /pulp/api/v3/content/ansible/collection_versions/
galaxy-api 2020-03-11 13:48:48,152 WARNING django.request: Forbidden: /api/automation-hub/v3/_ui/collections/

[cutwater]

Checked pulp-api logs. It responds with 403 Forbidden.

[alikins]

(Possible shot in the dark)

In https://github.com/ansible/galaxy-api/blob/a52cb7bb0b1de4be39f20845e14669ae12bd37aa/galaxy_api/common/pulp.py#L7

It looks like the config may "hardcode" http scheme? Is it possible QA galaxy-api -> pulp REST API requires https? (Haven't dug into details yet, so take with a grain of salt)

[ironfroggy]

Everything is working now after @cutwater created a missing admin user. 👍