Closed cdino closed 11 months ago
skatsaounis
commented
11 months ago This is the line that most probably is causing 401: https://git.launchpad.net/maas/tree/src/maasserver/api/interfaces.py?h=3.3#n218
May I ask you if the API key is coming from a user with admin permissions? If not, could you try with such kind of user and let us know if the 401 persists?
cdino
commented
11 months ago I'm using admin for these tests.
I got the same errors when I tried to use data sources, but it seems that the problem occurs when the MAAS provider tries to retrieve data.
skatsaounis
commented
11 months ago Hi @cdino. As we can see from the access log when Terraform is trying to fetch the interface, there is an initial request with HTTP 1.1 that receives 301 and a subsequent request with HTTP 2 that receives 401. May I ask you if you are running a proxy/load balancer in front of MAAS?
cdino
commented
11 months ago Hi @skatsaounis, yes we have 2 hosts with keepalived and haproxy but currently on the provider I'm pointing directly to one of them.
provider "maas" {
api_version = "2.0"
api_key = "<hidden>"
api_url = "http://maas-reg02:5240/MAAS"
}and checking the HTTP logs of maas (I suppose):
==> /var/snap/maas/common/log/http/access.log <==
x.x.x.x - - [04/Oct/2023:15:56:14 +0200] "GET /MAAS/api/2.0/subnets/ HTTP/1.1" 301 178 "-" "Go-http-client/1.1"
x.x.x.x - - [04/Oct/2023:15:56:14 +0200] "GET /MAAS/api/2.0/fabrics/ HTTP/1.1" 301 178 "-" "Go-http-client/1.1"
x.x.x.x - - [04/Oct/2023:15:56:14 +0200] "GET /MAAS/api/2.0/fabrics/ HTTP/2.0" 401 9 "http://maas-reg02.cscs.ch:5240/MAAS/api/2.0/fabrics/" "Go-http-client/2.0"
x.x.x.x- - [04/Oct/2023:15:56:14 +0200] "GET /MAAS/api/2.0/subnets/ HTTP/2.0" 401 9 "http://maas-reg02.cscs.ch:5240/MAAS/api/2.0/subnets/" "Go-http-client/2.0"These logs are from my last test with data providers:
data "maas_fabric" "default" {
name = "fabric-0"
}
data "maas_vlan" "vlan" {
fabric = data.maas_fabric.default.id
vlan = 340
}
data "maas_subnet" "subnet" {
cidr = "10.10.25.160/27"
}
output "maas_fabric_id" {
value = data.maas_fabric.default.id
}
output "maas_vlan_id" {
value = data.maas_vlan.vlan.id
}
output "maas_subnet_id" {
value = data.maas_subnet.subnet.id
}There is high chance that your haproxy config is causing the issue, assuming that maas-reg02:5240 is your haproxy frontend which is sending requests to MAAS, declared as a haproxy backend. Your haproxy config may have a redirect directive that is producing 301 and informing clients to use HTTP2 for the next request. Could you please share your redacted haproxy config?
cdino
commented
11 months ago Hi, i don't think that the haproxy is involved because maas-reg02:5240 is not defined as frontend but as backend, here the configuration of haproxy:
defaults
retries 3
option redispatch
timeout client 90s
timeout connect 90s
timeout server 90s
frontend maas
bind *:80
option http-server-close
default_backend maas
backend maas
balance source
hash-type consistent
server maas-region-server-01 10.10.10.13:5240 check
server maas-region-server-02 10.10.10.15:5240 checkI'm using maas-reg01 that resolve to 10.10.10.13
To be sure I stopped haproxy and tested again:
○ haproxy.service - HAProxy Load Balancer
Loaded: loaded (/lib/systemd/system/haproxy.service; enabled; vendor preset: enabled)
Active: inactive (dead) since Wed 2023-10-04 17:18:11 CEST; 35s ago
Docs: man:haproxy(1)
file:/usr/share/doc/haproxy/configuration.txt.gz
Process: 981 ExecStart=/usr/sbin/haproxy -Ws -f $CONFIG -p $PIDFILE $EXTRAOPTS (code=exited, status=0/SUCCESS)
Main PID: 981 (code=exited, status=0/SUCCESS)
CPU: 24.964s
Oct 03 15:47:15 maas-reg01 haproxy[992]: [WARNING] (992) : Server maas/maas-region-server-01 is UP, reason: Layer4 check passed, check duration: 0ms. 2 active a>
Oct 04 17:18:11 maas-reg01 systemd[1]: Stopping HAProxy Load Balancer...
Oct 04 17:18:11 maas-reg01 haproxy[981]: [WARNING] (981) : Exiting Master process...
Oct 04 17:18:11 maas-reg01 haproxy[981]: [NOTICE] (981) : haproxy version is 2.4.22-0ubuntu0.22.04.2
Oct 04 17:18:11 maas-reg01 haproxy[981]: [NOTICE] (981) : path to executable is /usr/sbin/haproxy
Oct 04 17:18:11 maas-reg01 haproxy[981]: [ALERT] (981) : Current worker #1 (992) exited with code 143 (Terminated)
Oct 04 17:18:11 maas-reg01 haproxy[981]: [WARNING] (981) : All workers exited. Exiting... (0)
Oct 04 17:18:11 maas-reg01 systemd[1]: haproxy.service: Deactivated successfully.
Oct 04 17:18:11 maas-reg01 systemd[1]: Stopped HAProxy Load Balancer.
Oct 04 17:18:11 maas-reg01 systemd[1]: haproxy.service: Consumed 24.964s CPU time.
Hi, digging around the logs I also found that:
==> /var/snap/maas/common/log/regiond.log <==
2023-10-06 06:17:34 regiond: [info] 127.0.0.1 GET /MAAS/api/2.0/fabrics/ HTTP/1.1 --> 401 UNAUTHORIZED (referrer: http://maas-reg01:5240/MAAS/api/2.0/fabrics/; agent: Go-http-client/2.0)Hi again,
Unfortunately, I am not able to reproduce the 401 in my local MAAS setup. However, I would like you to try something just to confirm that you can stop go client from upgrading to HTTP 2.
Could you please try to use the provider with this variable set GODEBUG=http2client=0 and let me know about the outcome? Source: https://pkg.go.dev/net/http#hdr-HTTP_2
Note that in case the above fixes the 401s it shouldn't be considered a permanent solution since it is sweeping the original problem under the carpet.
cdino
commented
11 months ago We solved the issue by using HAProxy in the front of the two maas-region servers. In this way, the request is proxied locally and the 401 disappeared.
skatsaounis
commented
11 months ago Hi @cdino. I am glad you managed to make it work. Out of curiosity, I want to know the following. When you were trying direct access to the region servers, did you have native TLS enabled on MAAS? While waiting on your reply I was thinking that this could be also the root cause for your 401s.
Being more specific, with HTTPS enabled, MAAS region server nginx config allows HTTP for specific resources, like machines. But since other resource endpoints are redirected to HTTPS, which is set to http2, I would expect redirects, leading to 401s for every resource except machines. This is what you initially reported if I am not mistaken.
cdino
commented
11 months ago Hi @skatsaounis, native TLS is disabled on region servers, and I was doing the request directly on the region server port 5240 in plain HTTP. Now we have HAProxy enabled with HTTPS and trusted certificates, and plain HTTP on the backend.
Your thoughts seem to have sense
skatsaounis
commented
11 months ago I am closing this issue since it has been resolved. In case you are using self signed certificates please be informed that when #101 is released, you will be able to set the CA cert directly to the provider, rather than trusting it at system level.
I am getting an Unauthorized error while configuring the network of a new machine.
Im using MAAS 3.3 Terraform provider 1.3
As suggested in #88 I also tried to regenerate the API key.
Here the debugging info and error:
Here some log i found on the maas node: