mrmassis
commented
2 years ago Hi everyone, I have the same problem. Requests return Bad Request 400 when going through a NSX/AVI load balancer. In my pod logs the return is very similar:
7400 TLS:0xc00324b8c0})]]]]
I0119 21:20:37.201341 1 auth.go:49] Using VCD OpenAPI version [36.0]
** REQUEST **
time: 2022-01-19T21:20:37.808Z
method: GET
host: dcvirtual-bsa.hom.serpro
length: 0
URL: https://dcvirtual-bsa.hom.serpro/api/versions
header: [
User-Agent => [go-vcloud-director]
Authorization => [********]
Accept => [application/*+xml;version=36.0]
X-Vmware-Vcloud-Token-Type => [Bearer]
]
payload:
## RESPONSE ##
time: 2022-01-19T21:20:37.938Z
status: 400 - 400 Bad Request
length: 167
header: map[Content-Length:[167] Content-Type:[text/html] Date:[Wed, 19 Jan 2022 21:20:39 GMT] Server:[NSX LB]]
body: <html>
<head><title>400 Bad Request</title></head>
<body bgcolor="white">
<center><h1>400 Bad Request</h1></center>
<hr><center>NSX LB</center>
</body>
</html>
I0119 21:20:37.938734 1 cloud.go:92] Error initializing client from secrets: [unable to get swagger client from secrets: [unable to get bearer token from serets: [failed to set authorization header: [error finding LoginUrl: could not find valid version for login: could not retrieve supported versions: error fetching versions: [ParseErr]: error parsing error body for non-200 request: XML syntax error on line 6: element <hr> closed by </body> (&{Status:400 Bad Request StatusCode:400 Proto:HTTP/1.1 ProtoMajor:1 ProtoMinor:1 Header:map[Content-Length:[167] Content-Type:[text/html] Date:[Wed, 19 Jan 2022 21:20:39 GMT] Server:[NSX LB]] Body:0xc001fe2c00 ContentLength:167 TransferEncoding:[] Close:true Uncompressed:false Trailer:map[] Request:0xc002ef2a00 TLS:0xc00322dad0})]]]]In the NSX/AVI logs I get the error:
022/01/19 15:00:16 [info] 1247#0: *32731028 client sent duplicate header line: "Authorization: bearer Bearer (...)"What we infer here in the lab is that the request is arriving with a duplicate header. That is, two authentication headers. This must be checked and dealt with, we were able to simulate the problem by sending a curl with the same duplicated field. What it seems to me is that when we send it directly to the VCD this duplication is not verified, because it is not necessary to have a token to search for the versions. Now when you have an NSX/AVI in between, a more complete parsing is performed on the message and this problem is identified. Among the possible solutions is the correction of the sent fields.
When we change the load balancer from L7 to L4 the process is fine. The problem is L7, that check headers.
an0nz
Didainius
arunmk
Describe the bug
Requests return Bad Request 400 when going through a Cloudflare load balancer, someone else has it with AVI also but I do not know their config, it is in the bug report on cloud-provider-for-cloud-director
I am getting this issue in 2 downstream projects used for the vcd container service extension (cloud-provider-for-cloud-director and cloud-director-named-disk-csi-driver)
It looks like requests being made by this piece of code get a 400 bad request when multiple vcd cells are behind a Cloudflare L7 Load Balancer or an AVI Load Balancer (Unknown if its L4 or L7)
I have not submitted a report to cloud-director-named-disk-csi-driver as I just discovered that and the common component is go-vcloud-director. If I add hostaliases to go direct to a particular vcd cell bypassing the load balancer everything works correctly. This also works using the same headers via curl within the containers that have this issue.
Expected behavior
Requests succeed with 200 OK
Additional context
No response