search

swift-server / async-http-client

HTTP client library built on SwiftNIO
https://swiftpackageindex.com/swift-server/async-http-client/main/documentation/asynchttpclient
Apache License 2.0
917 stars 118 forks source link

badGateway errors when using AsyncHTTPClient version > 1.6.4 with Vapor 4 project #563

Open pbodsk opened 2 years ago

pbodsk commented 2 years ago

AsyncHTTPClient commit hash:

ec2e080d7011a81bd67f10bf41efe6104d7799d6

Swift Version:

swift-driver version: 1.26.21 Apple Swift version 5.5.2 (swiftlang-1300.0.47.5 clang-1300.0.29.30) Target: x86_64-apple-macosx12.0

Context

We have a Vapor 4 project that has been running in production close to a year now. This project communicates with 3rd party APIs using a Client which has worked fine so far. Vapors Client (EventLoopHTTPClient to be specific) wraps a HTTPClient from AsyncHTTPClient.

Recently I was tasked with implementing some new features and as part of this I updated all dependencies so I could take advantage of async/await.

Issue

After updating Vapor and other dependencies I noticed that the existing calls to 3rd party APIs stopped working. Calling the endpoints would return a .badGateway error instead. I could write this off as errors in the 3rd party API if it was one of them, but when all of them started to fail...that indicated that the error was on my side 😅

Workaround

I managed to get the endpoints working again by manually adding an .exact dependency to async-http-client in my Package.swift file like so:

.package(url: "https://github.com/swift-server/async-http-client.git", .exact("1.6.4"))

If I upgrade to anything above 1.6.4, I start seeing the .badGateway errors on my existing Client calls (which are using EventLoopFutures btw.)

I can see that AsyncHTTPClient 1.7.0 introduces automatic HTTP/2 support by default and I wonder if that has broken something in my setup.

Additional Workaround

As mentioned in this issue, setting the httpVersion manually also does the trick and means that you do not have to stay on 1.6.4.

So, somewhere in your config before you start using the client add this: app.http.client.configuration.httpVersion = .http1Only

Additional Observations

I know that this is probably a case of me doing something wrong somehow but I hope you can assist me in pinpointing where the issue lies 😄

Thank you for your time.

Lukasa commented 2 years ago

Thanks for filing this issue! I think to begin with we'd like to try to see packet captures using tools like Wireshark or tcpdump to compare what's happening in both cases.

pbodsk commented 2 years ago

Hey @Lukasa

First of all, sorry for the late reply! I managed to get something working (mentioned in the "Additional Workaround" section) so the initial fire was put out 😅

You asked for Wireshark files. I've tried...hope it is useful. Attached here you'll find two files (delete the .txt extension to get them as pcapng files 🤷 ):

in both cases I ran Wireshark to capture data and then filtered to only get packets send to the 3rd party API. I furthermore took the liberty of "anonymizing" both my own and the 3rd party API IP addresses (just to say you won't find the 3rd party API at 10.10.10.10 😄 )

I'm a noob when it comes to Wireshark files so I hope this is what you asked for...if not...just let me know and I'll happily make another attempt.

WorkingFiltered.pcapng.txt NotWorkingFiltered.pcapng.txt

Lukasa commented 2 years ago

Hmm, in both cases we're using TLS to connect so the Wireshark trace is not immediately useful to us. @fabianfett do we have a good way to grab the plaintext by inserting a pcap handler?

pbodsk commented 2 years ago

I initially tried using Charles for this but couldn't detect traffic. I suspect it is because the communication done using AsyncHTTPClient is on a lower level in the network stack...maybe...(I'm out of my league here 😉 )

Lukasa commented 2 years ago

I think in the case of Charles you need to manually configure the HTTP proxy. If you told async-http-client to use the Charles proxy directly (by setting the config appropriately) then it should work fine.

pbodsk commented 2 years ago

Hey @Lukasa

Good news and bad news.

That did indeed the trick! Adding these line to my config file

let proxy = AsyncHTTPClient.HTTPClient.Configuration.Proxy.server(host: "127.0.0.1", port: 8888)
app.http.client.configuration.proxy = proxy

Allowed me to see traffic from my Vapor app in Charles 🎉

As part of the setup I also installed a root certificate from Charles on my machine (following this description), allowing me to see SSL traffic.

However (uh oh!)

If I did that...and then removed this line:

app.http.client.configuration.httpVersion = .http1Only

In theory making my app go back to not working....it still worked!

So...going through a proxy and adding a root certificate so I could see the traffic also means that my app works as expected, even thought it shouldn't.

So to recap

Without Proxy

With Proxy and SSL proxying

With Proxy and NO SSL proxying

Observations

Both of the calls return 200 OK it seems in Charles...but I still see badGateway errors 🤔

I've added a couple of screenshots from Charles, don't know if they help you or not

Here's the traffic when running with app.http.client.configuration.httpVersion = .http1Only added

Working

And here's how it looks when just running default (not working)

Not working

Hope it gives you a straw to grasp for 😄

Lukasa commented 2 years ago

The 200 there seems to be the Charles response, not the response from the server. Can we see an actual request running through that proxy?