search

HtmlUnit / htmlunit

HtmlUnit is a "GUI-Less browser for Java programs".
https://www.htmlunit.org
Apache License 2.0
875 stars 172 forks source link

Support for HTTP/2 #370

Open jowerner opened 3 years ago

jowerner commented 3 years ago

Nowadays more and more servers talk HTTP/2. Would be great if HtmlUnit would support that protocol as well. Do you have plans to add that feature to HtmlUnit in the near future, now that Apache HttpClient 5.x is available? If so, let me know if we can help with that. Thanks!

rbri commented 3 years ago

Plans - yes, as always the time is a problem. It will be fantastic, if you can provide a PR.

jowerner commented 3 years ago

Yeah, same here. Okay, let's see what we can do. Will get back to you if we have something to review or just need your advice. :)

rbri commented 3 years ago

Great....

jowerner commented 3 years ago

Meanwhile, I spent some time migrating HtmlUnit to Apache HttpClient 5.x and now want to give you a short update.

HC 5.x comes in two flavors, synchronous (or "classic") and asynchronous. I started with the synchronous variant as it is very similar to 4.x and fits in very well. Only when this was done did I realize that the synchronous client does not support HTTP/2 which ultimately is the goal of this exercise. So I went ahead and reworked the code once more to use the asynchronous client.

The asynchronous and the synchronous variants share a bit of code, but are otherwise two different beasts with different sets of support classes. That's why customization is quite different as well. Many things have been moved to other places. For example, we can no longer set socket factories for plain and SSL sockets (or I simply did not find this). However, there are alternative ways to set custom key/trust stores, cipher suites, protocols, etc. That's why some functionality currently located in the socket factories is still missing. Thinking of SOCKS proxy support, for example. Furthermore, there are couple of test cases in package com.gargoylesoftware.htmlunit that are still failing at the moment.

As you can see there is still a bit of work to do. But before putting more effort into this, it is probably time to get feedback from you whether we are on the right track. If you like, I can provide a PR, as a sneak preview for you and a place where we can discuss details? Please let me know what you think. Thanks!

rbri commented 3 years ago

Wow, this sounds really great.

From my point of view we

If you can make a pr i have a look at your code on Friday, and if you like we can have an online meeting about this also (maybe also on Friday).

rbri commented 3 years ago

Looks like we have to switch to slf4j for logging also

jowerner commented 3 years ago

Sounds like a plan! Will make the PR available later this day or maybe tomorrow morning. Would be great if we can have our online meeting tomorrow as well because next week I'll be on vacation. Tomorrow afternoon anytime after 14:00 would suit me best. How about you? Will you or should I take care of the meeting?

rbri commented 3 years ago

please send a private mail to rbri at rbri dot de and i will send you an invitation

14:00 is fine for me - i have a private jitsi server instance .... and will send you the access details.

rbri commented 3 years ago

Have done a quick look at OkHtml and Jetty Client - both look also promising. Will have a look at the interface between HtmlUnit itself and the HttpClient - maybe we can have a clear interface for the transport layer.

mbucc commented 3 years ago

FYI, Java 11's java.net.http.HttpClient:

supports HTTP/1.1 and HTTP/2, both synchronous and asynchronous programming models, handles request and response bodies as reactive-streams, and follows the familiar builder pattern.

https://openjdk.java.net/groups/net/httpclient/intro.html

rbri commented 6 months ago

see https://github.com/HtmlUnit/htmlunit/issues/776