search

ulixee / secret-agent

The web scraper that's nearly impossible to block - now called @ulixee/hero
https://secretagent.dev
MIT License
675 stars 45 forks source link

feature-request: provide interface for MITM Proxy #435

Open GlenDC opened 2 years ago

GlenDC commented 2 years ago

Currently the MITM (Go, socket) proxy can already been disabled using the SA_DISABLE_MITM env variable. The other side of the choice is to have it be enabled (the default). Two things about that:

  1. SA_DISABLE_MITM is still considered experimental, would be cool if in Hero that could be considered a standard opt-in option. While indeed it can come with a warning label that this might make fingerprinting you a lot easier if done without the care it requires, it should in the end still be about the user. Might also be cool if instead of having to do that via an environment variable it would be done via the launch configuration or something similar.
  2. It would be cool if the MITM API could be abstracted away via an interface:
    • API would be fairly simple as it just requires connections to be established and utilised;
    • Perhaps also some kind of options for creation;

Request (2) would allow the ability for someone to plugin their own MITM proxy while still being able to configure it from within SA to set the correct TCP Options/flags and TLS ClientHello configuration.

GlenDC commented 2 years ago

Also while I can see why you would prefer Unix sockets over for example an HTTP interface, that doesn't really have to be part of the abstracted API, and can be left as an opaque config of the used MITM implementation. This way HTTP MITM proxies could also be used if desired.

GlenDC commented 2 years ago

Not sure if should be considered a separate issue, but what I really would like to abstract away is the entire MITM code. Meaning not only the GO, socket server but also the HTTP layer above.

GlenDC commented 2 years ago

Related issue: https://github.com/ulixee/secret-agent/issues/436

blakebyrnes commented 2 years ago

SA_DISABLE_MITM is still considered experimental, would be cool if in Hero that could be considered a standard opt-in option. While indeed it can come with a warning label that this might make fingerprinting you a lot easier if done without the care it requires, it should in the end still be about the user. Might also be cool if instead of having to do that via an environment variable it would be done via the launch configuration or something similar.

This is such a hard one to think through how to do this and keep things working reliably. When I said "experimental", it's there because every once in a while, I found myself turning it off to see if I could get through a site without the mitm (ie, are they picking up on something here we're doing different than chrome?). I don't know if/when we'll go down the path of experimental meaning "ready for production".

It would be cool if the MITM API could be abstracted away via an interface:

We do want to have an ability to run plain http requests at some point. That said, making plain http requests keeps getting more and more complicated. Chrome sends different headers, cookies and order based on what sites you've "first party visited", Http2 headers and frames are vastly different that http1 request, etc, etc. And Quic changes things to run over a whole different transport.. ha. I can barely grasp it, so trying to document in a clean way is pretty daunting. I suspect most people want it to "just happen".

On our unofficial "ideas" roadmap, we've talked about the best way to truly do the MITM is to do a build of Chrome at each version where we only build the network stack. Then throw an API in front of it. Doesn't help with this task, but that's the dream :)

GlenDC commented 2 years ago

Awesome. Thanks for that. I guess to emulate the Network Stack of a specific Chrome version easiest would just be to have SA/Hero use the exact desired chrome version. But for faking other browsers that would indeed require a proxy.

That being said, I do not see a reason to have it so tightly attached to SA/Hero. While one can be shipped with it for the regular users, there are other users (like myself) who do want to have their own proxy.

I do follow you, things get complicated really quickly. With the advance of Quick it's only getting crazier... People are still catching up on HTTP/2.