Proxy per request

[ivanovevgeny]

It would be nice to add the ability to specify a proxy server.

[tmenier]

You can do this with a custom factory:

using Flurl.Http.Configuration;

public class ProxyHttpClientFactory : DefaultHttpClientFactory
{
    private string _address;

    public ProxyHttpClientFactory(string address) {
        _address = address;
    }

    public override HttpMessageHandler CreateMessageHandler() {
        return new HttpClientHandler {
            Proxy = new WebProxy(_address),
            UseProxy = true
        };
    }
}

To register it globally on startup:

FlurlHttp.Configure(settings => {
    settings.HttpClientFactory = new ProxyHttpClientFactory("http://myproxyserver");
});

[ivanovevgeny]

Thanks!

[matteocontrini]

Hi, what if I wanted to set a proxy for a specific HTTP request?

[CreatedAutomated]

The custom proxy factory is nice, however, it would be awesome if you added proxy support that was much easier to use rather than creating a custom factory. I know httpclient does not allow proxy changing once the client has been created, however! There is actually a way of doing this. I've seen it done before and lost the code for it. Maybe add a "WithProxy(proxy)" feature that automatically sets the proxy with each request. It would be pretty awesome

[tmenier]

@matteocontrini @CreatedAutomated Can you describe the use case for why you want to proxy some calls and not others?

[CreatedAutomated]

@matteocontrini @CreatedAutomated Can you describe the use case for why you want to proxy some calls and not others?

If a proxy is dead and does not respond for a client instance, and we want to change it to a working proxy, it means the client needs disposed and a new client needs created just to use another proxy. It's more efficient just to keep one client instance, and be able to dynamically change the proxy.

[matteocontrini]

My case is similar. I have a number of proxies, and I want each request to use one proxy, randomly chosen from that pool of proxies. With the current way HttpClient and Flurl work, this seems impossible, and it's very limiting. In many other languages setting the proxy for a request is a matter of passing a parameter to a method!

[tmenier]

@CreatedAutomated

I know httpclient does not allow proxy changing once the client has been created

HttpClientHandler.Proxy is a read/write property, so I don't see why not, unless there's some logic that throws if you attempt to do it. Can you provide a reference on that?

[CreatedAutomated]

HttpClientHandler

The only issue is with this, once you set it, you cannot change it. You need to create a new httpclient instance. But there is a way to change it on the fly, I forgot how. If I find out, I will share the solution

[tmenier]

@matteocontrini @CreatedAutomated Yeah, I think this is a limitation of the HttpClient stack. I can't think of a way Flurl could work around the fact that you can't set a different proxy per request on a shard instance in a thread-safe way. All I can think of maintaining a pool of clients (one per proxy) and switching between those as needed. You should actually be able to implement random or round-robin proxies with a custom FlurlClientFactory so it's completely abstracted away from requests. The details are a little beyond the scope of this issue but if you need help with an implementation feel free to ask on Stack Overflow, I typically answer everything tagged flurl.

[matteocontrini]

For anyone looking for some code beyond the discussion, here's a sample implementation of a FlurlClientFactory that creates and caches a client per proxy, for requests that are supposed to be proxied, and a client per host for all the other requests.

I haven't tested it thoroughly, but it should work. The disadvantage to this approach is that you need to know in advance a strategy for deciding whether the request should go through a proxy, based on its URL. This means that you can't actually choose whether to enable the proxy or not on a per-request basis.

EDIT: this code has a side effect that could create unexpected results in some cases. See #374 for details and an alternative solution

public class FlurlClientFactory : IFlurlClientFactory
{
    private readonly ConcurrentDictionary<string, IFlurlClient> clients =
        new ConcurrentDictionary<string, IFlurlClient>();

    public IFlurlClient Get(Url url)
    {
        if (url == null)
        {
            throw new ArgumentNullException(nameof(url));
        }

        if (ShouldGoThroughProxy(url))
        {
            string randomProxyUrl = ChooseRandomProxy();

            return ProxiedClientFromCache(randomProxyUrl);
        }
        else
        {
            return PerHostClientFromCache(url);
        }
    }

    private bool ShouldGoThroughProxy(Url url)
    {
        throw new NotImplementedException();
    }

    private string ChooseRandomProxy()
    {
        throw new NotImplementedException();
    }

    private IFlurlClient PerHostClientFromCache(Url url)
    {
        return clients.AddOrUpdate(
            key: url.ToUri().Host,
            addValueFactory: u => {
                return new FlurlClient();
            },
            updateValueFactory: (u, client) => {
                return client.IsDisposed ? new FlurlClient() : client;
            }
        );
    }

    private IFlurlClient ProxiedClientFromCache(string proxyUrl)
    {
        return clients.AddOrUpdate(
            key: proxyUrl,
            addValueFactory: u => {
                return CreateProxiedClient(proxyUrl);
            },
            updateValueFactory: (u, client) => {
                return client.IsDisposed ? CreateProxiedClient(proxyUrl) : client;
            }
        );
    }

    private IFlurlClient CreateProxiedClient(string proxyUrl)
    {
        HttpMessageHandler handler = new HttpClientHandler()
        {
            Proxy = new WebProxy(proxyUrl),
            UseProxy = true
        };

        HttpClient client = new HttpClient(handler);

        return new FlurlClient(client);
    }

    /// <summary>
    /// Disposes all cached IFlurlClient instances and clears the cache.
    /// </summary>
    public void Dispose()
    {
        foreach (var kv in clients)
        {
            if (!kv.Value.IsDisposed)
                kv.Value.Dispose();
        }

        clients.Clear();
    }

}

[tmenier]

I'm going to give some fresh consideration to WithProxy. Based on some planned changes for 3.0, it'll be possible to define a strategy for choosing an HttpClient instance based on other request criteria besides just the URL, which should make the implementation a lot easier.

@CreatedAutomated

I know httpclient does not allow proxy changing once the client has been created, however! There is actually a way of doing this. I've seen it done before and lost the code for it. If I find out, I will share the solution ... once you set it, you cannot change it. You need to create a new httpclient instance. But there is a way to change it on the fly, I forgot how.

I don't want to hold you to something you said over a year ago, and you're sort of saying "you can't but you can" so I might be misunderstanding anyway, but if you're suggesting you can somehow change the proxy on an existing instance of HttpClient, I believe (based on some extensive googling) that this is false. Earlier in this thread I was uncertain based on the fact that it's a read/write property, but I confirmed it will throw an exception if you try to change it after the first call is made.

So I think we'll be stuck with an HttpClient instance per proxy under the hood, but Flurl can abstract that. If you're worried about the famous socket exhaustion problem caused by all those instances, here's a crucial question: Isn't a socket by definition specific to one address? If so, wouldn't 1000 proxies require 1000 new sockets to be opened anyway, regardless of whether you could somehow do it with a single HttpClient instance? I'm not an expert at the low-level sockets stuff, so I'm honestly not 100% certain. Hoping someone here might be able to confirm that.

[CreatedAutomated]

This is not false. I had code to do this in the past. I sourced it from another developer on skype. I will need to try dig the code up somehow. I will post it for you when I find it. Its 100% possible

On Friday, January 17, 2020, Todd Menier notifications@github.com wrote:

Reopened #228 https://github.com/tmenier/Flurl/issues/228.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmenier/Flurl/issues/228?email_source=notifications&email_token=AKIZM2KNH5BGJXYICDKETATQ6ILQTA5CNFSM4D5S4KWKYY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOWBS7H4Y#event-2959471603, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKIZM2LEV52STOQ2XRUM7UTQ6ILQTANCNFSM4D5S4KWA .

[developervariety]

Only true way of changing the proxy while using the same HttpClient is something like Titanium-Web-Proxy. Although if you want a proxy per specific request, it's practically impossible with HttpClient honestly (kinda sucks).

[CreatedAutomated]

I forgot about this. When I'm at my desk I will search my desktop. The code I had was pretty low level. Httpclients is just a wrapper for the HttpWebRequest class. There is a way you can actually modify the httpwebrequest proxy property. Once I find the code I'm gonna share it here.

On Monday, March 23, 2020, Justin Lopez notifications@github.com wrote:

Only true way of changing the proxy while using the same HttpClient is something like Titanium-Web-Proxy https://github.com/justcoding121/Titanium-Web-Proxy. Although if you want a proxy per specific request, it's practically impossible with HttpClient honestly (kinda sucks).

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmenier/Flurl/issues/228#issuecomment-602377350, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKIZM2IU4NTH5E3DQA3SWCLRI3P33ANCNFSM4D5S4KWA .

[shravan2x]

@tmenier

Can you describe the use case for why you want to proxy some calls and not others?

Wanted to add another use-case here - I have an application that runs a number of separate tasks/operations/services in the same program i.e. a "monolith". Here, different proxies are used by separate tasks for their own use-cases eg. some for IP load balancing, some to circumvent geo-restrictions, some for corporate authentication, etc.

However, this doesn't require setting the proxy per request, setting it per client works fine (and I believe that's already possible via new FlurlClient { Settings = new ClientFlurlHttpSettings { HttpClientFactory = ... } }. Creating a custom HttpClientFactory is inconvenient though, as explained in @CreatedAutomated's comment.

[nogginbox]

My usecase is I have created an ApiWrapper and would like to offer users of the wrapper a way to disable the proxy, but I don't want my library to change the global behaviour of Flurl if the consumer of my wrapper is using it for something else.

[tmenier]

@NogginBox I'll keep that use case in mind. Due to contraints of the HttpClient stack, there would still need to be 2 clients at play here, but if you're using global configuration (i.e. FlurlHttp.Configure) it'll be easy to keep all your other settings the same for both.

[lenniebriscoe]

@tmenier do you have a timeline when this feature will be implemented please?

[ngoquoctoandev]

@tmenier I am doing scratching on a website, and that site blocks my IP if I call many incoming requests in a minute, in this situation I am forced to use a Proxy. If I use a Proxy server that supports cyclic Proxy, each request I call will have a different IP. That's great, but there are very few providers like this with good quality. Some other vendors download the list of available Proxies, but to me this is not possible to declare in Flurl settings. I want you to think about developing an extension method like WithProxy (host: port) ... that can make each call on a different IP.

[tmenier]

Apologies that I didn't follow through on this in 3.0 as I mentioned earlier in this thread. A ton went into that release and this didn't make the cut. It also seems clear from some of these comments that a primary use case is circumventing rate limits, so it's also a bit of an ethical dilemma as to whether I want to help people do that. This is still in the backlog but I do not have a timeframe on when or even if it will be done. For those who need this capability, I would strongly suggest looking at @matteocontrini 's solution above and in more detail (with updates) here.

[CreatedAutomated]

Sorry, I lost access to my account.

Sadly, I do not have a solution for this right now. But as I said before, I did have.

I was in contact with another developer a few years back who sent me his code for doing this. I cannot remember exactly how it was done, since I copied and pasted it. Wayyyyy back on my old computer.

But I remember that it used reflection to modify the proxy property. It can be done 100%. But, I never use reflection or i'd give it a go.

Maybe someone else can take reflection and mess with it?

[ngoquoctoandev]

I would love to use Flurl as a preferred choice. So I really need the code that can change the Proxy after every HTTP Request call, can you research the code that supports this function?

On Wed, Oct 20, 2021 at 11:15 PM CreatedAutomated @.***> wrote:

Sorry, I lost access to my account.

Sadly, I do not have a solution for this right now. But as I said before, I did have.

I was in contact with another developer a few years back who sent me his code for doing this. I cannot remember exactly how it was done, since I copied and pasted it. Wayyyyy back on my old computer.

But I remember that it used reflection to modify the proxy property. It can be done 100%. But, I never use reflection or i'd give it a go.

Maybe someone else can take reflection and mess with it?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tmenier/Flurl/issues/228#issuecomment-947823268, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASW2VZHAFYSWU6Q5UFNJFGDUH3TKLANCNFSM4D5S4KWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[CreatedAutomated]

No, you will have to research it yourself like everyone else has to. It does not impact me enough to look into it.

I've simply created a class that creates flurlclients with proxy (if needed), and then it will pull that client every time I use that specific proxy. It is efficient enough for what I need.

Previously I would create a new client every time which meant duplicate use of clients with proxies. So now, it is more efficient as it only creates one client per proxy, and not several, and it will be returned to me by passing my proxy object over.

If I ever want to get around creating multiple clients, I will probably end up writing my own kinda "Flurl-like" library, but using TcpClient instead.

On Wed, Oct 20, 2021 at 5:22 PM Banned @.***> wrote:

I would love to use Flurl as a preferred choice. So I really need the code that can change the Proxy after every HTTP Request call, can you research the code that supports this function?

On Wed, Oct 20, 2021 at 11:15 PM CreatedAutomated @.***> wrote:

Sorry, I lost access to my account.

Sadly, I do not have a solution for this right now. But as I said before, I did have.

I was in contact with another developer a few years back who sent me his code for doing this. I cannot remember exactly how it was done, since I copied and pasted it. Wayyyyy back on my old computer.

But I remember that it used reflection to modify the proxy property. It can be done 100%. But, I never use reflection or i'd give it a go.

Maybe someone else can take reflection and mess with it?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/tmenier/Flurl/issues/228#issuecomment-947823268, or unsubscribe < https://github.com/notifications/unsubscribe-auth/ASW2VZHAFYSWU6Q5UFNJFGDUH3TKLANCNFSM4D5S4KWA

. Triage notifications on the go with GitHub Mobile for iOS < https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675

or Android < https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub .

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/tmenier/Flurl/issues/228#issuecomment-947828862, or unsubscribe https://github.com/notifications/unsubscribe-auth/AKIZM2IAOK6DJMM7LP6ZUMLUH3UD3ANCNFSM4D5S4KWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

[SnowyWhite]

I would love to use Flurl as a preferred choice. So I really need the code that can change the Proxy after every HTTP Request call, can you research the code that supports this function? … On Wed, Oct 20, 2021 at 11:15 PM CreatedAutomated @.***> wrote: Sorry, I lost access to my account. Sadly, I do not have a solution for this right now. But as I said before, I did have. I was in contact with another developer a few years back who sent me his code for doing this. I cannot remember exactly how it was done, since I copied and pasted it. Wayyyyy back on my old computer. But I remember that it used reflection to modify the proxy property. It can be done 100%. But, I never use reflection or i'd give it a go. Maybe someone else can take reflection and mess with it? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#228 (comment)>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ASW2VZHAFYSWU6Q5UFNJFGDUH3TKLANCNFSM4D5S4KWA . Triage notifications on the go with GitHub Mobile for iOS https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675 or Android https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub.

@CreatedAutomated I think I can help you in this case since I'm pretty sure I stumbled across that piece of code just recently because I am searching for a solution as well.

Please have a look at this repository here: https://github.com/mboukhlouf/WebProxyService

I've tested this WebProxy approach with this method here:

[HttpGet]
public async Task Get()
{
    _proxyService = new WebProxyService();
    _handler = new HttpClientHandler { UseProxy = true, Proxy = _proxyService };
    _httpClient = new HttpClient( _handler );

    var proxies = new[]
    {
        new WebProxy("207.229.93.67", 1026),
        new WebProxy("207.229.93.67", 1025),
        new WebProxy("207.229.93.67", 1027),
        new WebProxy("207.229.93.67", 1029)
    };

    foreach (var proxy in proxies)
    {
        var ip = await GetIpWithProxyAsync( proxy );
        _logger.LogInformation( $"Proxy: {proxy.Address.Host}:{proxy.Address.Port}, IP: {ip}" );
    }
}

private async Task<string> GetIpWithProxyAsync(IWebProxy proxy)
{
    _proxyService.Proxy = proxy;
    return await _httpClient.GetStringAsync( "https://api.ipify.org/" );
}

And got this result which looks correct to me:

Proxy: 207.229.93.67:1026, IP: 192.3.139.188 Proxy: 207.229.93.67:1025, IP: 107.173.248.223 Proxy: 207.229.93.67:1027, IP: 107.173.248.74 Proxy: 207.229.93.67:1029, IP: 192.3.139.178