Manu404
commented
1 year ago Another comment, my code was .net Core 3.1, so I used WebClient and similar, which are deprecated in .Net 7 I'm switching to.
I switched to HttpClient who rely a lot more on the implementation of the network layer of the OS, but still some issues.
I was able to solve the issue using HttpClient and forcing ipv4 resolution, not requiring anymore to change the client configuration
That made this library work too, as the HttpClient used in the lib is injected, so I could inject my "forced ipv4" httpclient into the tool fetcher.
Sadly, the service I was targeting with your lib have blocked most Tor exit node, so I got it working, but for nothing... I tried with the Tor browser (never used it before) and couldn't reach the service neither, should have tried it before thinking it would be work...
Finally, I got back to some throttling to get around the "429 : Too many request" error, at the cost of execution time. And it's really focused on my datacenter. I'm working on a tool that analyze the references used in a wiki and check their status. Currently on my machine, from home, I'm able to analyze and verify around 700 references in 1:30min, without any 429 error. But from an OVH datacenter it takes 30min with a 10second throttling and degree of parallelism at one for the "web fetch" part to that service. I tried with a 5 second throttling, got banned at 90% of the analysis... I'm instantly banned without throttling. Hughh...
But thanks anyway for your lib, it could have saved my day !
Dear,
I'm working on a project where I was still using core 3.1 for portability and want to integrate your library in it. I've decided also to migrate to .net 7 as core is totally deprecated.
It works on my Windows and on server under Ubuntu 18.04 and in WSL Ubuntu 20.04.
I've been trying with the sample project I made to learn your lib on default config fresh install virtual machines of newer Ubuntu versions, and it wasn't working. (minimal server install of 20.04, 22.04 and 22.10 and desktop version of 20.04 [as I thought maybe it was also a dependency issue]).
In the Ubuntu virtual machines, I also had troubles to NuGet add your package to a new project for test, hanging forever in "updating the project", problem for NuGet restore, problem with a lot of things related to dotnet and often mentioning a lot of certificate issues.
So I thought it was an OpenSSL issue, maybe the one used by Ubuntu is stricter than windows and the old one of my server. I've even downloaded the CA from NuGet to add them manually and that kind of stuff, I've been searching for hours.
So I finally decided to clone your repo, copy your project and use it as a ref of my test app, putting few console.writeline to know exactly where it was faulty. The Ubuntu machine wasn't able to debug anything etc. (as the Ubuntu machines were barely able to compile anything with NuGet deps) and the stack trace indicated an exception in FetcherHelpers.GetStringAsync, which for me was another signal in the "SSL direction", the first webrequest hangs.
Also, wget the URLs had issues, so it was even more for me an SSL enforcing issue... but mentioning
--no-check-certificate... So it wasn't OpenSSL... then I stumble upon something:The problem was ipv6 !
Your lib will not work on Ubuntu 20 >and other systems who default DNS resolution to ipv6 due to the servers they rely on and maybe part of the .net framework itself.
The workaround: default to ipv4, everything worked like a charm, NuGet etc. included !
In Ubuntu 20.04 >, the fix to do it is to add
precedence ::ffff:0:0/96 100in the file/etc/gai.confI create an issue because it's maybe something worth mentioning, that the lib will not work on system defaulting DNS resolution to ipv6.
So might have some other people in the same position as me. With a default fresh install, ipv6 by default, an infrastructure not meant for it, and nothing working, blaming the lib when it is the host.
It's been a long day, but hope this pointer help others too, that's why I'm sharing this info, so maybe someone in the same situation will find this thread. Now, time to use your lib in my project ❤