search

lwthiker / curl-impersonate

curl-impersonate: A special build of curl that can impersonate Chrome & Firefox
MIT License
3.45k stars 229 forks source link

Discrepancy between the LD_PRELOAD hack and the curl script #143

Closed wrobelda closed 1 year ago

wrobelda commented 1 year ago

I noticed that the rss-bridge, which I enabled to use curl-impersonate via LD_PRELOAD hack, no longer impersonates properly.

I confirmed that with your upstream docker image with:

$ docker run -it --rm lwthiker/curl-impersonate:0.5-ff-slim-buster bash
> export LD_PRELOAD=/usr/local/lib/libcurl-impersonate-ff.so
> export CURL_IMPERSONATE=ff100
> apt install curl
> curl "https://allegro.pl/strefaokazji"
<html><head><title>allegro.pl</title><style>#cmsg{animation: A 1.5s;}@keyframes A{0%{opacity:0;}99%{opacity:0;}100%{opacity:1;}}</style></head><body style="margin:0"><p id="cmsg">Please enable JS and disable any ad blocker</p><script data-cfasync="false">var dd={'rt':'c','cid':'AHrlqAAAAAMAcPdqlWfimskAV2MuLw==','hsh':'77DC0FFBAA0B77570F6B414F8E5BDB','t':'bv','s':29560,'e':'c6c5a3b28b9ef65d144329e7023e6996c5a7d1e88fb3c2cc902d6474a15a7ead','host':'geo.captcha-delivery.com'}</script><script data-cfasync="false" src="https://ct.captcha-delivery.com/c.js"></script></body></html> 
> curl_ff100 "https://allegro.pl/strefaokazji"
> (returns valid HTML)

It worked just fine previously, so I am not sure if it regressed due to some underlying debian image updates (just checked, Alpine is also affected), or the impersonation is somehow flawed in LD_PRELOAD mode, and the tightening on allegro.pl end eventually caught it.

EDIT: also checked with curl-impersonate:0.5-chrome, same problem.

lwthiker commented 1 year ago

The LD_PRELOAD method does not work with stock curl. This is because curl overrides the settings set by libcurl-impersonate after they are set. This behavior is documented on the README:

Note that the LD_PRELOAD method will NOT WORK for curl itself because the curl tool overrides the TLS settings. Use the wrapper scripts instead.

There is not much to do about it, as we don't have control over the behavior of the upstream curl.

wrobelda commented 1 year ago

There is not much to do about it, as we don't have control over the behavior of the upstream curl.

Oh well, my bad! Somehow I was under impression that standalone curl worked previously. That having said, the rss-bridge uses libcurl, so it should work there just fine. And things got back to normal yesterday such that I am no longer getting the 403s. I don't know how to explain that, to be honest. I will close this for now and if this happens again, I'll investigate and reopen.

wrobelda commented 5 months ago

@lwthiker I am getting this issue again and it is a Duplicate of https://github.com/lwthiker/curl-impersonate/issues/147