search

elgatito / script.elementum.burst

Development of this addon has been stopped!
MIT License
115 stars 118 forks source link

CloudFlare protected providers #240

Open elgatito opened 3 years ago

elgatito commented 3 years ago

There was built-in CloudFlare bypass in Elementum, which helped to solve CloudFlare defense and get pages. But with the latest defense update we cannot bypass it, so providers, that use CloudFlare to avoid robots (like us) to request pages, are not accessible for the moment.

I don't know any easy way of solving this issue. Would be great if someone could help on this.

iz8mbw commented 3 years ago

Hi @elgatito Denis, hi all. I'm not a developer so I cannot help on this, but what about to use FlareSolverr https://github.com/FlareSolverr/FlareSolverr ? Thanks!

elgatito commented 3 years ago

@iz8mbw It requires installation of Nodejs on running host. If you think it can help you - you can install it manually and use in Burst, as a proxy server.

iz8mbw commented 3 years ago

@elgatito Denis I use "Jackett" with "FlareSolverr" on my Linux ARM64 board. I don't have ideas how to configure Burst to use my "Jackett" (listening on HTTP on my LAN) for searches.

iz8mbw commented 3 years ago

@elgatito I mean if it's possible to set to Burst to use "Jackett" please let me know how to do that.

iz8mbw commented 3 years ago

@elgatito I know there is this addon https://github.com/fugkco/script.elementum.jackett but it's not compatible with Kodi 19 (Python 3) so I cannot use and test it. If it's possible to set Burst to use "Jackett" please let me know how to do that. Many thanks!

elgatito commented 3 years ago

@iz8mbw I have no experience with Jackett. https://github.com/elgatito/script.elementum.burst/issues/246

iz8mbw commented 3 years ago

@elgatito I have added "Jackett.json" into my "userdata/addon_data/script.elementum.burst/providers/" directory but I cannot see this custom provider into Elementum Burst list of providers both Public and Private. Where I wrong?

elgatito commented 3 years ago

@iz8mbw You cannot see custom providers in settings. They are supposed to be enabled by default.

iz8mbw commented 3 years ago

OK, thanks!

elgatito commented 3 years ago

@iz8mbw @hoshsadiq I can add, if someone makes a Kodi plugin which runs/installs FlareSolverr - then Burst could even check for this addon and proxy requests through it.

Same with Jackett. When separate addon is complete - I can add it to Elementum repository, so that it gets available when you install Elementum.

hoshsadiq commented 3 years ago

Did you mean to tag me? Think you meant @fugkco.

Anyway, haven't used Elementum in a while, seems FlareSolverr is using hcaptcha-solver, however, it doesn't seem to be working:

At this time none of the captcha solvers work. You can check the status in the open issues. Any help is welcome.

So I'm not really sure what the point would be to add FlareSolverr support until there's a solid method of solving the captcha.

iz8mbw commented 3 years ago

There was built-in CloudFlare bypass in Elementum, which helped to solve CloudFlare defense and get pages. But with the latest defense update we cannot bypass it, so providers, that use CloudFlare to avoid robots (like us) to request pages, are not accessible for the moment.

I don't know any easy way of solving this issue. Would be great if someone could help on this.

Hi @elgatito Denis. May be this https://github.com/jairoxyz/script.module.cloudscraper can help?

elgatito commented 3 years ago

@iz8mbw Requires JavaScript engine.

ShlomiD83 commented 3 years ago

https://github.com/NoahCardoza/CloudProxy

@elgatito how about this project?

elgatito commented 3 years ago

@ShlomiD83 Have no idea. If it works for you - you can use it as proxy in Burst.

ShlomiD83 commented 3 years ago

I wish I knew how to use it, don't have a clue.

burekas7 commented 3 years ago

@elgatito

There is an addon that called "mando". I see there are some intersting functions that maybe could help, I don't know actually, I guess you will know if so better than me.

The two functions are: cloudflare_request (resources/modules/general.py) get_tokens_with_headers (resources/modules/cfscrape/run.py) => Maybe other files in the "cfscrape" folder.

You can download the addon from here: https://f2h.io/1co1sn73arhf

burekas7 commented 3 years ago

https://github.com/NoahCardoza/CloudProxy

@elgatito how about this project?

It requires to create and use a server based on javascript/nodejs, So this is not a proper solution.

elgatito commented 3 years ago

Any development help is welcome. I have no time for this.

fugkco commented 3 years ago

The Jackett plugin is pretty much beta. Has been working for me, but I know there's a few bugs that people have reported that I've not really been able to look into. Plus no Kodi Matrix support, which I know Kodi is gearing up for.

If Elementum repo gets the latest releases from the GitHub releases API, I say feel free to add it to the repository, but if not, maybe wait a little bit longer until I've had a chance to migrate to 19 and get a minute to migrate the addon too.

antonsoroko commented 11 months ago

probably makes sense to pin this issue.

elgatito commented 11 months ago

@antonsoroko It is already pinned for 2 years. That is the only pinned issue, actually, here.

elgatito commented 11 months ago

@burekas7 @SubEdict @antonsoroko

What providers are now not accessible because of Cloudflare? Apart of Idope.

antonsoroko commented 11 months ago

@elgatito ilcorsaroblu Torrentleech iptorrents - those do not work for me and for other people as I can see in other issues. In browser I literally see cloudflare page first and then redirect. Same page in burst debug output.

New yggtorrent URL also has Cloudflare but works for me for some reason (I made a PR for it). Also have it (at least in headers, but I do not see the page and redirect), but work: iTorrent YTS limetorrents

SubEdict commented 11 months ago

@elgatito I'm using it right now. All English Public providers works (excluding #341).

elgatito commented 11 months ago

@antonsoroko

New yggtorrent URL also has Cloudflare but works for me for some reason (I made a PR for it).

They have different levels of protection set on different websites. Some check basic stuff and we usually pass that check. Some have a javascript page that needs browser interaction. Some require checking checkbox with a browser. Some require solving a captcha.

On sites like ilcorsaroblu if we want to download torrent - Cloudflare allows to do it. If trying to login to website - they require to pass a checkbox test.

They also detect a platform, so that same Burst/same Elementum, same settings, but running on another platform can lead to a Cloudflare check failure.

antonsoroko commented 11 months ago

Just out of interest I also tried to add my cookies (after passing the check) to burst cookie file, but without success. Anyway, for me those "protected" trackers are not interesting, not sure how popular they are.

elgatito commented 11 months ago

@antonsoroko It will not work like that with a cookie, because it checks for all the headers and signatures when you use the cookie.

@scakemyer had a simple, but good solution - https://github.com/scakemyer/cloudhole .

Basically, you installed a plugin into a browser, and that plugin uploads your headers and Cloudflare cookies into a website. Later Burst was taking those cookies and was able to fetch those pages.

Why it worked:

But that extension was for Firefox. If someone wants to make a plugin for Chrome - I can make a server part, that would store those secrets in the same way cloudhole is storing them.

burekas7 commented 5 months ago

@burekas7 @SubEdict @antonsoroko

What providers are now not accessible because of Cloudflare? Apart of Idope.

1337x starts using it :/ I tried to find any mirror that doesn't use it and the search is correct, but there isn't.

Do we have any solution for these prolematic CloudFlate providers by the way?

ShlomiD83 commented 5 months ago

@burekas7 @SubEdict @antonsoroko What providers are now not accessible because of Cloudflare? Apart of Idope.

1337x starts using it :/ I tried to find any mirror that doesn't use it and the search is correct, but there isn't.

Do we have any solution for these prolematic CloudFlate providers by the way?

you can try to bypass it using Jackett, FlareSolverr & script.elementum.jackett. it seems to return results in Jackettt, need to test it inside Kodi.

elgatito commented 2 months ago

@ShlomiD83 @burekas7 @SubEdict #414 is now in testing.

elgatito commented 2 months ago

I managed to make ilcorsaroblu/torrentleech to work with Burst 0.0.87 and cookie sync enabled for those domains and itorrents.org.

One of them started to work when I changed Burst Useragent to become like the browser that used to create those cookies (otherwise Cloudflare was giving robot protection page).

By default Cookie Sync does not store user agent and TLS fingerprints.

antonsoroko commented 1 month ago

i did some testing:

torrentgalaxy (own protection) - previously it was giving me captcha page in burst debug log, with "cookie sync" - ok

cloudflare: extranet (ext.to and extranet.torrentbay.st) - fail, still gives "Just a moment" page idope - fail, still gives "Just a moment" page yggtorrent - fail, still gives "Just a moment" page. but here it looks like extension does not add cf_clearance cookie to gist. if i manually add my cf_clearance to .kodi/temp/burst/common_cookies.jar - site works.

(tried even with/without my own "user agent" - same result)

other providers do not give me "Just a moment" page, so can't say anything about them.

would be nice if other people will try this feature as well.