search

justchokingaround / lobster

mmm lobster
GNU General Public License v2.0
559 stars 37 forks source link

[bug]: fails to get video links #193

Closed 1bn3mar closed 5 months ago

1bn3mar commented 8 months ago

*IDKy but the script just refused to work for me I'm using macOS and mpv as my media player i didn't do anything special.

For me what happened was i just followed the installation process then tried to watch a show Steps to reproduce the behavior:

  1. $ lobster
  2. specify which show exactly season & episode etc.
  3. then it just outputs this: gsed: -e expression #1, char 0: no previous regular expression No subtitles found

P.S. this happens for anything i try to watch. lobster.log

pitsi commented 7 months ago

Leaving aside that lobster works, at least for today and on my end, I just noticed that you are the dev of the project. Are you advertising it or are you asking for any bug reports to be made there? Please be more accurate.

stillmanpeeer commented 7 months ago

sorry my bad, because the cost is limited, I have to limit my daily use of this API. Because there are so many users, I encountered this problem.

justchokingaround commented 7 months ago

do u see statistics about where most requests are coming from

pitsi commented 7 months ago

I am ok with that, as long as you mention it somewhere.

p.s. I am talking as an angry user of other apis which lowered (or even eliminated) their access limits without any notice, ever.

stillmanpeeer commented 7 months ago

do u see statistics about where most requests are coming from

mostly from vercel application.

stillmanpeeer commented 7 months ago

I am ok with that, as long as you mention it somewhere.

p.s. I am talking as an angry user of other apis which lowered (or even eliminated) their access limits without any notice, ever.

I am ok with that, as long as you mention it somewhere.

p.s. I am talking as an angry user of other apis which lowered (or even eliminated) their access limits without any notice, ever.

Yes, I will write down the information clearly

justchokingaround commented 7 months ago

if ure willing to share the code (it can stay closed source, i don't mind that, i actually prefer it that way) with me, i might be able to find a hosting solution from someone in the discord server

pitsi commented 7 months ago

Guess what happened...

After 1+ week of not touching it at all (because I have nothing to watch these days), I launched lobster today to test it. And it simply fails, because, as you can see in the log below, the api it now uses has blocked access due to too many requests! https://paste.debian.net/hidden/7fc3cc0a/

If the api's dev (@stillmanpeeer) is reading this, please explain how I managed to pass the 5000 requests/day limit that you have set (assuming that the limit is indeed at 5000 requests and not lower or zero), when lobster called it ONCE (or maybe 4 times as you can see in the log) in order to work. I am asking it as an angry user of youtube's api, which "charges" 1 api call per 1 video view, but a single search "costs" 100 api calls. And all that with its daily limit being at 10000 calls!

In other news, keys4fun (the domain) shut down last week and its dev's account here on github gets to a 404 today, so things are now pretty bleak for lobster :(

Dragon-Batch commented 7 months ago

anyone got a decryption method for /v2/embed-4/ player?

justchokingaround commented 7 months ago

techincally yes, but i cant open source it

Dragon-Batch commented 7 months ago

techincally yes, but i cant open source it

i have a handful of questions that have just been scratching that part of my brain for a while. if you dont mind answering even just a few i would be greatful :3 :3 :3 :3 :3 :3 :3

  1. is it available as an api or something? if so, will it cost money to use? i need it to rip all content from flixhq as i have found a method of obtaining free storage space.

  2. why do the providers keep changing the encryption? it just keeps getting cracked so i dont see the point in changing it if its gonna get destroyed anyway.

  3. do we know if there is a known api endpoint that these sort of providers use to communicate/generate links for the stream url hosts? because the urls for each stream link are different when decrypted again and again (i think) and not returning a duplicate one so they must be generating them somehow.

  4. do we know how they store the video segments? do they just have a gigachad storage method or are they using something like mass generated google drive accounts with the entire 15GB free storage used.

  5. do we know the owners of the providers? do they have contact information available to the public?

  6. are the providers just really shit at coding because (this community) is breaking their decryption so easily (it seems).

  7. some of the videos are encoded with AV1 and that is a pretty serious codec, it requires both good hardware and fine tuning. how do the providers compress the videos so quickly? it seems they add new stuff every day!!!

  8. why are some of the m3u8 playlists you get from some of the providers corrupt? "segments" returning 404 & 502 errors. is there a way to stop this from happening? i have included my own checking and just skipping the videos that dont match within 5 seconds of the playlists supposed length and the mp4 file.

  9. how do the providers stay up for so long? particularly rabbitstream as their website just looks like a normal video streaming service but given the knowledge of that it does, it just looks like a "front" for piracy. how have the servers not been raided by the fbi or like a swat team or something???

  10. is there a better alternative to flixhq in terms of crackability and larger selection of videos? you can see the list of movies and series on said website if you goto /sitemap.xml, as this will have a bunch of xml pages that follow something to this format: sitemap-list-[0-.1.2.3..].xml. different providers and stuff.

Ciarands commented 7 months ago

@Dragon-Batch A fair few questions lol but ill answer some for you

  1. Chomsky is talking about something I was working on, I haven't turned it into an API yet just because busy with life, there is no API that I am aware of currently though.
  2. Saves them money I'd guess, take us @ movie-web for instance, we probably cost them a bit of money just with our users, and we're not serving ads for them, there's certainly reason to want to quickly break stuff, however they're not super creative, best src I've seen so far is flixhq, but even they're kinda trivial to scrape due to some major oversights on their behalf.
  3. Not that I'm aware of?
  4. It depends on the provider, many host on DMCA ignored VPS's, its really not that expensive and they're making a fair bit of money.
  5. No, we don't scrape sites that we know the owners of usually lol.
  6. Its not that they're shit at coding, its that its very difficult to come up with a creative sln to stop us due to the nature of the web, I can think of a few methods that would stop 99% of scrapers, but all it takes is 1 person to break it.
Dragon-Batch commented 7 months ago

@Ciarands if you want, i guess i consider my self experienced at making apis with flask in python so i think i could help you make one if you want :33

triorr commented 6 months ago

The key is generated inside a web assembly module disguised as a PNG image in "loading.png". Reverse engineering skills is needed to fix this script. For more info check this article https://www.pudn.com/Download/item/id/1711319547629731.html

luvcie commented 6 months ago

https://github.com/stillmanpeeer/rabbitstream

Could you be more specific on what is this? @stillmanpeeer

Dragon-Batch commented 6 months ago

its a advertisement to some shity api that costs 160$/month

On Fri, 26 Apr 2024, 12:45 pm luvcie, @.***> wrote:

https://github.com/stillmanpeeer/rabbitstream Could you be more specific on what is this?

— Reply to this email directly, view it on GitHub https://github.com/justchokingaround/lobster/issues/193#issuecomment-2078538762, or unsubscribe https://github.com/notifications/unsubscribe-auth/ARIM46NTSNB4CZX5PXVHFRTY7G5VFAVCNFSM6AAAAABEARSO6SVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDANZYGUZTQNZWGI . You are receiving this because you were mentioned.Message ID: @.***>

pitsi commented 6 months ago

Although I disagree with the price asked for accessing the api (because its a lot for a single user who will use it) and with its lack of documentation (especially about the requests per day limit), I have to admit that the api works. When lobster was first patched so as to use it, it did return streams and subtitles for them, so yea, it does work as intended.

justchokingaround commented 6 months ago

image

so i'm currently implementing a vidsrc scraper to switch over to it from rabbitstream. if anyone has ever implemented rc4 decryption using openssl pls hmu, bc porting my bash code to posix shell, without using arrays is gonna be a bit painful

pitsi commented 6 months ago

All this is what must be done for getting the keys? If so, I may know a github repo that generates them. How can I contact you though? I am not on matrix and neither on most known and unknown social media.

---edit Also, can someone please change the title accordingly and pin the issue so that every new issue reporter can see it?

justchokingaround commented 6 months ago

@chomkerman:matrix.org

pitsi commented 6 months ago

Yes, I know that, it's on your profile's info. I am not on matrix. I also do not make accounts here and there.

justchokingaround commented 6 months ago

oh sorry i thought i said u are on matrix, my bad

justchokingaround commented 6 months ago

i have the keys actually, but its hard to implement the decryption using those keys

pitsi commented 6 months ago

So the problem is that gibberish on the url, like %3A and %2F, we see on the right terminal?

justchokingaround commented 6 months ago

no, the problem is that the decryption code i wrote is bash specific and not posix compliant, since i use an array for S

pitsi commented 6 months ago

Sorry, I can't help you with that :(

triorr commented 6 months ago

@justchokingaround Try something like this

echo -n "yB5_gMJfMGtHJCTb6P4aLZpbJu9hxhicEj==" | sed 's/_/\//g' | sed 's/-/+/g' | base64 -d | openssl rc4 -K $(echo -n "WXrUARXb1aDLaZjI" | xxd -p | tr -d '\n') -nopad
justchokingaround commented 6 months ago

image did you generate this using AI ? did this work when u ran it on ur system?

i've tried getting rc4 to work using openssl but couldn't, even with my experience with it. i need someone who has actually done it before and got it working.

also echo -n isn't posix compliant, you should use printf instead, it doesnt output a newline by default

triorr commented 6 months ago

@justchokingaround Sorry. I'm not very familiar with POSIX compliance. I have tested this on bash, sh, zsh and even fish they all work for me with "echo -n" and "printf". Try adding the commands one by one to check whats wrong. First your base64 has a problem with that input try the full base64 url from vidsrc. Because I cut it down for readability and it worked on my machine. About openssl I think you have the legacy algorithms disabled for security reasons check with this:

$ openssl list -providers
Providers:
  default
    name: OpenSSL Default Provider
    version: 3.2.1
    status: active
  legacy
    name: OpenSSL Legacy Provider
    version: 3.2.1
    status: active

to activate the legacy algorithms you need to add/uncomment these lines 

legacy = legacy_sect

[legacy_sect]
activate = 1

[default_sect]
activate = 1

in your /etc/ssl/openssl.cnf

justchokingaround commented 6 months ago

okay thanks a lot, i'll check it later

justchokingaround commented 6 months ago

that worked just fine, thank you a lot @triorr for helping me figure out that the problem was the openssl package i was using (i just changed from openssl to openssl_legacy on nix)

printf "%s" "$url" | sed "s/_/\//g;s/-/+/g" | base64 -d | openssl rc4 -K "$(printf "%s" "$key" | xxd -p)" -nopad
Kat299 commented 5 months ago

Guess what happened...

After 1+ week of not touching it at all (because I have nothing to watch these days), I launched lobster today to test it. And it simply fails, because, as you can see in the log below, the api it now uses has blocked access due to too many requests! https://paste.debian.net/hidden/7fc3cc0a/

If the api's dev (@stillmanpeeer) is reading this, please explain how I managed to pass the 5000 requests/day limit that you have set (assuming that the limit is indeed at 5000 requests and not lower or zero), when lobster called it ONCE (or maybe 4 times as you can see in the log) in order to work. I am asking it as an angry user of youtube's api, which "charges" 1 api call per 1 video view, but a single search "costs" 100 api calls. And all that with its daily limit being at 10000 calls!

In other news, keys4fun (the domain) shut down last week and its dev's account here on github gets to a 404 today, so things are now pretty bleak for lobster :(

Yep, that's happening for me too.

justchokingaround commented 5 months ago

dd205d91b703df3d6d41a6d617c96a35a45d42b7