TheBeastLT / torrentio-scraper

Apache License 2.0
364 stars 63 forks source link

Downtime #199

Closed 0o120 closed 7 months ago

0o120 commented 9 months ago

Torrentio experiencing lots of downtime the last couple of days.

https://www.reddit.com/r/Stremio/comments/195a8zc/down_again/

Vance-ng-vn commented 9 months ago

@TheBeastLT

you should check the cache, with my subtitles addon, when i set the cache to 4 hours, and the cache can reach up to 512mb and the app restarts without any notification at all, if you are having only 1gb ram, you should check it

TheBeastLT commented 9 months ago

I'm aware that there's around an hour of downtime for past 4 days. It started with my VPS being automatically migrated to new servers and seems it's no longer able to handle peak time loads. I'm trying to look into it and fixing it but it's not that easy to catch as it always happens at night, so it trying to change one thing or another and waiting for the next morning to see if it went down or not.

tundrablasen commented 9 months ago

I'm aware that there's around an hour of downtime for past 4 days. It started with my VPS being automatically migrated to new servers and seems it's no longer able to handle peak time loads. I'm trying to look into it and fixing it but it's not that easy to catch as it always happens at night, so it trying to change one thing or another and waiting for the next morning to see if it went down or not.

Thanks for looking into the issue. Also don’t stress and take your time. This here is your hobby and we all don’t pay shit for it.

If you want donations to keep up the costs of running the VPS, maybe update the GitHub readme.

thanks for your work

nreis96 commented 9 months ago

@TheBeastLT I just want to say thank you for providing us with torrentio and if ever needed please provide a donation or multiple donation methods I would happily throw some financial support!

Vance-ng-vn commented 9 months ago

I'm aware that there's around an hour of downtime for past 4 days. It started with my VPS being automatically migrated to new servers and seems it's no longer able to handle peak time loads. I'm trying to look into it and fixing it but it's not that easy to catch as it always happens at night, so it trying to change one thing or another and waiting for the next morning to see if it went down or not.

It's true that a bit laggy during peak hours, but is it just lag or has your app crashed?

Dave2337 commented 9 months ago

1 week into your service and was highly impressed. However the recent downtime and errors with api are concerning. The concenus on forums is that a lot of people would happily donate or pay you for this service either purely as a thanks or in the hope it would increase performance by allowing bigger investment. I am one of those people and if you had a donation or fee policy I would participate. Thanks for hardwork and hope you can fix current issues

nreis96 commented 9 months ago

1 week into your service and was highly impressed. However the recent downtime and errors with api are concerning. The concenus on forums is that a lot of people would happily donate or pay you for this service either purely as a thanks or in the hope it would increase performance by allowing bigger investment. I am one of those people and if you had a donation or fee policy I would participate. Thanks for hardwork and hope you can fix current issues

I would support more a donations policy

Benmyk commented 9 months ago

I'm aware that there's around an hour of downtime for past 4 days. It started with my VPS being automatically migrated to new servers and seems it's no longer able to handle peak time loads. I'm trying to look into it and fixing it but it's not that easy to catch as it always happens at night, so it trying to change one thing or another and waiting for the next morning to see if it went down or not.

Many thanks for what you do, really appreciate it, you make many apps here feel very useful with your work. We are solidly behind you as you try to correct the problem: All the best

ralorimer commented 9 months ago

As a newcomer to the service, I can only apologise to everyone for adding to the server strain! Thank you for everything you do, and I wish to add to the voices offering financial support.

arramire commented 9 months ago

Thank you, I’ve been using with stremio and it’s been working fine for me, Samsung tv via stremio and real debrid

jffluis commented 9 months ago

Thank you for you work! Keep going!

witchyface commented 9 months ago

Thank you for your efforts. I appreciate them!

fr40-ua commented 9 months ago

I want to add that I would be very happy to donate to keep things running! Thanks again for your hard work :)

redwing887 commented 9 months ago

If you want a more powerful VM, I have access to lot of infrastructure you can use to host it for free.

californias66 commented 9 months ago

Thank you for your hard work and we're truly grateful for all that you do for this community. Came here from Reddit specifically to say thanks and if anyway to help just let us know 🙏🏽🔥 thanks bro

TheBeastLT commented 9 months ago

If you want a more powerful VM, I have access to lot of infrastructure you can use to host it for free.

If you could reach out to me on discord (same nickname) maybe we could discuss this further.

Iamforilla commented 9 months ago

@TheBeastLT

you should check the cache, with my subtitles addon, when i set the cache to 4 hours, and the cache can reach up to 512mb and the app restarts without any notification at all, if you are having only 1gb ram, you should check it

Just want to say thank you for the hard work! Appreciate your addon. And your efforts! Keep up the good work.

hkay26 commented 9 months ago

Thank you so much for making this possible. It has improved our family time so much, I've been using it for a month. Just perfect with RD+

If there is a link for donations, I'd be glad to donate.

redwing887 commented 9 months ago

If you want a more powerful VM, I have access to lot of infrastructure you can use to host it for free.

If you could reach out to me on discord (same nickname) maybe we could discuss this further.

Sent friend request on discord

robertronq commented 9 months ago

Thank you for creating this. It is so awesome !!!

TheBeastLT commented 9 months ago

I've increased the allowed requests concurrency, since I'm now using selfhosted mongo and can support more connections and it seems to have helped with the problem. So I consider this solved atm.

KvHx commented 9 months ago

It's been intermittent in the past hour. Is there any issues going on in the backend?

dashnak commented 8 months ago

Aaaaaaaand seems the outages are back. Mine hasn't worked for like 4 hours.

TheBeastLT commented 8 months ago

I'm aware of the frustrations users have with the addon, but please be understanding that this is a free service hosted from my own will for the good of Stremio community, and with the recent explosion in the user base in Stremio and alternate streaming platforms in general (almost all similar apps have an integration with Torrentio where initially it was only meant for Stremio platform) it's much harder to maintain a community developed addon as a passion project. Even paid solutions are having problems when users started to trickle down from Torrentio, so you can image the load such a passion project has. Just to give some numbers this addon has over 10mil daily requests, almost 7 mil monthly unique users, and at peak times it has over 200 requests per second. I don't want to start taking donations as it was a passion project for me to improve on my initial addon (TPB+ which is still going, albeit only P2P), and taking donations would just mean additional risk for me as an individual and people feeling even more entitled to something that is provided for free from a passion project. Reasons outline very well in this comment from some reddit user which gets it. Many people also asked about self hosting the addon, but its just not meant for self hosting as outlined in https://github.com/TheBeastLT/torrentio-scraper/issues/113#issuecomment-1479151276 . There are many better projects for such use cases like jackett, *arr stack, or some self hosted DHT crawlers (which are morally debatable in the torrent community and I don't want to get into an argument for). In the end I'll try to improve the up-time of the addon and and make the addon available at peak times , but please be aware that this is something developed and hosted from a single person's free time (over 3500 hours over 5 years*) and the goodwill for the community of Stremio. The addon is not dead or abandoned at this time so please be more understandable and patient, or otherwise channel your anger/frustrations to finding alternative addons/services to fuel your demands.

*Anime is a b**ch to index and maintain.

Gabisonfire commented 8 months ago

Hey @TheBeastLT , I'd like to start by thanking you for this amazing work. I'm a hobbyist dev, I understand building an app like this takes a lot of time and effort. That being said, I have a few questions you might be able to clarify.

but its just not meant for self hosting

Why do you say that? I understand you would not want to share the DB for DMCA reasons, but scrapers are a totally different topic and you are just collecting public information. You are not sharing the torrents/magnets themselves. Other projects like Jacket and *arr are good, but they are more of an on-demand fashion. Why not just publish scrapers as well. We could help fix them when needed and help improved them or add more sources.

With the help of the community, and I've seen some very knowledgable in the comments, I can see this as project growing into some sort of a mesh network where people could opt to share their database and sync with others data to make this service HA in it's own way. People want to help.

In the end I'll try to improve the up-time of the addon and and make the addon available at peak times

Please reach out if you need help, I'm an infra engineer, I'd be glad to help you scale this if needed. (my handle is the same on Discord)

Thanks again for your great work. People frustrated can go ahead and build their own addons, you owe them nothing.

Goldenfreddy0703 commented 8 months ago

*Anime is a b**ch to index and maintain.

Hahaha thats so true 🤣, this is why i never made an anime addon on Stremio cause Stremio is a pain and limited with its coding and i didn't even know you had to host your addons on stremio, That's why i prefer coding with python on kodi sense kodi is way better and easier but i do say stremio is faster though.

Btw, as the last comment said, amazing work with torrentio-scraper, reminds me of a4kScrapers but hosted. Great and hope you find a solution for hosting in the future.

Thank you

0o120 commented 8 months ago

I've been working on my own addon similar to torrentio. The biggest issue I've come across is getting all a torrent sites torrents\hashes. Sure you could scrape the whole site, but some sites limit how far you can go back. Yea you could do search query when a request comes in then cache the results but it would be better to already have it in the database. Then you would just need to scrape newly added torrents.

@TheBeastLT

If you would be willing to release the database with the torrent info and hashes, it would be helpful for anyone else that wants to step up and try to offer some alternative solutions. If you're worried about DMCA, it could be done anonymously. Just set up a torrent and float the torrent hash around somewhere.

Thanks for all the work you put in. I'm sure you weren't expecting it to blow up so big and it definitely has to be stressful maintaining.

ultranova commented 8 months ago

I'm aware of the frustrations users have with the addon, but please be understanding that this is a free service hosted from my own will for the good of Stremio community, and with the recent explosion in the user base in Stremio and alternate streaming platforms in general (almost all similar apps have an integration with Torrentio where initially it was only meant for Stremio platform) it's much harder to maintain a community developed addon as a passion project. Even paid solutions are having problems when users started to trickle down from Torrentio, so you can image the load such a passion project has. Just to give some numbers this addon has over 10mil daily requests, almost 7 mil monthly unique users, and at peak times it has over 200 requests per second. I don't want to start taking donations as it was a passion project for me to improve on my initial addon (TPB+ which is still going, albeit only P2P), and taking donations would just mean additional risk for me as an individual and people feeling even more entitled to something that is provided for free from a passion project. Reasons outline very well in this comment from some reddit user which gets it. Many people also asked about self hosting the addon, but its just not meant for self hosting as outlined in #113 (comment) . There are many better projects for such use cases like jackett, arr stack, or some self hosted DHT crawlers (which are morally debatable in the torrent community and I don't want to get into an argument for). In the end I'll try to improve the up-time of the addon and and make the addon available at peak times , but please be aware that this is something developed and hosted from a single person's free time (over 3500 hours over 5 years) and the goodwill for the community of Stremio. The addon is not dead or abandoned at this time so please be more understandable and patient, or otherwise channel your anger/frustrations to finding alternative addons/services to fuel your demands.

*Anime is a b**ch to index and maintain.

thank you. please don't let go please

lovelydumpling commented 8 months ago

I don't want to start taking donations as it was a passion project for me to improve on my initial addon (TPB+ which is still going, albeit only P2P), and taking donations would just mean additional risk for me as an individual and people feeling even more entitled to something that is provided for free from a passion project

There's a simple solution. Accept donations for another project you're working on that isn't semi-illegal, and we can go donate there instead.

I think that's what they call "money laundering" and would itself be pretty risky, especially with the suggestion being made to do so on a public forum.

base08 commented 8 months ago

I'm aware of the frustrations users have with the addon, but please be understanding that this is a free service hosted from my own will for the good of Stremio community, and with the recent explosion in the user base in Stremio and alternate streaming platforms in general (almost all similar apps have an integration with Torrentio where initially it was only meant for Stremio platform) it's much harder to maintain a community developed addon as a passion project. Even paid solutions are having problems when users started to trickle down from Torrentio, so you can image the load such a passion project has. Just to give some numbers this addon has over 10mil daily requests, almost 7 mil monthly unique users, and at peak times it has over 200 requests per second. I don't want to start taking donations as it was a passion project for me to improve on my initial addon (TPB+ which is still going, albeit only P2P), and taking donations would just mean additional risk for me as an individual and people feeling even more entitled to something that is provided for free from a passion project. Reasons outline very well in this comment from some reddit user which gets it. Many people also asked about self hosting the addon, but its just not meant for self hosting as outlined in #113 (comment) . There are many better projects for such use cases like jackett, arr stack, or some self hosted DHT crawlers (which are morally debatable in the torrent community and I don't want to get into an argument for). In the end I'll try to improve the up-time of the addon and and make the addon available at peak times , but please be aware that this is something developed and hosted from a single person's free time (over 3500 hours over 5 years) and the goodwill for the community of Stremio. The addon is not dead or abandoned at this time so please be more understandable and patient, or otherwise channel your anger/frustrations to finding alternative addons/services to fuel your demands.

*Anime is a b**ch to index and maintain.

Thanks @TheBeastLT ! Don't let ignorant, negative comments get to you... I think the majority of users are truly grateful for the work you've done for the community... Take your time in fixing whatever needed and let us know if we can help with something!

metacatdud commented 8 months ago

Hello, I don't know how these addons work, but I got the gist of it.

Where do you think lies the problem? In the hosting? Or in the codding?

Why I am asking? I wonder if it might help to consider different language, smth more robust such as Go or Rust? Judging from what I saw in the codebase it might be an idea? If is hosting, would you consider seting a cryptowallet maybe or similar where ppl can donate? Just some thoughts.

0o120 commented 8 months ago

@lovelydumpling That is definitely not money laundering. What TheBeastLT is doing isn't illegal anyway. Otherwise, uTorrent or qBittorrent would also be committing crimes and doing money laundering. uTorrent has a paid version, for example.

Illegal or not, it still attracts potential lawsuits. Unless he has money to waste, nobody wants to burn money fighting any legal battles.

serniebanders commented 8 months ago

Do not ever accept donations. The second you do you will be immediately sued. They are definitely watching you.

lovelydumpling commented 8 months ago

@lovelydumpling That is definitely not money laundering. What TheBeastLT is doing isn't illegal anyway. Otherwise, uTorrent or qBittorrent would also be committing crimes and doing money laundering. uTorrent has a paid version, for example.

uTorrent and qBittorrent aren't designed specifically for the cataloging and linking of copyrighted material. Torrentio is. It's closer to The Pirate Bay than it is a torrent client. And we all know what happened to them.

Anyway, having donations for torrentio be under the guise of an unrelated product is by definition laundering the money. There wouldn't be any need to do so if the legality wasn't in question.

Ghawas1981 commented 3 months ago

https://github.com/TheBeastLT