search

donderjoekel / Mangarr

An *arr inspired approach to downloading manga using individual sources
MIT License
34 stars 3 forks source link

Sonarr/Radarr/Lidar Code base #7

Open ericflix opened 2 months ago

ericflix commented 2 months ago

Any chance of moving over the Sonarr/Radarr/Lidar Code base? Pretty much all the *arr apps were forked off Sonarr and modified. There is even a fork in the works for non manga comics - Kapowarr.

This would remove the added work of the UI other than customizations and let you work on the backend.

Thundernerd commented 2 months ago

I would really like to yeah. I initially looked at it and saw the massive code base without any proper guide on how to use/modify it without spending a lot of time myself trying to figure things out. It might be something that is actually worth it even though it'll cost a lot of time, but I'll have to look into that.

ericflix commented 2 months ago

Seeing the Kapowarr is a comic arr app, it might be quicker to fork off them.

I do like your approach on pulling from all the manga/manhwa/manhua sites, makes it much easier.

Thundernerd commented 2 months ago

I'll have a look at that one!

I specifically wanted to support all the websites that serve content out there because a lot of people have their own favorite fan translations that they want to read from and not everything gets uploaded to newsgroups or torrents. It would be nice if it can support both of those down the line though, but it isn't the main focus for now!

ETA:

Kapowarr actually seems like a custom implementation in python, so they aren't forked from Sonarr or any of the others. I might just have to dive into the original one

Thundernerd commented 2 months ago

So I finally had a look at Sonarr and was able to make some modifications to get some things going. It isn't fully working yet but made some promising progress!

Added a bunch of random manga/manhua/manhwa to the overview image

And was able to properly check one, though the season information is a bit borked image

Figured I'd might as well fork Jackett to see if I can use that as a indexer gateway and was able to make a small bit of progress there as well

image

image

ericflix commented 2 months ago

That looks great! How are you handling scraping chapters from different sites? Is that what you are modifying Jackett for? Will it be able to scrape sites and even parse/ingest cbr/cbz files?

Since you are using the Sonarr code base I am hoping that you will support postgres too. I'm not a dev, but if you need testing done on a docker/kubernetes environment let me know.

Thundernerd commented 2 months ago

If Docker supports postgres then this will automatically as well, I'm not really making any breaking changes, or at least not planning to!

The way it currently works is:

  1. You search for a manga, it uses mangaupdates.com to map those search results
  2. One you've selected/added your manga it'll scan the entry on mangaupdates.com for the releases, these are then converted to "episodes"
  3. Whenever Sonarr does its refresh, or whenever you manually look for a specific item it'll contact Jackett
  4. Whenever there's a match for an episode/chapter Sonarr will ask Jackett for a torrent file, I am basically abusing torrents as regular data containers
  5. These torrents contain links to all images belonging to a chapter, which Sonarr will internally download through regular http requests
  6. (it doesn't do this yet) It'll then create a ComicInfo.xml file and together with that bundle it all up into a cbz
  7. That cbz file can then be automatically processed and imported by Sonarr putting it in the right place!
ericflix commented 2 months ago

Sounds great. I'm looking forward to trying it out when you get it to the point you want to release it.

Postgres is supported on all the *arr apps now, Directions are on the Servarr site. I'll let you know how it works then there is a container available for it.

alexandru0-dev commented 2 months ago

@Thundernerd That sounds great, do u have a repo/branch where i can find the wip?

Thundernerd commented 2 months ago

These are not yet available, I'm currently looking into making builds for both the Sonarr and Jackett forks and then turn them into docker images! Once I've confirmed that that is working as intended I'll be sure to notify both of you (and anyone else that wants to)

alexandru0-dev commented 2 months ago

I don't really need to be in a functional stage, just curious and if I have time I can look into it

Btw for builds I can help you, I'm also going to create nix builds for myself.

I've quickly seen your jackett fork commits but doesn't your changes completely remove the old jackett behavior?

So I would be a fork for only indexing manga sources no?

Thundernerd commented 2 months ago

These are the repo+branch: https://github.com/tnrd-org/Sonarr/tree/mangarr https://github.com/tnrd-org/Jackett/tree/mangarr

I expect them both to be completely separate solutions, as in the functionality won't be PR'd into the original repositories as what I'm doing doesn't match with the intended use of the original software.

In fact, the Jackett one abuses torrents as a whole and simply uses them as data containers that will be read by my Sonarr fork and the "Fake Torrent" download client that I've built, which in turn is quite a bit different from how Sonarr usually works

alexandru0-dev commented 2 months ago

Thx, I was looking at your profile instead of the org

Also can I ask why did u choose jackett over prowlarr?

Thundernerd commented 2 months ago

I am not familiar with Prowlarr, so I’ll have to take a look at it! I used Jackett because I know of it and figured it might be a nice base to start from. I’ll definitely have a look at Prowlarr though

alexandru0-dev commented 2 months ago

Sure, I've looked a bit and integrations of other *rr services should be easier and this also could lead to possibly upstream it.

Tho idk how the fact that mangas are mainly DDL and you should manage different sources could impact upstream.

If I have some answers I will ping you.

Thx for the great work again

Thundernerd commented 2 months ago

@alexandru0-dev I'm planning on switching over to Prowlarr. Most of the leg work for the indexers has already been done in Jackett so I "just" have to port them over!

alexandru0-dev commented 2 months ago

@Thundernerd first of all thx again

Is there a way I can contact you directly Discord or Matrix for example?

I've asked in the prowlarr's discord server and there are some problems to tackle about multiple sources I think or I didn't understand how your implementation would work

So I wanted to chat properly

Thundernerd commented 2 months ago

@alexandru0-dev You can hit me up on Discord. I'm also in the Servarr discord under the same name as here so feel free to send me a message request there!

markschrik commented 1 month ago

Hi!

Nice work :) Once this is available with a sonarr base, I am really interested in running it. If you want, please keep me posted too!

Thundernerd commented 1 month ago

I have made quite some progress and have successfully been running it for a couple of days on my own server now!

I can post a docker compose along with some setup steps if y’all are interested?

It is far from finished though, especially one major issue is that there’s no one source of truth when it comes to available chapters. I’m currently writing an indexer that periodically scrapes all sources to and then links them all together so I can use this to figure out what chapters have been released.

Either way, still quite some work to do, but it is possible to run already!

g-nogueira commented 1 month ago

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

Thundernerd commented 1 month ago

I've just "finished" writing the source for MangaDex, so it would be quite useful to have the "rr" version so that I can see If I can implement the same.

This is the repo for my Prowlarr fork which is where the sources will go: https://github.com/donderjoekel/Prowlarr

ericflix commented 1 month ago

Have you reached out to the Servarr Wiki? They added the Whisparr project/fork to their list of arr apps. If you can get Mangarr included to the arr ecosystem, that would open up a large group in the community of devs that can help get Mangarr to a mature state faster.

It is also a possibility to get your manga scrappers added into the prowlarr project if there is no major changes needed.

alexandru0-dev commented 1 month ago

@ericflix I did, but unfortunately mangarr works only with the jackett/prowlarr forks and are not merge-able with upstream (excluding indexers, the rest is shared)

We can find people that are willing to collaborate on the project there, surely.

The issue about incompatibility is mainly having different sources for DDLs (how different sources need a different way to be downloaded, aggregated etc) As for now it's not compatible with upstream unfortunately and I hardly think it is ever going to be in the short run especially with the project not having that much of traction rn. As of now existing indexers are replaced with manga sources.

@Thundernerd correct me if I'm wrong.

Thundernerd commented 1 month ago

Yup that’s correct.

My forks of Sonarr and Prowlarr are not huilt to support torrents and/or newsgroups. Unfortunately from my experience manga isn’t uploaded in agreed-upon standards so it would be very hard to use that.

Instead what my forks do is utilize the fair share of manga hosting websites and turns them into sources that can be used to download the images directly and pack it up into an archive for you to use in your favorite reader.

The core of both my Sonarr and Prowlarr fork have strayed so far from the original intended functionality that these will never be merged upstream.

jordanfelle commented 1 month ago

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

Thundernerd commented 1 month ago

Will the fork of Sonarr work with stock Prowlarr still for when there are good torrent/newsgroup sources?

Maybe? I'm honestly not too sue actually! I'm trying to (ab)use as much as I can from stock Sonarr and just build on top of it, so it could very well work. That said, there's a high chance that it will not work, or already isn't working. If the upload community can band together and come up with a workable standard then I'm 100% down to make it work though.

g-nogueira commented 1 month ago

Didn't think you would try making it work with torrent. That's awesome! Nonetheless, it seems you're focusing on the scrapping part, which I do believe is the better way of getting manga/comics. Thank you for the hard work!

I'm really looking forward to give a look at the code base but, first, I'm trying to make my fork of the sonarr_youtubedl work well so that I can try merging it (but I don't have high expectations, as there are already some stale PR's there).