search

hurlenko / mloader

Download manga from mangaplus.shueisha.co.jp
GNU General Public License v3.0
148 stars 31 forks source link

Add support for App-Exclusive Mangas/Chapters #36

Open G-Utopia opened 2 years ago

G-Utopia commented 2 years ago

Like the title suggest there are free chapters and mangas which are App exclusive. Maybe you cant implement them.

darodi commented 2 years ago

Do you have an example?

G-Utopia commented 2 years ago

The entire Chainsaw man manga is available via App.

hurlenko commented 2 years ago

From what I see, they are now using a different api endpoint for manga details (title_detailV2 vs title_detail used by mloader) along with some secret, we can try and utilise it.

meshstyles commented 2 years ago

It would be interesting if the other endpoint also lists the delisted chapters that are even hidden from the website and the app (like most of the re-editions e.g., Naruto). Some of these in between chapters seem to remain on the server and are downloadable atm. I might consider doing an 'API dump' if these are not visible.

hurlenko commented 2 years ago

I've made an update to use the mobile app's API. Seems to be working fine for me (and faster). If anyone wants to try it out, please install from the develop branch:

pip install git+https://github.com/hurlenko/mloader.git@develop

I'd like to hear some feedback before releasing a new version.

meshstyles commented 2 years ago

with naruto [title: 100018 ] you don't get the recent chapters in the app but then currently for me it only downloads up to including the 7th chapter while there are 9 in the app available and at least up to chapter 10 [chapter: 1000406 ] is on the server.

Edit: also yeah it's a lot faster for some reason

Edit Edit: the formating on spy x family [100056] for chapter 58 part 1 [1011977] and chapter 58 part 2 [1012693] does seem to not work as intended. i tried something to fix this with #38 but feel free to disregard it entirely i'm not really someone who does stuff with python

hurlenko commented 2 years ago

@meshstyles Thanks a lot for the feedback, will take a look at it in the next couple days, don't have much time now

darodi commented 2 years ago

there seems to be a problem with this version. The last chapter is not always correct:

with those arguments in develop branch

-l -f raw -t 100023

I'm getting

06.08.2022 20:46:39 |   INFO   |   loader.py    145  | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:46:39 |   INFO   |   loader.py    146  |     Author: Nobuhiro Watsuki
06.08.2022 20:46:39 |   INFO   |   loader.py    157  |     1/1) Chapter #005: Act 5: The Fight Merchant

in master branch:

-l -r --chapter-subdir -t 100023

I'm getting

06.08.2022 20:49:00 |   INFO   |  __main__.py   206  | Started export
06.08.2022 20:49:06 |   INFO   |   loader.py    138  | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:49:06 |   INFO   |   loader.py    139  |     Author: Nobuhiro Watsuki
06.08.2022 20:49:07 |   INFO   |   loader.py    150  |     1/1) Chapter #190: Act 190: Battle on Three Fronts: Round 1, Part 2
#190
06.08.2022 20:49:13 |   INFO   |  __main__.py   224  | SUCCESS
G-Utopia commented 2 years ago

In my case I could download all 101 chainsaw man chapters at once with no struggle.

meshstyles commented 2 years ago

i kinda explined this poorly in my comment but it seems like the mobile app api does not have the latest chapters for some manga. but the website api does have them. This explains how there are diffrent last chapters using the dev and master branch. for example chainsaw man on the other hand has all chapters in the app but only the earliest and latest chapers on the website. So for optimal results youd need to consider both apus in some cases but i think it is not visible through the api if there are diffrences in available chapters between both apis. The only option i see here is to either request data from both api or just use a user input flag to check if the website api or the app api should be used

janowakps commented 2 years ago

so there is a problem with develop branch

root@nextcloudpi:~# mloader -t 100034

            _                 _
  _ __ ___ | | ___   __ _  __| | ___ _ __
 | '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
 | | | | | | | (_) | (_| | (_| |  __/ |
 |_| |_| |_|_|\___/ \__,_|\__,_|\___|_|

04.09.2022 13:29:32 |   INFO   |  __main__.py   202  | Started export
04.09.2022 13:29:33 |   INFO   |   loader.py    145  | 1/1) Manga:
04.09.2022 13:29:33 |   INFO   |   loader.py    146  |     Author:
04.09.2022 13:29:33 |   INFO   |  __main__.py   219  | SUCCESS

After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time

IsaBNekomata commented 1 year ago

How can I fix this? image

I'm new to this command-line thing and want to read chainsaw man without having to deal with the "only avaible to read one time" bs of manga-plus

meshstyles commented 1 year ago

so there is a problem with develop branch

root@nextcloudpi:~# mloader -t 100034

            _                 _
  _ __ ___ | | ___   __ _  __| | ___ _ __
 | '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
 | | | | | | | (_) | (_| | (_| |  __/ |
 |_| |_| |_|_|\___/ \__,_|\__,_|\___|_|

04.09.2022 13:29:32 |   INFO   |  __main__.py   202  | Started export
04.09.2022 13:29:33 |   INFO   |   loader.py    145  | 1/1) Manga:
04.09.2022 13:29:33 |   INFO   |   loader.py    146  |     Author:
04.09.2022 13:29:33 |   INFO   |  __main__.py   219  | SUCCESS

After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time

I'm actually shocked but all you need to do is change the app version. seems like the secret is valid for some time but yeah it looks like some continous updates buming the version number are required for mobile.

bysilvaart commented 1 year ago

How can I download that Free chapters from the App? I need to change the secret value?

G-Utopia commented 1 year ago

It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.

meshstyles commented 1 year ago

How can I download that Free chapters from the App? I need to change the secret value?

you need the development branch the feature is yet to be made public

It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.

idk if that would m+ would like that tbh

BenjaminOchocki commented 1 year ago

So for everybody coming to this thread and thinking about how to get the secret key:

@hurlenko your develop branch is working like a charm. The download order is sometimes a bit weird but works completely fine

KingSlowDown commented 1 year ago

Same problem as @IsaBNekomata and @meshstyles, changing the secret code didn't change anything.

hydreg commented 11 months ago

@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.

@hurlenko As you probably know Shueisha locked some series behind a paywall of subscription. These series only have 3 first chapters available and after downloading them, the script throws an error. Assassination Clasroom (100001) is an example. Would be great if the error could throw some human readable message. I'm also curious - if I were to pay for subscription, would it be recognized by the script and let me download the completed series? Does secret key carry that information?

hurlenko commented 11 months ago

@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.

hydreg commented 11 months ago

@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways.

meshstyles commented 11 months ago

@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.

I think i did a rather poor job at explining what i ment with the app version number. to make it easier for anyone not in the know mloader with app api support was probaly bulilt looking at the ios app. so when the ios app has an update you need to change the app version in loader.py AUTH_PARAMS to a current version of ios app. Tbh this seems like it'd be a chore to keep up with so i think it might be best to have a cli param that could so people don't need to install from source if it takes a while for an update through pip.

@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.

I think they've introduced that at the start of the month so it's still kinda new. But the highest tier allows you access to basically everything they had.

@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways. I'll be honest some manga have poor quality so... yeah idk if this is the approach I'd recomend other then to have fun or learn how automate or datamine.

saxx0n commented 10 months ago

One issue I discovered why attempting to use the devel branch was that it did not detect the .5 chapters. Example would be for Hokkaido Gals Are Super Adorable! (100116) which it found chapters 100 and 101, but not 100.5

rokibhasansagar commented 9 months ago

I can from Desktop Browser that they are using https://jumpg-webapi.tokyo-cdn.com/api/title_detailV3?title_id=xxxx for the api. Rest addresses are still the same.

rokibhasansagar commented 9 months ago

Another note: Even if the manga page from website shows more than 6 chapters free to read on desktop without subscription, mloader can only get the oldest 3 + latest 3 chapters 😢 Any way to fix that?

rokibhasansagar commented 9 months ago

Hmm, I updated the AUTH_PARAMS values from my Phone's App and it's currently downloading All chapters. 😄

10SecondsToWaste commented 7 months ago

Any ideas if BlueStacks can be used? I rooted it by modifying bluestacks.conf, then found the config.xml and got the secret, but couldn't get it to work.

Here's the config I put in loader.py:

AUTH_PARAMS = {
    "app_ver": "1.9.19",
    "os": "android",
    "os_ver": "7.1.2",
    "secret": "[redacted]",
}

Here's the traceback:

28.01.2024 20:43:22 |   INFO   |  __main__.py   206  | Started export
28.01.2024 20:43:23 |   INFO   |   loader.py    140  | 1/2) Manga: 
28.01.2024 20:43:23 |   INFO   |   loader.py    141  |     Author: 
28.01.2024 20:43:23 |  ERROR   |  __main__.py   226  | Failed to download manga
Traceback (most recent call last):
  File "[redacted]\venv\Lib\site-packages\mloader\__main__.py", line 218, in main
    loader.download(
  File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 191, in download
    self._download(manga_list)
  File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 146, in _download
    chapter = viewer.pages[-1].last_page.current_chapter
              ~~~~~~~~~~~~^^^^
  File "[redacted]\venv\Lib\site-packages\google\protobuf\internal\containers.py", line 93, in __getitem__
    return self._values[key]
           ~~~~~~~~~~~~^^^^^
IndexError: list index out of range
28.01.2024 20:43:23 |   INFO   |  __main__.py   227  | SUCCESS

I'm currently on BlueStacks 5.8 if that makes a difference.

rokibhasansagar commented 7 months ago

@10SecondsToWaste, when the error says IndexError: list index out of range, it's mostly because the manga you want to download needs MangaPlus MAX subscription (Premium). Some other cause is also possible. Try different manga and see what happens.

10SecondsToWaste commented 7 months ago

@rokibhasansagar

when the error says IndexError: list index out of range, it's mostly because the manga you want to download needs MangaPlus MAX subscription (Premium)

I don't think so. The chapters I want to download are available for one-time reading and I have not read any of them since I downloaded the app.

Try different manga and see what happens.

Same error. Not even the latest chapters can be downloaded. Here's what the commands I used look like:

$ python -m mloader --out ./foo --format raw --chapter 1019618

I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.

hydreg commented 7 months ago

I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.

Does it download correctly if you use ids you can access via browser, for example 1019569? I'm almost certain the id you've provided has a typo and should be 1019608.

10SecondsToWaste commented 7 months ago

@hydreg

Does it download correctly if you use ids you can access via browser, for example 1019569?

Nope. In fact, not even the original AUTH_PARAMS work. Does it work for you?

I'm almost certain the id you've provided has a typo and should be 1019608.

Maybe. It is from the post for chapter 11 of Two On Ice. Now that you mention it, the IDs for chapter 10 and chapter 12 are 1019174 and 1019590, which are both smaller than 1019618.

rokibhasansagar commented 7 months ago

@10SecondsToWaste, "Two On Ice" Manga ID is "100276" In your loader.py (develop branch, of course), change api/title_detailV2 to api/title_detailV3 and change the secret again. With just those changes, I am able to download the whole manga (17 chapters).

10SecondsToWaste commented 7 months ago

@rokibhasansagar Now there is no error, but chapters is an empty set and mloader just exits without downloading anything.

Just to be crystal clear, I installed the @develop branch anew, then changed its AUTH_PARAMS (with the correct secret of course) and the API endpoint like this:

@@ -23,10 +23,10 @@ log = logging.getLogger()
 MangaList = Dict[int, Set[int]]  # Title ID: Set[Chapter ID]

 AUTH_PARAMS = {
-    "app_ver": "1.8.3",
-    "os": "ios",
-    "os_ver": "15.5",
-    "secret": "f40080bcb01a9a963912f46688d411a3",
+    "app_ver": "1.9.19",
+    "os": "android",
+    "os_ver": "7.1.2",
+    "secret": "[redacted]",
 }

@@ -67,7 +67,7 @@ class MangaLoader:
     @lru_cache(None)
     def _get_title_details(self, title_id: Union[str, int]) -> TitleDetailView:
         resp = self.session.get(
-            f"{self._api_url}/api/title_detailV2",
+            f"{self._api_url}/api/title_detailV3",
             params={
                 **AUTH_PARAMS,
                 "title_id": title_id,

You can see the change for yourself here. The same behaviour can be replicated using a GitHub codespace on this exact branch.

I also wiped all app data to get a new secret. Nothing changed.

rokibhasansagar commented 7 months ago

@10SecondsToWaste, change the app_ver to 97, That way, everything works.

P.S: Just use actual secret key in production 😅

10SecondsToWaste commented 7 months ago

@rokibhasansagar Thanks a bunch! It's finally downloading.

Now, are there anything I can do to help this feature become a part of the @master branch?

hauveli commented 5 months ago

Attempting the same. Have my own secret key (paid for deluxe).

Modifying code from here to have

        self._api_url = "https://jumpg-webapi.tokyo-cdn.com"

Tried above, failed. (Only free chapters found)

Tried modifying most recent master, failed. (Only free chapters found)

Attempted permutations of these AUTH_PARAMS:

    "app_ver": "97",
    "os": "android",
    "os_ver": "13",
    "os_ver": "7.1.2",
    "User-Agent": "JumpPlus/1 CFNetwork/1333.0.4 Darwin/21.5.0",
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0",
    "secret": "mysecretkey",

I have not installed mloader via pip, and am running it as follows:

python -m mloader.__main__ https://mangaplus.shueisha.co.jp/titles/<id>/ -o .

Does the method from the other posts still work? I will try to get the HTTP headers again and update my post if they work.

rokibhasansagar commented 5 months ago

@hauveli, change the app_ver to 124 at least. MangaPlus's Internal (API) App Version changed a month ago. Now, it can even be 131 if 124 does not work any more. And using the Mobile API Endpoint instead of Web API Endpoint is also preferred. Last but not least, add this change in your code too: https://github.com/hurlenko/mloader/issues/36#issuecomment-1916796950

ThoughtzThruKeyz commented 2 months ago

Hi, so after reading through this thread several times, I'm still confused as to where to find this "key"? Like in the apk what is it called? What file/folder is it located in?? Additionally, can this same thing be found via iOS? Because I know the default setting for mloader is based on the iOS version of the app, so how can I scrape my "key" from there without having to change the entire api to only take in the android version? Thanks.

10SecondsToWaste commented 1 month ago

@rokibhasansagar How do you know the correct value for app_ver? I wanted to download a new manga again today, and got the exact same problem. Both Play Store and App Store versions are specified as "1.9.*". I guess this is an internal detail?

rokibhasansagar commented 1 month ago

@10SecondsToWaste @ThoughtzThruKeyz As I have no idea about iOS, I can share my method for Android only.

  1. To get the app_ver, you need to install "Lucky Patcher" via official "LuckyPatcherInstaller". Details on the first pin post in "r/luckypatcher" subreddit. After installation, Tap the MangaPlus app and from the options list, select "App Info". You'll see Build after the "Version" string.

  2. To get the secret key, you need a network capture app, I recommend "Reqable" or it's previous name "HttpCanary" (from APKPure), or other apps. Root might be needed and works fine until Android v7.1.2 (not tested on latest Android).

meshstyles commented 1 month ago

@rokibhasansagar How do you know the correct value for app_ver? I wanted to download a new manga again today, and got the exact same problem. Both Play Store and App Store versions are specified as "1.9.*". I guess this is an internal detail?

It used to be the app version for iPhones from the Apple app store. The app versions for iOS apps are below the description on the app store website in the change log.

Fyi for On android every app has the app version number on the Play Store page. It's derived from the APK or it at least used to be that way. Its located in the info about the app where all the other info is ( e. g. compatible android versions).

So for the versions number you don't even need the app just a web browser.

rokibhasansagar commented 1 month ago

@meshstyles The app_ver is Not the simple version of the app anymore. If you'd read all the comments and the APIv3 AUTH_PARAMS, you'd know. It needs a build number of the APK (for Android). The App's public version string (also know as Release version) is indeed 1.9.19 (and someone might be getting 1.9.20 too), but the Internal app_ver string in APIv3 Calls don't use that anymore. Currently, for my Android 14, the build id is 165. That's to be put as app_ver inside AUTH_PARAMS.

I also explained it in https://github.com/hurlenko/mloader/issues/36#issuecomment-1916796950 and https://github.com/hurlenko/mloader/issues/36#issuecomment-1918412990 several months ago.

gauchoux commented 2 weeks ago

@rokibhasansagar thanks a lot for the help to get the version and the secret key. I am still struggling on the part for the secret key tho, I have installed the reqable app, but I am not sure what to look for from there. I launch it and analyse the requests that are made from the MangaPlus app but can't find any secret key, where should I look ? Thanks again for your help ! :)

rokibhasansagar commented 1 week ago

@gauchoux As I recently had a new smartphone with Android 14, I can't get a working Network Capturing app to fully work in my device for Android's updated Security changes. I could with Reqable app, but it needs a rooted device to do so now, and I can't root my phone while being under Warranty period. Reqable's previous brand name, "HttpCanary" (old version) did work fully on my old Android 7.1.2 rooted device, but that's dead now :( and the VPN generated by Reqable can't connect to MangaPlus's server on my new device. So, I too can't get a new secret key anymore 😢 Fun fact: Reqable fully works on Facebook Lite app though 😆

I have to find other Network Capturing apps which can work on unrooted latest Android system 💭