Open G-Utopia opened 2 years ago
Do you have an example?
The entire Chainsaw man manga is available via App.
From what I see, they are now using a different api endpoint for manga details (title_detailV2
vs title_detail
used by mloader) along with some secret, we can try and utilise it.
It would be interesting if the other endpoint also lists the delisted chapters that are even hidden from the website and the app (like most of the re-editions e.g., Naruto). Some of these in between chapters seem to remain on the server and are downloadable atm. I might consider doing an 'API dump' if these are not visible.
I've made an update to use the mobile app's API. Seems to be working fine for me (and faster). If anyone wants to try it out, please install from the develop
branch:
pip install git+https://github.com/hurlenko/mloader.git@develop
I'd like to hear some feedback before releasing a new version.
with naruto [title: 100018 ] you don't get the recent chapters in the app but then currently for me it only downloads up to including the 7th chapter while there are 9 in the app available and at least up to chapter 10 [chapter: 1000406 ] is on the server.
Edit: also yeah it's a lot faster for some reason
Edit Edit: the formating on spy x family [100056] for chapter 58 part 1 [1011977] and chapter 58 part 2 [1012693] does seem to not work as intended. i tried something to fix this with #38 but feel free to disregard it entirely i'm not really someone who does stuff with python
@meshstyles Thanks a lot for the feedback, will take a look at it in the next couple days, don't have much time now
there seems to be a problem with this version. The last chapter is not always correct:
with those arguments in develop branch
-l -f raw -t 100023
I'm getting
06.08.2022 20:46:39 | INFO | loader.py 145 | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:46:39 | INFO | loader.py 146 | Author: Nobuhiro Watsuki
06.08.2022 20:46:39 | INFO | loader.py 157 | 1/1) Chapter #005: Act 5: The Fight Merchant
in master branch:
-l -r --chapter-subdir -t 100023
I'm getting
06.08.2022 20:49:00 | INFO | __main__.py 206 | Started export
06.08.2022 20:49:06 | INFO | loader.py 138 | 1/1) Manga: Rurouni Kenshin
06.08.2022 20:49:06 | INFO | loader.py 139 | Author: Nobuhiro Watsuki
06.08.2022 20:49:07 | INFO | loader.py 150 | 1/1) Chapter #190: Act 190: Battle on Three Fronts: Round 1, Part 2
#190
06.08.2022 20:49:13 | INFO | __main__.py 224 | SUCCESS
In my case I could download all 101 chainsaw man chapters at once with no struggle.
i kinda explined this poorly in my comment but it seems like the mobile app api does not have the latest chapters for some manga. but the website api does have them. This explains how there are diffrent last chapters using the dev and master branch. for example chainsaw man on the other hand has all chapters in the app but only the earliest and latest chapers on the website. So for optimal results youd need to consider both apus in some cases but i think it is not visible through the api if there are diffrences in available chapters between both apis. The only option i see here is to either request data from both api or just use a user input flag to check if the website api or the app api should be used
so there is a problem with develop branch
root@nextcloudpi:~# mloader -t 100034
_ _
_ __ ___ | | ___ __ _ __| | ___ _ __
| '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
| | | | | | | (_) | (_| | (_| | __/ |
|_| |_| |_|_|\___/ \__,_|\__,_|\___|_|
04.09.2022 13:29:32 | INFO | __main__.py 202 | Started export
04.09.2022 13:29:33 | INFO | loader.py 145 | 1/1) Manga:
04.09.2022 13:29:33 | INFO | loader.py 146 | Author:
04.09.2022 13:29:33 | INFO | __main__.py 219 | SUCCESS
After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time
How can I fix this?
I'm new to this command-line thing and want to read chainsaw man without having to deal with the "only avaible to read one time" bs of manga-plus
so there is a problem with develop branch
root@nextcloudpi:~# mloader -t 100034 _ _ _ __ ___ | | ___ __ _ __| | ___ _ __ | '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__| | | | | | | | (_) | (_| | (_| | __/ | |_| |_| |_|_|\___/ \__,_|\__,_|\___|_| 04.09.2022 13:29:32 | INFO | __main__.py 202 | Started export 04.09.2022 13:29:33 | INFO | loader.py 145 | 1/1) Manga: 04.09.2022 13:29:33 | INFO | loader.py 146 | Author: 04.09.2022 13:29:33 | INFO | __main__.py 219 | SUCCESS
After updating AUTH_PARAMS obtained from my android phone, everything works, so unfortunately the "secret" value has a certain expiration time
I'm actually shocked but all you need to do is change the app version. seems like the secret is valid for some time but yeah it looks like some continous updates buming the version number are required for mobile.
How can I download that Free chapters from the App? I need to change the secret value?
It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.
How can I download that Free chapters from the App? I need to change the secret value?
you need the development branch the feature is yet to be made public
It would be nice if @janowakps or @hurlenko write how to get the secret key in the app and set it in mloader.
idk if that would m+ would like that tbh
So for everybody coming to this thread and thinking about how to get the secret key:
@hurlenko your develop branch is working like a charm. The download order is sometimes a bit weird but works completely fine
Same problem as @IsaBNekomata and @meshstyles, changing the secret code didn't change anything.
@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.
@hurlenko As you probably know Shueisha locked some series behind a paywall of subscription. These series only have 3 first chapters available and after downloading them, the script throws an error. Assassination Clasroom (100001) is an example. Would be great if the error could throw some human readable message. I'm also curious - if I were to pay for subscription, would it be recognized by the script and let me download the completed series? Does secret key carry that information?
@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.
@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways.
@IsaBNekomata and @meshstyles you just need to update version number (check the app store to get it) in loader.py. You need the version from development branch to even have the option though.
I think i did a rather poor job at explining what i ment with the app version number. to make it easier for anyone not in the know mloader with app api support was probaly bulilt looking at the ios app. so when the ios app has an update you need to change the app version in loader.py AUTH_PARAMS to a current version of ios app. Tbh this seems like it'd be a chore to keep up with so i think it might be best to have a cli param that could so people don't need to install from source if it takes a while for an update through pip.
@hydreg didn't know about paywall. The script has no authorization so it knows nothing about you paying for subscription. I've opened two issues (#50 and #51) but I can't give the estimates on when I might get to them.
I think they've introduced that at the start of the month so it's still kinda new. But the highest tier allows you access to basically everything they had.
@hurlenko Take your time, if it ever works out we'll be able to get complete archives anyways. I'll be honest some manga have poor quality so... yeah idk if this is the approach I'd recomend other then to have fun or learn how automate or datamine.
One issue I discovered why attempting to use the devel branch was that it did not detect the .5 chapters. Example would be for Hokkaido Gals Are Super Adorable! (100116) which it found chapters 100 and 101, but not 100.5
I can from Desktop Browser that they are using https://jumpg-webapi.tokyo-cdn.com/api/title_detailV3?title_id=xxxx
for the api. Rest addresses are still the same.
Another note: Even if the manga page from website shows more than 6 chapters free to read on desktop without subscription, mloader
can only get the oldest 3 + latest 3 chapters š¢
Any way to fix that?
Hmm, I updated the AUTH_PARAMS
values from my Phone's App and it's currently downloading All chapters. š
Any ideas if BlueStacks can be used? I rooted it by modifying bluestacks.conf
, then found the config.xml
and got the secret, but couldn't get it to work.
Here's the config I put in loader.py
:
AUTH_PARAMS = {
"app_ver": "1.9.19",
"os": "android",
"os_ver": "7.1.2",
"secret": "[redacted]",
}
Here's the traceback:
28.01.2024 20:43:22 | INFO | __main__.py 206 | Started export
28.01.2024 20:43:23 | INFO | loader.py 140 | 1/2) Manga:
28.01.2024 20:43:23 | INFO | loader.py 141 | Author:
28.01.2024 20:43:23 | ERROR | __main__.py 226 | Failed to download manga
Traceback (most recent call last):
File "[redacted]\venv\Lib\site-packages\mloader\__main__.py", line 218, in main
loader.download(
File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 191, in download
self._download(manga_list)
File "[redacted]\venv\Lib\site-packages\mloader\loader.py", line 146, in _download
chapter = viewer.pages[-1].last_page.current_chapter
~~~~~~~~~~~~^^^^
File "[redacted]\venv\Lib\site-packages\google\protobuf\internal\containers.py", line 93, in __getitem__
return self._values[key]
~~~~~~~~~~~~^^^^^
IndexError: list index out of range
28.01.2024 20:43:23 | INFO | __main__.py 227 | SUCCESS
I'm currently on BlueStacks 5.8 if that makes a difference.
@10SecondsToWaste, when the error says IndexError: list index out of range
, it's mostly because the manga you want to download needs MangaPlus MAX
subscription (Premium). Some other cause is also possible.
Try different manga and see what happens.
@rokibhasansagar
when the error says
IndexError: list index out of range
, it's mostly because the manga you want to download needsMangaPlus MAX
subscription (Premium)
I don't think so. The chapters I want to download are available for one-time reading and I have not read any of them since I downloaded the app.
Try different manga and see what happens.
Same error. Not even the latest chapters can be downloaded. Here's what the commands I used look like:
$ python -m mloader --out ./foo --format raw --chapter 1019618
I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.
I got the chapter IDs from old Reddit posts in each manga's subreddit (for Two On Ice it's r/TwoOnIce, etc.) so they are definitely correct.
Does it download correctly if you use ids you can access via browser, for example 1019569? I'm almost certain the id you've provided has a typo and should be 1019608.
@hydreg
Does it download correctly if you use ids you can access via browser, for example 1019569?
Nope. In fact, not even the original AUTH_PARAMS
work. Does it work for you?
I'm almost certain the id you've provided has a typo and should be 1019608.
Maybe. It is from the post for chapter 11 of Two On Ice. Now that you mention it, the IDs for chapter 10 and chapter 12 are 1019174 and 1019590, which are both smaller than 1019618.
@10SecondsToWaste, "Two On Ice" Manga ID is "100276"
In your loader.py
(develop branch, of course), change api/title_detailV2
to api/title_detailV3
and change the secret
again.
With just those changes, I am able to download the whole manga (17 chapters).
@rokibhasansagar Now there is no error, but chapters
is an empty set and mloader
just exits without downloading anything.
Just to be crystal clear, I installed the @develop
branch anew, then changed its AUTH_PARAMS
(with the correct secret of course) and the API endpoint like this:
@@ -23,10 +23,10 @@ log = logging.getLogger()
MangaList = Dict[int, Set[int]] # Title ID: Set[Chapter ID]
AUTH_PARAMS = {
- "app_ver": "1.8.3",
- "os": "ios",
- "os_ver": "15.5",
- "secret": "f40080bcb01a9a963912f46688d411a3",
+ "app_ver": "1.9.19",
+ "os": "android",
+ "os_ver": "7.1.2",
+ "secret": "[redacted]",
}
@@ -67,7 +67,7 @@ class MangaLoader:
@lru_cache(None)
def _get_title_details(self, title_id: Union[str, int]) -> TitleDetailView:
resp = self.session.get(
- f"{self._api_url}/api/title_detailV2",
+ f"{self._api_url}/api/title_detailV3",
params={
**AUTH_PARAMS,
"title_id": title_id,
You can see the change for yourself here. The same behaviour can be replicated using a GitHub codespace on this exact branch.
I also wiped all app data to get a new secret. Nothing changed.
@10SecondsToWaste, change the app_ver
to 97
,
That way, everything works.
P.S: Just use actual secret
key in production š
@rokibhasansagar Thanks a bunch! It's finally downloading.
Now, are there anything I can do to help this feature become a part of the @master
branch?
Attempting the same. Have my own secret key (paid for deluxe).
Modifying code from here to have
self._api_url = "https://jumpg-webapi.tokyo-cdn.com"
Tried above, failed. (Only free chapters found)
Tried modifying most recent master, failed. (Only free chapters found)
Attempted permutations of these AUTH_PARAMS:
"app_ver": "97",
"os": "android",
"os_ver": "13",
"os_ver": "7.1.2",
"User-Agent": "JumpPlus/1 CFNetwork/1333.0.4 Darwin/21.5.0",
"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:72.0) Gecko/20100101 Firefox/72.0",
"secret": "mysecretkey",
I have not installed mloader via pip, and am running it as follows:
python -m mloader.__main__ https://mangaplus.shueisha.co.jp/titles/<id>/ -o .
Does the method from the other posts still work? I will try to get the HTTP headers again and update my post if they work.
@hauveli, change the app_ver
to 124
at least. MangaPlus's Internal (API) App Version changed a month ago. Now, it can even be 131
if 124
does not work any more.
And using the Mobile API Endpoint instead of Web API Endpoint is also preferred.
Last but not least, add this change in your code too: https://github.com/hurlenko/mloader/issues/36#issuecomment-1916796950
Hi, so after reading through this thread several times, I'm still confused as to where to find this "key"? Like in the apk what is it called? What file/folder is it located in?? Additionally, can this same thing be found via iOS? Because I know the default setting for mloader is based on the iOS version of the app, so how can I scrape my "key" from there without having to change the entire api to only take in the android version? Thanks.
@rokibhasansagar How do you know the correct value for app_ver
? I wanted to download a new manga again today, and got the exact same problem. Both Play Store and App Store versions are specified as "1.9.*". I guess this is an internal detail?
@10SecondsToWaste @ThoughtzThruKeyz As I have no idea about iOS, I can share my method for Android only.
To get the app_ver
, you need to install "Lucky Patcher" via official "LuckyPatcherInstaller". Details on the first pin post in "r/luckypatcher" subreddit. After installation, Tap the MangaPlus app and from the options list, select "App Info". You'll see Build
after the "Version" string.
To get the secret
key, you need a network capture app, I recommend "Reqable" or it's previous name "HttpCanary" (from APKPure), or other apps. Root might be needed and works fine until Android v7.1.2 (not tested on latest Android).
@rokibhasansagar How do you know the correct value for
app_ver
? I wanted to download a new manga again today, and got the exact same problem. Both Play Store and App Store versions are specified as "1.9.*". I guess this is an internal detail?
It used to be the app version for iPhones from the Apple app store. The app versions for iOS apps are below the description on the app store website in the change log.
Fyi for On android every app has the app version number on the Play Store page. It's derived from the APK or it at least used to be that way. Its located in the info about the app where all the other info is ( e. g. compatible android versions).
So for the versions number you don't even need the app just a web browser.
@meshstyles
The app_ver
is Not the simple version of the app anymore. If you'd read all the comments and the APIv3 AUTH_PARAMS
, you'd know.
It needs a build number of the APK (for Android).
The App's public version string (also know as Release version) is indeed 1.9.19 (and someone might be getting 1.9.20 too), but the Internal app_ver
string in APIv3 Calls don't use that anymore. Currently, for my Android 14, the build id is 165
. That's to be put as app_ver
inside AUTH_PARAMS
.
I also explained it in https://github.com/hurlenko/mloader/issues/36#issuecomment-1916796950 and https://github.com/hurlenko/mloader/issues/36#issuecomment-1918412990 several months ago.
@rokibhasansagar thanks a lot for the help to get the version and the secret key. I am still struggling on the part for the secret key tho, I have installed the reqable app, but I am not sure what to look for from there. I launch it and analyse the requests that are made from the MangaPlus app but can't find any secret key, where should I look ? Thanks again for your help ! :)
@gauchoux As I recently had a new smartphone with Android 14, I can't get a working Network Capturing app to fully work in my device for Android's updated Security changes. I could with Reqable app, but it needs a rooted device to do so now, and I can't root my phone while being under Warranty period. Reqable's previous brand name, "HttpCanary" (old version) did work fully on my old Android 7.1.2 rooted device, but that's dead now :( and the VPN generated by Reqable can't connect to MangaPlus's server on my new device. So, I too can't get a new secret key anymore š¢ Fun fact: Reqable fully works on Facebook Lite app though š
I have to find other Network Capturing apps which can work on unrooted latest Android system š
Hi!
I've been trying to get the secret key too and I would ask for some help here.
I tried with Reqable but I feeled like @gauchoux, I didn't know what I was looking for.
I also tried installing MangaPlus apk to my using WSATools, but I'm finding the path that @BenjaminOchocki was saying.
/data/data/{M+ APP directory}/shared_prefs/config.xml
I'll go to rest now but I would apreciate some help for tomorrow.
PD: Thanks a lot for the project.
https://github.com/hurlenko/mloader/issues/36#issuecomment-1917778485
2nd day, 2nd round. This way worked! I've been able to download an issue only aviable in app.
lowitle@JMBR:~$ mloader https://mangaplus.shueisha.co.jp/titles/200007
_ _
_ __ ___ | | ___ __ _ __| | ___ _ __
| '_ ` _ \| |/ _ \ / _` |/ _` |/ _ \ '__|
| | | | | | | (_) | (_| | (_| | __/ |
|_| |_| |_|_|\___/ \__,_|\__,_|\___|_|
06.11.2024 22:51:06 | INFO | __main__.py 202 | Started export
06.11.2024 22:51:07 | INFO | loader.py 145 | 1/1) Manga: One Piece re-edition
06.11.2024 22:51:07 | INFO | loader.py 146 | Author: Eiichiro Oda
06.11.2024 22:51:07 | INFO | loader.py 157 | 1/10) Chapter #001: CapĆtulo 1:ROMANCE DAWN āEL AMANECER DE LA AVENTURAā
#001 [####################################] 51/51
06.11.2024 22:51:28 | INFO | loader.py 157 | 2/10) Chapter #002: CapĆtulo 2:ESE HOMBRE, LUFFY āSOMBRERO DE PAJAā
#002 [####################################] 23/23
06.11.2024 22:51:35 | INFO | loader.py 157 | 3/10) Chapter #003: CapĆtulo 3:LA APARICIĆN DE ZORO āEL CAZAPIRATASā
#003 [####################################] 21/21
06.11.2024 22:51:43 | INFO | loader.py 157 | 4/10) Chapter #007: CapĆtulo 7:AMIGOS
#007 [####################################] 20/20
06.11.2024 22:51:50 | INFO | loader.py 157 | 5/10) Chapter #008: CapĆtulo 8:LA APARICIĆN DE NAMI
#008 [####################################] 19/19
06.11.2024 22:51:56 | INFO | loader.py 157 | 6/10) Chapter #009: CapĆtulo 9:LA HECHICERA DIABĆLICA
#009 [####################################] 21/21
06.11.2024 22:52:02 | INFO | loader.py 157 | 7/10) Chapter #010: CapĆtulo 10:LO QUE SUCEDIĆ EN LA TABERNA
#010 [####################################] 23/23
06.11.2024 22:52:09 | INFO | loader.py 157 | 8/10) Chapter #011: CapĆtulo 11:A LA FUGA
#011 [####################################] 19/19
06.11.2024 22:52:15 | INFO | loader.py 157 | 9/10) Chapter #012: CapĆtulo 12:EL PERRO
#012 [####################################] 21/21
06.11.2024 22:52:22 | INFO | loader.py 157 | 10/10) Chapter #013: CapĆtulo 13:UN TESORO
#013 [####################################] 19/19
06.11.2024 22:52:22 | INFO | __main__.py 219 | SUCCESS
I used the "Title" option and mloader downloaded "aviable chapters" but these are not the aviable chapters that you can see through the web. Using web browser the chapters that I can download are: 001, 002, 003, 290, 291, 292, 293, 294, 295 and 296.
Note that oficially at website I have a total of 10 chapters aviable, mloader downloads 10 chapters, but starting with chapter 7 (after the 1st 3 episodes). I'm sure this is not a coincidence.
EDIT: Silly of me, for sure is not a coincidence, 10 chapters are the ones aviable using the app with the device I used to get the secred, app_ver, etc.
I think that I'll try to subscribe one month, let's see what can this marvelous app do!
Woah! I've been trying to figure this out for months! Do you mind assisting me in finding the secret key? @Lowitle
I used Reqable on an old android phone where I could install Magisk.
With Magisk installed I had root privileges and could install the "Reqable Certificate Installer".
With that Certificate installed the info I got from the Reqable monitoring was complete, when I navigate through diferent mangas I could see diferent traffic monitored in the app, Some of this traffic in the URL has all the info we need: "app_ver" and "secret".
The full URL is something like this:
https://jumpg-api.tokyo-cdn.com/api/comment?chapter_id=1000306&os=android&os_ver=33&app_ver=182&secret=d54a7d4a703512345610bf9e3a386e11
I hope I helped. I trully would've liked to use some standard method like an android emulator, maybe tomorrow I'll try something with Android Studio.
Let me know your progress about this, I'll try to help.
Hey thanks for that! Unfortuantely, I can't even get past the simplest step of installing magisk lol. I suspect it's some type of restriction on my device
I used Reqable on an old android phone where I could install Magisk.
With Magisk installed I had root privileges and could install the "Reqable Certificate Installer".
With that Certificate installed the info I got from the Reqable monitoring was complete, when I navigate through diferent mangas I could see diferent traffic monitored in the app, Some of this traffic in the URL has all the info we need: "app_ver" and "secret". The full URL is something like this:
https://jumpg-api.tokyo-cdn.com/api/comment?chapter_id=1000306&os=android&os_ver=33&app_ver=182&secret=d54a7d4a703512345610bf9e3a386e11
I hope I helped. I trully would've liked to use some standard method like an android emulator, maybe tomorrow I'll try something with Android Studio.
Let me know your progress about this, I'll try to help.
Hello @ThoughtzThruKeyz,
If you don't have a rootable phone, you can do what I've just done.
Follow this guide to have WSA (Windows Subsystem for Android) on your PC with Magisk and Google Apps (After following this guide you'll have Google Play and Magysk runing on your PC) https://github.com/LSPosed/MagiskOnWSALocal
Install Manga Plus
Install Reqable
Install Certificate, select the Magisk option, save zip file.
Go to Magisk app and flash zip file (I had to restart the Magisk app because I couldn't see the zip certificate).
Then start monitoring with Reqable.
Navigate through Manga Plus app.
Now with that Certificate installed we get info from the Reqable app when we navigate through the app. We can see diferent traffic monitored. Some of this traffic in the URL has all the info we need: "app_ver" and "secret". The full URL is something like this: https://jumpg-api.tokyo-cdn.com/api/comment?chapter_id=1000306&os=android&os_ver=33&app_ver=182&secret=d54a7d4a703512345610bf9e3a386e11
Like the title suggest there are free chapters and mangas which are App exclusive. Maybe you cant implement them.