search

mikage-emu / saveShop

Let's save the 3DS and Wii U eShop!
90 stars 2 forks source link

Some missing domains #15

Open www2000 opened 1 year ago

www2000 commented 1 year ago

He i have done some datamining and found some more urls that missing img-eshop.cdn.nintendo.net # firth image cdn m1.nintendo.net # pdf www.nintendo.co.jp # pdf

neobrain commented 1 year ago

Could you provide more information on where these URLs referenced?

www2000 commented 1 year ago

I have use this script to file this url's https://pastebin.com/8CgQLz9N The url https://img-eshop.cdn.nintendo.net are found in unknown4 www.nintendo.co.jp, m1.nintendo.net and https://samurai-wup.cdn.nintendo.net/d/ i have find in the japanes set for the wiiu

urls looks like: ` https://img-eshop.cdn.nintendo.net/i/b312b2db4bfa6c77f3528bc5506cb4af5414de8974aaee3402335ff8e4fb1b0f.jpg

https://samurai-wup.cdn.nintendo.net/d/man_JCXJ_00.pdf

https://www.nintendo.co.jp/data/software/manual/27023.pdf

http://m1.nintendo.net/docvc/NUS/JPN/NMFJ/NMFJ_J.pdf `

neobrain commented 1 year ago

Can you tell me specific metadata files (from the samurai folder) these are referenced in so this can be verified independently? The links by themselves aren't too useful unless I understand how they are referenced in the metadata.

neobrain commented 1 year ago

For 3DS, only PDFs from samurai servers are referenced, and only on JP region for some titles. Sadly all of them return status 404 with the message "Sorry, not found" when trying to fetch them using curl, however.

Other than JP, only region ES has exactly one title (50010000044217) that references a PDF on a capcom server. This server is likely unaffected by the impending eShop shutdown, but since it's only one file it's out of scope for saveShop scraping. Interested people should download this file manually.

www2000 commented 1 year ago

the pdf's are in samurai/{}/{}/title/{titleid} and https://img-eshop.cdn.nintendo.net are the default of unknown4

www2000 commented 1 year ago

and try https://www.nintendo.co.jp/data/software/manual/{filename} as a backup option

neobrain commented 1 year ago

Very interesting, that works!

I won't be able to integrate this quickly enough into saveShop, but interested people can run the following UNIX command chain to download the files manually:

mkdir manuals && cd manuals
ack "\.pdf" ../samurai/JP/ja/ | tr '>' '\n' | grep 'https.*pdf' | cut -d'<' -f1 | sort | sed 's/samurai-ctr.cdn.nintendo.net\/d/www.nintendo.co.jp\/data\/software\/manual/g' | xargs -I{} wget '{}'