Closed pokemontest1234 closed 1 year ago
Thanks for testing!
Looks like there's another edge case for Wii U metadata. The error you found should be fixed using this additional patch:
diff --git src/main.rs src/main.rs
index 3009681..327bb77 100644
--- src/main.rs
+++ src/main.rs
@@ -432,6 +432,7 @@ async fn fetch_content_list(client: &reqwest::Client, endpoint: EndPoint, locale
if doc.contents.total == 0 {
println!("No contents available");
+ full_list.push(resp);
break;
}
@@ -539,6 +540,7 @@ async fn handle_directory_content(client: &reqwest::Client, directory_id: &str,
if doc.directory.contents.as_ref().map(|c| c.total).unwrap_or(0) == 0 {
println!("No contents available");
+ full_list.push(resp);
directory_info = Some(doc);
break;
}
@@ -1012,7 +1014,6 @@ async fn fetch_media_resources(client: &reqwest::Client, region: &str, args: &Ar
println!(" Directory {} ({} out of {})", &directory.display(), dir_index + 1, directory_set.len());
let parsed_xml: DirectoryDocument = quick_xml::de::from_str(&String::from_utf8(fs::read(directory).unwrap()).unwrap()).unwrap();
let directory = parsed_xml.directory;
- assert!(directory.contents.is_some());
println!(" Name: {}", &directory.name.replace("\n", " "));
After applying the patch, you'll need to re-fetch metadata for the affected titles. The most reliable way of doing so is deleting the samurai/AT folder and redownloading all metadata from scratch.
Let me know if you find any other issues!
thanks for the response and the patch worked for images but it doesn't work for videos. it might have something to do with 3ds using moflex while wii u uses mp4
Error message when trying to download videos:
fetching movie from https://kanzashi-movie-wup.cdn.nintendo.net/m/b0243a3177f21445671855a0f27358d2c7c44bd87c610f2482b5430d79f6c80e.mp4
thread 'main' panicked at 'called Option::unwrap()
on a None
value', src\main.rs:845:101
I can't look into that issue without knowing what title that movie is from. What does the log say?
Title: samurai/AT\de\title\20010000000242 (1 out of 1710) Name: Nano Assault Neo and http_log doesn't mention the mp4.(or do you mean another log file?) the last thing it mentions is
{
"url": "https://kanzashi-wup.cdn.nintendo.net/i/02aa23f21a40d43741ce6ee0875ff92e761c0a8df6bd1f0a913f4356c852bbd2.jpg",
"response_headers": {
"server": "nginx",
"content-type": "image/jpeg",
"content-length": "38733",
"accept-ranges": "bytes",
"last-modified": "Tue, 08 Sep 2015 22:29:24 GMT",
"date": "Sat, 18 Mar 2023 20:19:10 GMT",
"connection": "keep-alive"
}
}
-------------------------------------------------
and if you need the rest of the output before it crashes at the mp4 here:
Fetching icon from https://kanzashi-wup.cdn.nintendo.net/i/ff6e75fd59ec61b33bd8b313dfa47347c32074d949ab4ac7b29ab5c809fc29f7.jpg
... already exists on disk (7 KiB), skipping
Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/84092be964bf17ec0574d6687c18f2d0880fdf7c153770e1272bfe82ecd1da25.jpg
... already exists on disk (161 KiB), skipping
Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/2f79a2a29115a348ea550db26737a60a5a4b566d447790c9b69973efd8547521.jpg
... already exists on disk (3 KiB), skipping
Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/e52dc024128fd00254365ab56ed8188b1580ee7261139708be9beeb906e60547.jpg
... already exists on disk (2 KiB), skipping
Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/c4620c4104c29c1f1451b88491f63f8fb50f55c25eae732a12da168013ec5bf7.jpg
... already exists on disk (559 KiB), skipping
Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/fa58b9f10cb034fafb1bacb27190ec2665ce68417ded14ec2b91cec533e301dc.jpg
... already exists on disk (15 KiB), skipping
Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/02cdf91e7b4d6c2a2b97a48fcfc6faa32f1a71c5344735d7c5514c1507ac880f.jpg
... already exists on disk (357 KiB), skipping
Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/69e43f4132f9b24b54f8f412196ca029a64997479014116b50aed293101ace9e.jpg
... already exists on disk (12 KiB), skipping
Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/7a001958a6c31181913d5d34cc1ebf07812cbffc920d5bc0cb803fa696cb9a0b.jpg
... already exists on disk (423 KiB), skipping
Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/7cc2f8adb3be73574b60d29a9a5c974ed31e0bd2f55bf4ff182c0c50d08362fe.jpg
... already exists on disk (13 KiB), skipping
Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/18fe5f3aa26371e677d49a7061ff8a84bff8b8a8e52b5841b76d855dafdec120.jpg
... already exists on disk (559 KiB), skipping
Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/965caeb409df23caf8d1d820c60d74cadcce3b706a0e2aa58220047a78a9219a.jpg
... already exists on disk (14 KiB), skipping
Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/48e13f31a2d067f4a64f2b69909c5b18361f41ef27abd52f63fb01e4aaf51b47.jpg
... already exists on disk (603 KiB), skipping
Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/3dcda6ee292cecbcc5884a821b71a16b5808038ce11de396d3e136a7da157cea.jpg
... already exists on disk (17 KiB), skipping
Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/02aa23f21a40d43741ce6ee0875ff92e761c0a8df6bd1f0a913f4356c852bbd2.jpg
... already exists on disk (37 KiB), skipping
Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/2f79a2a29115a348ea550db26737a60a5a4b566d447790c9b69973efd8547521.jpg (3 KiB, cached)
Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/e52dc024128fd00254365ab56ed8188b1580ee7261139708be9beeb906e60547.jpg (2 KiB, cached)
The Title: samurai/AT\de\title\20010000000242 (1 out of 1710)
was what I needed here, thanks :)
Looks like we're getting there. Here's another patch:
diff --git src/main.rs src/main.rs
index 327bb77..1bd86d8 100644
--- src/main.rs
+++ src/main.rs
@@ -848,7 +848,14 @@ fn url_to_filename(url: &str) -> String {
}
fn movie_url_to_filename(url: &str) -> String {
- format!("kanzashi-movie/{}", url.strip_prefix("https://kanzashi-movie-ctr.cdn.nintendo.net/m/").unwrap())
+ let url = url.strip_prefix("https://").unwrap();
+ let base_url = &url[0..url.find("/").unwrap() + 1];
+ let path = url.strip_prefix(base_url).unwrap();
+ match base_url {
+ "kanzashi-movie-ctr.cdn.nintendo.net/" => { format!("kanzashi-movie/{}", path.strip_prefix("m/").unwrap()) },
+ "kanzashi-movie-wup.cdn.nintendo.net/" => { format!("kanzashi-movie/{}", path.strip_prefix("m/").unwrap()) },
+ _ => panic!("Unrecognized resource URL \"{}\"", url)
+ }
}
async fn fetch_movie_file(client: &reqwest::Client, file: &NodeMovieFile) -> Result<(), Box<dyn std::error::Error>> {
@@ -860,9 +867,8 @@ async fn fetch_movie_file(client: &reqwest::Client, file: &NodeMovieFile) -> Res
}
let filename = movie_url_to_filename(&file.movie_url);
- assert_eq!("moflex", std::path::Path::new(&filename).extension().unwrap());
- // Skip if content size matches the moflex on disk
+ // Skip if content size matches the file on disk
if let Ok(existing_file) = fs::metadata(&filename) {
let response = client.get(&file.movie_url).send().await?;
let content_length = response.content_length();
No need to redownload metadata for this one. Just resume with fetch-media
.
thank you it is also now working with videos as i don't see any other issues come up im going to close this issue with this comment.Thank you so much for answering so fast
@pokemontest1234 great that its fixed, but probably best to keep the issue open until its merged into the repo.
sure
Patches are integrated in the latest version, along with a --platform
selection you can use to switch between Wii U and 3DS (use a separate directory for both though!)
after applying the patches from issue number #2 when trying to scrape media for the AT region i get this error message. how would i go about fixing this? Error Message: Directory samurai/AT/de/directory/1090722 (4 out of 51) Name: Videos Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/c3905b63ce0b7eb399d82fd05357564fecbfd13031dc498bccc931f72f5cf74e.jpg (3 KiB, cached) Directory samurai/AT/de/directory/1090723 (5 out of 51) thread 'main' panicked at 'called
Result::unwrap()
on anErr
value: UnexpectedEof', src/main.rs:1013:132 note: run withRUST_BACKTRACE=1
environment variable to display a backtrace