wiiu eshop media scrapper

[pokemontest1234]

after applying the patches from issue number #2 when trying to scrape media for the AT region i get this error message. how would i go about fixing this? Error Message: Directory samurai/AT/de/directory/1090722 (4 out of 51) Name: Videos Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/c3905b63ce0b7eb399d82fd05357564fecbfd13031dc498bccc931f72f5cf74e.jpg (3 KiB, cached) Directory samurai/AT/de/directory/1090723 (5 out of 51) thread 'main' panicked at 'called Result::unwrap() on an Err value: UnexpectedEof', src/main.rs:1013:132 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

[neobrain]

Thanks for testing!

Looks like there's another edge case for Wii U metadata. The error you found should be fixed using this additional patch:

diff --git src/main.rs src/main.rs
index 3009681..327bb77 100644
--- src/main.rs
+++ src/main.rs
@@ -432,6 +432,7 @@ async fn fetch_content_list(client: &reqwest::Client, endpoint: EndPoint, locale

         if doc.contents.total == 0 {
             println!("No contents available");
+            full_list.push(resp);
             break;
         }

@@ -539,6 +540,7 @@ async fn handle_directory_content(client: &reqwest::Client, directory_id: &str,

         if doc.directory.contents.as_ref().map(|c| c.total).unwrap_or(0) == 0 {
             println!("No contents available");
+            full_list.push(resp);
             directory_info = Some(doc);
             break;
         }
@@ -1012,7 +1014,6 @@ async fn fetch_media_resources(client: &reqwest::Client, region: &str, args: &Ar
             println!(" Directory {} ({} out of {})", &directory.display(), dir_index + 1, directory_set.len());
             let parsed_xml: DirectoryDocument = quick_xml::de::from_str(&String::from_utf8(fs::read(directory).unwrap()).unwrap()).unwrap();
             let directory = parsed_xml.directory;
-            assert!(directory.contents.is_some());

             println!("  Name: {}", &directory.name.replace("\n", " "));

After applying the patch, you'll need to re-fetch metadata for the affected titles. The most reliable way of doing so is deleting the samurai/AT folder and redownloading all metadata from scratch.

Let me know if you find any other issues!

[pokemontest1234]

thanks for the response and the patch worked for images but it doesn't work for videos. it might have something to do with 3ds using moflex while wii u uses mp4 Error message when trying to download videos: fetching movie from https://kanzashi-movie-wup.cdn.nintendo.net/m/b0243a3177f21445671855a0f27358d2c7c44bd87c610f2482b5430d79f6c80e.mp4 thread 'main' panicked at 'called Option::unwrap() on a None value', src\main.rs:845:101

[neobrain]

I can't look into that issue without knowing what title that movie is from. What does the log say?

[pokemontest1234]

Title: samurai/AT\de\title\20010000000242 (1 out of 1710) Name: Nano Assault Neo and http_log doesn't mention the mp4.(or do you mean another log file?) the last thing it mentions is

 {
  "url": "https://kanzashi-wup.cdn.nintendo.net/i/02aa23f21a40d43741ce6ee0875ff92e761c0a8df6bd1f0a913f4356c852bbd2.jpg",
  "response_headers": {
    "server": "nginx",
    "content-type": "image/jpeg",
    "content-length": "38733",
    "accept-ranges": "bytes",
    "last-modified": "Tue, 08 Sep 2015 22:29:24 GMT",
    "date": "Sat, 18 Mar 2023 20:19:10 GMT",
    "connection": "keep-alive"
  }
}
-------------------------------------------------

[pokemontest1234]

and if you need the rest of the output before it crashes at the mp4 here:

Fetching icon from https://kanzashi-wup.cdn.nintendo.net/i/ff6e75fd59ec61b33bd8b313dfa47347c32074d949ab4ac7b29ab5c809fc29f7.jpg
    ... already exists on disk (7 KiB), skipping
  Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/84092be964bf17ec0574d6687c18f2d0880fdf7c153770e1272bfe82ecd1da25.jpg
    ... already exists on disk (161 KiB), skipping
  Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/2f79a2a29115a348ea550db26737a60a5a4b566d447790c9b69973efd8547521.jpg
    ... already exists on disk (3 KiB), skipping
  Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/e52dc024128fd00254365ab56ed8188b1580ee7261139708be9beeb906e60547.jpg
    ... already exists on disk (2 KiB), skipping
  Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/c4620c4104c29c1f1451b88491f63f8fb50f55c25eae732a12da168013ec5bf7.jpg
    ... already exists on disk (559 KiB), skipping
  Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/fa58b9f10cb034fafb1bacb27190ec2665ce68417ded14ec2b91cec533e301dc.jpg
    ... already exists on disk (15 KiB), skipping
  Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/02cdf91e7b4d6c2a2b97a48fcfc6faa32f1a71c5344735d7c5514c1507ac880f.jpg
    ... already exists on disk (357 KiB), skipping
  Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/69e43f4132f9b24b54f8f412196ca029a64997479014116b50aed293101ace9e.jpg
    ... already exists on disk (12 KiB), skipping
  Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/7a001958a6c31181913d5d34cc1ebf07812cbffc920d5bc0cb803fa696cb9a0b.jpg
    ... already exists on disk (423 KiB), skipping
  Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/7cc2f8adb3be73574b60d29a9a5c974ed31e0bd2f55bf4ff182c0c50d08362fe.jpg
    ... already exists on disk (13 KiB), skipping
  Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/18fe5f3aa26371e677d49a7061ff8a84bff8b8a8e52b5841b76d855dafdec120.jpg
    ... already exists on disk (559 KiB), skipping
  Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/965caeb409df23caf8d1d820c60d74cadcce3b706a0e2aa58220047a78a9219a.jpg
    ... already exists on disk (14 KiB), skipping
  Fetching screenshot from https://kanzashi-wup.cdn.nintendo.net/i/48e13f31a2d067f4a64f2b69909c5b18361f41ef27abd52f63fb01e4aaf51b47.jpg
    ... already exists on disk (603 KiB), skipping
  Fetching thumbnail from https://kanzashi-wup.cdn.nintendo.net/i/3dcda6ee292cecbcc5884a821b71a16b5808038ce11de396d3e136a7da157cea.jpg
    ... already exists on disk (17 KiB), skipping
  Fetching banner from https://kanzashi-wup.cdn.nintendo.net/i/02aa23f21a40d43741ce6ee0875ff92e761c0a8df6bd1f0a913f4356c852bbd2.jpg
    ... already exists on disk (37 KiB), skipping
  Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/2f79a2a29115a348ea550db26737a60a5a4b566d447790c9b69973efd8547521.jpg (3 KiB, cached)
  Fetching rating icon from https://kanzashi-wup.cdn.nintendo.net/i/e52dc024128fd00254365ab56ed8188b1580ee7261139708be9beeb906e60547.jpg (2 KiB, cached)

[neobrain]

The Title: samurai/AT\de\title\20010000000242 (1 out of 1710) was what I needed here, thanks :)

Looks like we're getting there. Here's another patch:

diff --git src/main.rs src/main.rs
index 327bb77..1bd86d8 100644
--- src/main.rs
+++ src/main.rs
@@ -848,7 +848,14 @@ fn url_to_filename(url: &str) -> String {
 }

 fn movie_url_to_filename(url: &str) -> String {
-    format!("kanzashi-movie/{}", url.strip_prefix("https://kanzashi-movie-ctr.cdn.nintendo.net/m/").unwrap())
+    let url = url.strip_prefix("https://").unwrap();
+    let base_url = &url[0..url.find("/").unwrap() + 1];
+    let path = url.strip_prefix(base_url).unwrap();
+    match base_url {
+        "kanzashi-movie-ctr.cdn.nintendo.net/" => { format!("kanzashi-movie/{}", path.strip_prefix("m/").unwrap()) },
+        "kanzashi-movie-wup.cdn.nintendo.net/" => { format!("kanzashi-movie/{}", path.strip_prefix("m/").unwrap()) },
+        _ => panic!("Unrecognized resource URL \"{}\"", url)
+    }
 }

 async fn fetch_movie_file(client: &reqwest::Client, file: &NodeMovieFile) -> Result<(), Box<dyn std::error::Error>> {
@@ -860,9 +867,8 @@ async fn fetch_movie_file(client: &reqwest::Client, file: &NodeMovieFile) -> Res
     }

     let filename = movie_url_to_filename(&file.movie_url);
-    assert_eq!("moflex", std::path::Path::new(&filename).extension().unwrap());

-    // Skip if content size matches the moflex on disk
+    // Skip if content size matches the file on disk
     if let Ok(existing_file) = fs::metadata(&filename) {
         let response = client.get(&file.movie_url).send().await?;
         let content_length = response.content_length();

No need to redownload metadata for this one. Just resume with fetch-media.

[pokemontest1234]

thank you it is also now working with videos as i don't see any other issues come up im going to close this issue with this comment.Thank you so much for answering so fast

[mariomadproductions]

@pokemontest1234 great that its fixed, but probably best to keep the issue open until its merged into the repo.

[pokemontest1234]

sure

[neobrain]

Patches are integrated in the latest version, along with a --platform selection you can use to switch between Wii U and 3DS (use a separate directory for both though!)