search

mikage-emu / saveShop

Let's save the 3DS and Wii U eShop!
90 stars 2 forks source link

Wii U fetch-all random errors #14

Closed NVriezen closed 1 year ago

NVriezen commented 1 year ago

When doing a fresh run for the Wii U with the latest version of saveShop, it halts at different titles or rankings. The error is also different every time.

I already tried the things mentioned in my previous issue #11 However deleting all metadata does not help unfortunately.

The last error is as follows, happening when fetching data for title 20010000008386:

thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedEof', src\main.rs:555:39
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\saveShop.exe --platform wiiu --regions NL fetch-all --cert ctr-common-1.pem` (exit code: 101)

As you can see this error is an UnexpectedEof error, but I've also seen the no ranking error. But it happened with different titles. As far as I understand I can just use the 3DS certificate, so that's what I am using for the run.

NVriezen commented 1 year ago

Another failed run, just to show that it's not always the same error or the same title.

Fetching metadata for title 20010000027788 (775 out of 1707)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom("missing field `title`")', src\main.rs:555:39
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\saveShop.exe --platform wiiu --regions NL fetch-all --cert ctr-common-1.pem` (exit code: 101)
neobrain commented 1 year ago

What does http_log say about the failed files? I wonder if the recorded content-length matches the file size on disk, or if there are any other indicators of a partial download.

NVriezen commented 1 year ago

What does http_log say about the failed files? I wonder if the recorded content-length matches the file size on disk, or if there are any other indicators of a partial download.

I'm downloading the media I can currently. Will take a look at the http_log when it's done. Better have some stuff than nothing 😅

Divi-0 commented 1 year ago

Had these problems as well today, yesterday it was completely fine. When I looked into the xml response it said some proxy problems. Maybe they've already shut down some things (EU Region). It only occurs on metadata fetch

neobrain commented 1 year ago

When I looked into the xml response it said some proxy problems

I'm poking blind in this issue, but this would've been the content to post here... all these "it fails" reports are nice and all, but without actual information to go by there's nothing I could even try to do. Do you still have the xml response?

A full Wii U JP metadata fetch was completed without issue a few hours ago, so it still works in principle.

Divi-0 commented 1 year ago

When I looked into the xml response it said some proxy problems

I'm poking blind in this issue, but this would've been the content to post here... all these "it fails" reports are nice and all, but without actual information to go by there's nothing I could even try to do. Do you still have the xml response?

A full Wii U JP metadata fetch was completed without issue a few hours ago, so it still works in principle.

I think there is nothing you can do since it's a server side issue

but here is the response

<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">

502 Proxy Error

Proxy Error

The proxy server received an invalid response from an upstream server.
The proxy server could not handle the request GET /samurai/ws/DE/title/20010000005244.

Reason: Error reading from remote server

neobrain commented 1 year ago

What's the corresponding entry for that file in http_log?

Divi-0 commented 1 year ago

What's the corresponding entry for that file in http_log?

{
  "url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000005244?shop_id=2&lang=de",
  "response_headers": {
    "server": "nginx",
    "date": "Sun, 26 Mar 2023 14:15:46 GMT",
    "content-type": "text/html; charset=iso-8859-1",
    "content-length": "447",
    "connection": "keep-alive"
  }
}
--------------------------------------------------
{
  "url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000005244/ec_info?shop_id=2&lang=de",
  "response_headers": {
    "server": "nginx",
    "date": "Sun, 26 Mar 2023 14:15:46 GMT",
    "content-type": "application/xml",
    "content-length": "250",
    "connection": "keep-alive",
    "vary": "Accept-Encoding"
  }
}
--------------------------------------------------
{
  "url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000005244",
  "response_headers": {
    "server": "nginx",
    "date": "Sun, 26 Mar 2023 14:15:46 GMT",
    "content-type": "application/xml",
    "content-length": "1048",
    "connection": "keep-alive",
    "vary": "Accept-Encoding"
  }
}
--------------------------------------------------

Here are the last 3 entries of the file

Divi-0 commented 1 year ago

Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.

neobrain commented 1 year ago

Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.

saveShop should already retry on network failure, but what's interesting here is that the server evidently returns an error message but saveShop's underlying networking library doesn't report this error.

Does the error happen quickly for you? If yes (or you don't mind waiting), could you try another run with the following patch to produce some extra debug logs?

 src/main.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git src/main.rs src/main.rs
index 6303c68..f9ce9d6 100644
--- src/main.rs
+++ src/main.rs
@@ -87,8 +87,10 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
             C: Fn(reqwest::Response) -> F,
             F: std::future::Future<Output = Result<Output, reqwest::Error>> {
     return loop {
+        println!("  get_with_retry_generic: {}", &url);
         let err = match request.try_clone().unwrap().send().await {
                 Ok(response) => {
+                println!("  get_with_retry_generic Ok(): {} for url {}", response.status(), &url);
                 let headers = response.headers().clone();
                 match continuation(response).await {
                     Ok(response_text) => {
@@ -104,7 +106,7 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
                     Err(err) => err,
                 }
             }
-            Err(err) => err,
+            Err(err) => { println!("  get_with_retry_generic Err(): {} for url {}", err, &url); err },
         };
         println!("  Got error {}, retrying in 10 seconds", err);
         thread::sleep(time::Duration::from_secs(10));

(Note you can run saveShop from any directory, so you don't need to risk invalidating your existing archive.)

Divi-0 commented 1 year ago

Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.

saveShop should already retry on network failure, but what's interesting here is that the server evidently returns an error message but saveShop's underlying networking library doesn't report this error.

Does the error happen quickly for you? If yes, could you try another run with the following patch to produce some extra debug logs?

 src/main.rs | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)

diff --git src/main.rs src/main.rs
index 6303c68..f9ce9d6 100644
--- src/main.rs
+++ src/main.rs
@@ -87,8 +87,10 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
             C: Fn(reqwest::Response) -> F,
             F: std::future::Future<Output = Result<Output, reqwest::Error>> {
     return loop {
+        println!("  get_with_retry_generic: {}", &url);
         let err = match request.try_clone().unwrap().send().await {
                 Ok(response) => {
+                println!("  get_with_retry_generic Ok(): {} for url {}", response.status(), &url);
                 let headers = response.headers().clone();
                 match continuation(response).await {
                     Ok(response_text) => {
@@ -104,7 +106,7 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
                     Err(err) => err,
                 }
             }
-            Err(err) => err,
+            Err(err) => { println!("  get_with_retry_generic Err(): {} for url {}", err, &url); err },
         };
         println!("  Got error {}, retrying in 10 seconds", err);
         thread::sleep(time::Duration::from_secs(10));

(Note you can run saveShop from any directory, so you don't need to risk invalidating your existing archive.)

I will try a run, but sometimes it happens on the 24th out of ~1500 and sometimes at 900 or 1100

neobrain commented 1 year ago

I will try a run, but sometimes it happens on the 24th out of ~1500 and sometimes at 900 or 1100

Try changing the FETCH_DELAY at the top of the file to 0, that should speed it up a little at least.

Divi-0 commented 1 year ago

Fetching metadata for title 20010000000928 (37 out of 1671) get_with_retry_generic: https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de get_with_retry_generic Ok(): 500 Internal Server Error for url https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de get_with_retry_generic: https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de get_with_retry_generic Ok(): 200 OK for url https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de get_with_retry_generic: https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928 get_with_retry_generic Ok(): 200 OK for url https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928 thread 'main' panicked at 'called Result::unwrap() on an Err value: Custom("missing field title")', src\main.rs:564:39 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

http_log

{
  "url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de",
  "response_headers": {
    "server": "nginx",
    "date": "Mon, 27 Mar 2023 13:59:12 GMT",
    "content-type": "application/xml",
    "content-length": "314",
    "connection": "keep-alive",
    "access-control-allow-origin": "*"
  }
}
--------------------------------------------------
{
  "url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de",
  "response_headers": {
    "server": "nginx",
    "date": "Mon, 27 Mar 2023 13:59:12 GMT",
    "content-type": "application/xml",
    "content-length": "250",
    "connection": "keep-alive",
    "vary": "Accept-Encoding"
  }
}
--------------------------------------------------
{
  "url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928",
  "response_headers": {
    "server": "nginx",
    "date": "Mon, 27 Mar 2023 13:59:12 GMT",
    "content-type": "application/xml",
    "content-length": "345",
    "connection": "keep-alive",
    "vary": "Accept-Encoding"
  }
}
--------------------------------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><eshop><error><code>3003</code><message>Ein Fehler ist aufgetreten.
Bitte starte den Nintendo eShop
erneut und versuche es noch einmal.

Wenn das Problem bestehen bleibt,
notiere dir den Fehler-Code und
rufe support.nintendo.com auf.</message></error></eshop>

Ok this time a different error. The message just says an error occured restart the eshop

Divi-0 commented 1 year ago

Oh well, they shut it down, or "maintance mode", I can not open the eshop with my 3ds

neobrain commented 1 year ago

That's in line with what I was suspecting, thanks for confirming!

It turns out the reqwest library that saveShop uses internally only returns errors in exceptional circumstances. Non-success status codes have to be checked for explicitly. The library documentation vaguely implies this, but the introductory examples are somewhat misleading about it.

The good news is that now that we know this, fixing it is trivial. If you update to 64781b94ba828ffd4899efd78c8b7b80ee4fb206, everything should work fine now!

Oh well, they shut it down, or "maintance mode", I can not open the eshop with my 3ds

As far as the eShop API itself is concerned, at least some regions (JP, DE) are still accessible. But yeah, time is running out :(

Divi-0 commented 1 year ago

As far as the eShop API itself is concerned, at least some regions (JP, DE) are still accessible. But yeah, time is running out :(

Yea my media download is still working and running. Anyways I think it is time to say goodbye to the eshop :D thank you for the effort and for this application ❤️

mariomadproductions commented 1 year ago

Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.

neobrain commented 1 year ago

Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.

As far as I can tell, none of the eShop APIs used by saveShop are in maintenance right now!

NVriezen commented 1 year ago

Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.

As far as I can tell, none of the eShop APIs used by saveShop are in maintenance right now!

For me after downloading the latest code it isn't able to fetch past ranking. It tries to fetch director metadata, but it return error 404. So I think it's too late :( This is for Wii U btw.

Thanks again for your efforts and the late patch!

neobrain commented 1 year ago

It tries to fetch director metadata, but it return error 404. So I think it's too late :( This is for Wii U btw.

Actually this is normal: Wii U has "alias" directories that aren't backed by server-side resources. saveShop shouldn't attempt to scrape those in the first place, but as a workaround I just made it not retry fetching for 404 errors in 9a9ab28002fa3a5228de343673b0f98a6e370b3b.

You may try your luck yet one more time :)

NVriezen commented 1 year ago

You may try your luck yet one more time :)

It seems to be working! 😄 I do get more errors, but after saveShop retries them they seem to work just fine. So it probably takes a bit longer, but I think this way I am at least able to fetch all metadata! Thanks again! You're doing amazing work!