Closed NVriezen closed 1 year ago
Another failed run, just to show that it's not always the same error or the same title.
Fetching metadata for title 20010000027788 (775 out of 1707)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom("missing field `title`")', src\main.rs:555:39
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\saveShop.exe --platform wiiu --regions NL fetch-all --cert ctr-common-1.pem` (exit code: 101)
What does http_log
say about the failed files? I wonder if the recorded content-length
matches the file size on disk, or if there are any other indicators of a partial download.
What does
http_log
say about the failed files? I wonder if the recordedcontent-length
matches the file size on disk, or if there are any other indicators of a partial download.
I'm downloading the media I can currently.
Will take a look at the http_log
when it's done.
Better have some stuff than nothing 😅
Had these problems as well today, yesterday it was completely fine. When I looked into the xml response it said some proxy problems. Maybe they've already shut down some things (EU Region). It only occurs on metadata fetch
When I looked into the xml response it said some proxy problems
I'm poking blind in this issue, but this would've been the content to post here... all these "it fails" reports are nice and all, but without actual information to go by there's nothing I could even try to do. Do you still have the xml response?
A full Wii U JP metadata fetch was completed without issue a few hours ago, so it still works in principle.
When I looked into the xml response it said some proxy problems
I'm poking blind in this issue, but this would've been the content to post here... all these "it fails" reports are nice and all, but without actual information to go by there's nothing I could even try to do. Do you still have the xml response?
A full Wii U JP metadata fetch was completed without issue a few hours ago, so it still works in principle.
I think there is nothing you can do since it's a server side issue
but here is the response
<!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN">
The proxy server received an invalid
response from an upstream server.
The proxy server could not handle the request GET /samurai/ws/DE/title/20010000005244.
Reason: Error reading from remote server
What's the corresponding entry for that file in http_log
?
What's the corresponding entry for that file in
http_log
?
{
"url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000005244?shop_id=2&lang=de",
"response_headers": {
"server": "nginx",
"date": "Sun, 26 Mar 2023 14:15:46 GMT",
"content-type": "text/html; charset=iso-8859-1",
"content-length": "447",
"connection": "keep-alive"
}
}
--------------------------------------------------
{
"url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000005244/ec_info?shop_id=2&lang=de",
"response_headers": {
"server": "nginx",
"date": "Sun, 26 Mar 2023 14:15:46 GMT",
"content-type": "application/xml",
"content-length": "250",
"connection": "keep-alive",
"vary": "Accept-Encoding"
}
}
--------------------------------------------------
{
"url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000005244",
"response_headers": {
"server": "nginx",
"date": "Sun, 26 Mar 2023 14:15:46 GMT",
"content-type": "application/xml",
"content-length": "1048",
"connection": "keep-alive",
"vary": "Accept-Encoding"
}
}
--------------------------------------------------
Here are the last 3 entries of the file
Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.
Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.
saveShop should already retry on network failure, but what's interesting here is that the server evidently returns an error message but saveShop's underlying networking library doesn't report this error.
Does the error happen quickly for you? If yes (or you don't mind waiting), could you try another run with the following patch to produce some extra debug logs?
src/main.rs | 4 +++-
1 file changed, 3 insertions(+), 1 deletion(-)
diff --git src/main.rs src/main.rs
index 6303c68..f9ce9d6 100644
--- src/main.rs
+++ src/main.rs
@@ -87,8 +87,10 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
C: Fn(reqwest::Response) -> F,
F: std::future::Future<Output = Result<Output, reqwest::Error>> {
return loop {
+ println!(" get_with_retry_generic: {}", &url);
let err = match request.try_clone().unwrap().send().await {
Ok(response) => {
+ println!(" get_with_retry_generic Ok(): {} for url {}", response.status(), &url);
let headers = response.headers().clone();
match continuation(response).await {
Ok(response_text) => {
@@ -104,7 +106,7 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild
Err(err) => err,
}
}
- Err(err) => err,
+ Err(err) => { println!(" get_with_retry_generic Err(): {} for url {}", err, &url); err },
};
println!(" Got error {}, retrying in 10 seconds", err);
thread::sleep(time::Duration::from_secs(10));
(Note you can run saveShop from any directory, so you don't need to risk invalidating your existing archive.)
Maybe it's possible to code in a retry, because luckily I already had a full metadata backup and could just manually fetch the title metadata afterwards.
saveShop should already retry on network failure, but what's interesting here is that the server evidently returns an error message but saveShop's underlying networking library doesn't report this error.
Does the error happen quickly for you? If yes, could you try another run with the following patch to produce some extra debug logs?
src/main.rs | 4 +++- 1 file changed, 3 insertions(+), 1 deletion(-) diff --git src/main.rs src/main.rs index 6303c68..f9ce9d6 100644 --- src/main.rs +++ src/main.rs @@ -87,8 +87,10 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild C: Fn(reqwest::Response) -> F, F: std::future::Future<Output = Result<Output, reqwest::Error>> { return loop { + println!(" get_with_retry_generic: {}", &url); let err = match request.try_clone().unwrap().send().await { Ok(response) => { + println!(" get_with_retry_generic Ok(): {} for url {}", response.status(), &url); let headers = response.headers().clone(); match continuation(response).await { Ok(response_text) => { @@ -104,7 +106,7 @@ async fn get_with_retry_generic<U, C, F, Output>(request: &reqwest::RequestBuild Err(err) => err, } } - Err(err) => err, + Err(err) => { println!(" get_with_retry_generic Err(): {} for url {}", err, &url); err }, }; println!(" Got error {}, retrying in 10 seconds", err); thread::sleep(time::Duration::from_secs(10));
(Note you can run saveShop from any directory, so you don't need to risk invalidating your existing archive.)
I will try a run, but sometimes it happens on the 24th out of ~1500 and sometimes at 900 or 1100
I will try a run, but sometimes it happens on the 24th out of ~1500 and sometimes at 900 or 1100
Try changing the FETCH_DELAY
at the top of the file to 0, that should speed it up a little at least.
Fetching metadata for title 20010000000928 (37 out of 1671) get_with_retry_generic: https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de get_with_retry_generic Ok(): 500 Internal Server Error for url https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de get_with_retry_generic: https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de get_with_retry_generic Ok(): 200 OK for url https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de get_with_retry_generic: https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928 get_with_retry_generic Ok(): 200 OK for url https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928 thread 'main' panicked at 'called
Result::unwrap()
on anErr
value: Custom("missing fieldtitle
")', src\main.rs:564:39 note: run withRUST_BACKTRACE=1
environment variable to display a backtrace
http_log
{
"url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/DE/title/20010000000928?shop_id=2&lang=de",
"response_headers": {
"server": "nginx",
"date": "Mon, 27 Mar 2023 13:59:12 GMT",
"content-type": "application/xml",
"content-length": "314",
"connection": "keep-alive",
"access-control-allow-origin": "*"
}
}
--------------------------------------------------
{
"url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/title/20010000000928/ec_info?shop_id=2&lang=de",
"response_headers": {
"server": "nginx",
"date": "Mon, 27 Mar 2023 13:59:12 GMT",
"content-type": "application/xml",
"content-length": "250",
"connection": "keep-alive",
"vary": "Accept-Encoding"
}
}
--------------------------------------------------
{
"url": "https://ninja.ctr.shop.nintendo.net/ninja/ws/DE/titles/online_prices?shop_id=2&lang=de&title[]=20010000000928",
"response_headers": {
"server": "nginx",
"date": "Mon, 27 Mar 2023 13:59:12 GMT",
"content-type": "application/xml",
"content-length": "345",
"connection": "keep-alive",
"vary": "Accept-Encoding"
}
}
--------------------------------------------------
<?xml version="1.0" encoding="UTF-8" standalone="yes"?><eshop><error><code>3003</code><message>Ein Fehler ist aufgetreten.
Bitte starte den Nintendo eShop
erneut und versuche es noch einmal.
Wenn das Problem bestehen bleibt,
notiere dir den Fehler-Code und
rufe support.nintendo.com auf.</message></error></eshop>
Ok this time a different error. The message just says an error occured restart the eshop
Oh well, they shut it down, or "maintance mode", I can not open the eshop with my 3ds
That's in line with what I was suspecting, thanks for confirming!
It turns out the reqwest
library that saveShop uses internally only returns errors in exceptional circumstances. Non-success status codes have to be checked for explicitly. The library documentation vaguely implies this, but the introductory examples are somewhat misleading about it.
The good news is that now that we know this, fixing it is trivial. If you update to 64781b94ba828ffd4899efd78c8b7b80ee4fb206, everything should work fine now!
Oh well, they shut it down, or "maintance mode", I can not open the eshop with my 3ds
As far as the eShop API itself is concerned, at least some regions (JP, DE) are still accessible. But yeah, time is running out :(
As far as the eShop API itself is concerned, at least some regions (JP, DE) are still accessible. But yeah, time is running out :(
Yea my media download is still working and running. Anyways I think it is time to say goodbye to the eshop :D thank you for the effort and for this application ❤️
Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.
Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.
As far as I can tell, none of the eShop APIs used by saveShop are in maintenance right now!
Lets just hope after maintenance is finished, the metadata will still be available, or that someone has backed it all up using their own tools previously.
As far as I can tell, none of the eShop APIs used by saveShop are in maintenance right now!
For me after downloading the latest code it isn't able to fetch past ranking. It tries to fetch director metadata, but it return error 404. So I think it's too late :( This is for Wii U btw.
Thanks again for your efforts and the late patch!
It tries to fetch director metadata, but it return error 404. So I think it's too late :( This is for Wii U btw.
Actually this is normal: Wii U has "alias" directories that aren't backed by server-side resources. saveShop shouldn't attempt to scrape those in the first place, but as a workaround I just made it not retry fetching for 404 errors in 9a9ab28002fa3a5228de343673b0f98a6e370b3b.
You may try your luck yet one more time :)
You may try your luck yet one more time :)
It seems to be working! 😄 I do get more errors, but after saveShop retries them they seem to work just fine. So it probably takes a bit longer, but I think this way I am at least able to fetch all metadata! Thanks again! You're doing amazing work!
When doing a fresh run for the Wii U with the latest version of saveShop, it halts at different titles or rankings. The error is also different every time.
I already tried the things mentioned in my previous issue #11 However deleting all metadata does not help unfortunately.
The last error is as follows, happening when fetching data for title
20010000008386
:As you can see this error is an
UnexpectedEof
error, but I've also seen theno ranking
error. But it happened with different titles. As far as I understand I can just use the 3DS certificate, so that's what I am using for the run.