An error occurs after a while of running the program

[puyo2a]

It seems to work well until halfway through, but the result is the same no matter which region.

thread 'main' panicked at 'called Result::unwrap() on an Err value: Custom("missing field ranking")', src\main.rs:651:67 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace error: process didn't exit successfully: target\debug\saveShop.exe fetch-all --regions JP --cert D:\3DS_HAX\saveshop\ctr-common-1.pem (exit code: 101)

[NVriezen]

Was this a fresh run? Maybe specifying the platform helps, if you have the latest version that is. Otherwise you can try deleting the old metadata and trying again. Though it can take a while again to redownload all metadata. To delete the old metadata you can delete the ninja and samurai folders as well as the http_log file. At least that helped me yesterday after downloading the new version of the tool.

[neobrain]

The command you posted works fine for me, so I'd also recommend trying with a fresh output folder. (You can speed up the scrape by changing FETCH_DELAY at the top of main.rs to 0.)

If that doesn't help, it would be interesting to know the last few lines of console output before the error message. There should be something along the lines of "Ranking contents..." or maybe just "Fetching endpoint rankings". It would also be interesting to know if any of the numbered files in samurai/JP/ja/ranking/ do not have <?xml...?><eshop><ranking id=... at the beginning.

We probably have any time left to debug this in detail. As a workaround you can skip fetching rankings by commenting out lines 950-954 in main.rs. Hopefully the rest of the metadata will run through fine!

[puyo2a]

Thank you for your kind.

This was a new run. I tried deleting ninja, samurai folders and http_log, and running it again, or commenting out lines 950-954 with FETCH_DELAY set to 0, then deleting the files and running the same way, but in the process of Fetching metadata for title After some time, an error occurred and I could not complete the process.

Also, there was no problem with the numbered files in samurai/JP/en/ranking/.

Are there other factors that could be contributing to this?

For reference, errors that occurred after several attempts:

Fetching metadata for title 50010000024675 (1418 out of 2414) thread 'main' panicked at 'called Result::unwrap() on an Err value: Custom("missing field title")', src\main.rs:555:39 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Fetching metadata for title 50010000005682 (310 out of 2414) thread 'main' panicked at 'called Result::unwrap() on an Err value: Custom("missing field title")', src\main.rs:555:39 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

Fetching metadata for title 50010000012742 (798 out of 2414) thread 'main' panicked at 'called Result::unwrap() on an Err value: UnexpectedEof', src\main.rs:555:39 note: run with RUST_BACKTRACE=1 environment variable to display a backtrace error: process didn't exit successfully: target\debug\saveShop.exe --platform 3ds --regions JP fetch-all --cert ./ctr-common-1.pem (exit code: 101)

[neobrain]

It seems that the server is returning incomplete files to you. I'm not sure why that would happen. Your best bet is to keep trying until one of the runs will hopefully complete fine. (If you keep retrying, use fetch-metadata instead of fetch-all first. If you succeed, create a backup of everything and continue with fetch-media to download the remaining data. Interrupted fetch-media runs can more easily be continued.)

If you know some programming, you could also hack the source code to skip metadata files you already downloaded, so you can piece together partial runs manually. I won't be able to walk you through that process though.

[neobrain]

Oh, and if you have a VPN, it might be worthwhile running the scraping using that one.

[NVriezen]

I'm starting to think Nintendo servers are trying to work against us. I'm currently helping someone else privately with saveShop for 3DS. That person also got an UnexpectedEof error, while it worked for me yesterday. (Same commands and everything)

Could it be that they try to prevent us scraping the servers?

[neobrain]

Could it be that they try to prevent us scraping the servers?

Seems unlikely. If they wanted to block saveShop, it would be more hard-cut than a sporadic metadata download failure.

It's probably something silly that's worth a look into. Maybe y'all are running Windows and some of the networking code works different on there? Hard to say. My best guess is for whatever reason some requests return partial data, and fixing that would be as simple as just retrying the download. But somebody else would have to write the patch for that, since I'm not in a position to make time for it on short notice.

[NVriezen]

Could it be that they try to prevent us scraping the servers?

Seems unlikely. If they wanted to block saveShop, it would be more hard-cut than a sporadic metadata download failure.

It's probably something silly that's worth a look into. Maybe y'all are running Windows and some of the networking code works different on there? Hard to say. My best guess is for whatever reason some requests return partial data, and fixing that would be as simple as just retrying the download. But somebody else would have to write the patch for that, since I'm not in a position to make time for it on short notice.

Cool, I will give it a try then. Thanks for the quick replies :)

[puyo2a]

Thanks for the very quick reply.

For now, I am fetching media with incomplete metadata. Unfortunately, I can't to solve this problem because I don't have any knowledge about the Rust program.

US region didn't have the error until the very end, so I was able to start saving the majority of the files. JP region is giving me errors early, so I have trying many times.

Thanks anyway.

[NVriezen]

Could it be that they try to prevent us scraping the servers?

Seems unlikely. If they wanted to block saveShop, it would be more hard-cut than a sporadic metadata download failure. It's probably something silly that's worth a look into. Maybe y'all are running Windows and some of the networking code works different on there? Hard to say. My best guess is for whatever reason some requests return partial data, and fixing that would be as simple as just retrying the download. But somebody else would have to write the patch for that, since I'm not in a position to make time for it on short notice.

Cool, I will give it a try then. Thanks for the quick replies :)

I really don't get Rust unfortunately. I couldn't get my ideas to compile, I'm sorry. Hopefully all info we will need is still available after tomorrow...

[neobrain]

Thanks and well done for trying. Rust takes some time to get used to, hopefully this attempt will make the next one a bit easier :)

[neobrain]

I'm suspecting this is fixed with 64781b94ba828ffd4899efd78c8b7b80ee4fb206.

[puyo2a]

Thank you so much. It is working fine.

It must have been very difficult to fix it so close to the end of service. Thank you very much for your help!

[neobrain]

Glad it's working, thanks for testing!