Missing movie error

[NVriezen]

Since the last revision (d778af1f530a2fa32920fbef5b73d6ce9016388b) I get an error for the 3ds video fetching. Trying this with region NL and it stalls at title 50010000014012. The full error:

 Movie samurai/NL\nl\movie\50010000014012 (1362 out of 3442)
thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: Custom("missing field `movie`")', src\main.rs:1202:51
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: process didn't exit successfully: `target\debug\saveShop.exe --platform 3ds fetch-media --fetch-all-videos --regions NL` (exit code: 101)

I tried fixing it myself by inserting an if statement to check the second to last unwrap. However that still gives the error. I haven't programmed in Rust before, so I didn't expect much from it. It may be a simple thing, but I didn't have this error previously. Just wanted to scrape the remaining images and other data after some improvements had been made.

I did refetch the metadata before trying to fetch the videos.

[neobrain]

Something weird is going on here: The eShop knows two content types, titles and movies. Content 50010000014012 is a title, but somehow its metadata was placed in the movie directory for you. This is the cause of the error: It's trying to parse a non-movie metadata file as a movie.

Could you look for the id 50010000014012 in http_log and post any entries that number is mentioned in here? (I.e. the blocks between the ------- separators)

Did you execute any previous commands of saveShop in the same directory? Notably, a scrape from a different platform would mess up things. Scrapes of different platforms must be stored in separate directories, otherwise saveShop's followup operations won't execute properly.

[NVriezen]

The info from the log:

{
  "url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/NL/title/50010000014012?shop_id=1&lang=nl",
  "response_headers": {
    "server": "nginx",
    "date": "Sat, 11 Mar 2023 23:41:44 GMT",
    "content-type": "application/xml",
    "content-length": "8893",
    "connection": "keep-alive",
    "access-control-allow-origin": "*",
    "cache-control": "max-age=120",
    "vary": "Accept-Encoding"
  }
}

I only ran it previously for 3ds. So maybe the old data has screwed things up?

[neobrain]

So maybe the old data has screwed things up?

Maybe, though I don't see how either the current or the previous version would put a metadata file for a title into the movie directory.

How exactly did you "refetch the metadata"? Did you delete any previously fetched/created files at all? If not, does the timestamp of samurai/NL\nl\movie\50010000014012 indicate it was created in the previous run or in the new one? Do you know which saveShop version you were running before?

[piegit]

I'm getting the same error Fetching metadata for movie 50040000046203 (3275 out of 3445) thread 'main' panicked at 'called `Result::unwrap()` on an `Err` value: UnexpectedEof', src/main.rs:555:39 note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace here's what's in http_log

{ "url": "https://samurai.ctr.shop.nintendo.net/samurai/ws/DK/movie/50040000046203?shop_id=1&lang=en", "response_headers": { "server": "nginx", "date": "Fri, 24 Mar 2023 20:32:47 GMT", "content-length": "0", "connection": "keep-alive" } }

My version of saveshop is commit d778af1f530a2fa32920fbef5b73d6ce9016388b

[neobrain]

@piegit That's a different issue. Sounds like there was some error while downloading the metadata for this specific movie. As a workaround, download https://samurai.ctr.shop.nintendo.net/samurai/ws/DK/movie/50040000046203?shop_id=1&lang=en manually and replace samurai/DK/en/50040000046203 with it. If you have more entries in http_log with content-length=0, it would be best to start from scratch with a fresh output directory.

[NVriezen]

So maybe the old data has screwed things up?

Maybe, though I don't see how either the current or the previous version would put a metadata file for a title into the movie directory.

How exactly did you "refetch the metadata"? Did you delete any previously fetched/created files at all? If not, does the timestamp of samurai/NL\nl\movie\50010000014012 indicate it was created in the previous run or in the new one? Do you know which saveShop version you were running before?

Actually, there is no 50010000014012 movie file... The rest is all from the new run it seems. I'm going to throw away the metadata and just re-fetch it again. I didn't throw away the old metadata last time

[neobrain]

Actually, there is no 50010000014012 movie file... The rest is all from the new run it seems. I'm going to throw away the metadata and just re-fetch it again. I didn't throw away the old metadata last time

Very curious. Hope it works out with the new run!

[neobrain]

For the record, for a new run it's safe to keep the kanzashi/kanzashi-movie directories, but it would be best to also delete http_log. saveShop will then need to repopulate its cache, which will slow things down a bit, but it will still be faster than a complete fresh start.

[NVriezen]

Deleted the ninja folder and the http_log but still the error occurred... I don't know what old version I was using unfortunately. Couldn't it be that there simply is no movie for that title id?

[neobrain]

Did you also delete the samurai folder before re-running saveShop? That's the one containing the bulk of the metadata.

A clean run on that region (as outlined in my previous comment) should 100% work.

[NVriezen]

Did you also delete the samurai folder before re-running saveShop? That's the one containing the bulk of the metadata.

A clean run on that region (as outlined in my previous comment) should 100% work.

Seems to be working after deleting everything except the kanzashi folders. I was afraid that it would take ages to do a new run if I deleted the samurai folder. Luckily it didn't and everything seems to work fine again. Sorry for the confusion. Thanks again for the help and the tool. Amazing preservation effort! :)