Replace cutscenes with the remastered ones

[vitor251093]

The cutscenes currently inside the game have a DS resolution. The best way to improve that would be by replacing them with their remastered counterparts.

I currently don't know how to do that, since I'm not even sure about the methods used to render the ingame cutscenes.

[sandwichwater]

i have a vague and rough idea of how to do this , im no expert in programming by any means but i do know quite a lot about the inner workings of this game

in short : the video codec hand used when developing the game is mobiclip , its logo can be seen partially on the splash screen , it was distributed in nintendos sdk for devs . the file is abbreviated as ".mods" (MObiclip for nintendo DS) in the "mv" folder of the game , ive messed around with various success in replacing the files directly but that requires hacking the game which would mean distributing a modified rom which is already a grey area , but also was done with nintendos dev tools which definitely cant be distributed since aquiring the liscense has you sign an nda saying you wont use it for rom hacks etc

anyways , now that ive got that summary out of the way , the best way i can think to do it would be to have the raw remastered videos stored in a folder , then call those videos to play over the ds cutscenes whenever the code for the .mods files is executed , would still have the decent ds audio quality but would have high quality video . im sure its not as easy as i made it sound but it shouldnt be impossible at all

[vitor251093]

I see. This is helpful to know indeed :)

But unfortunately, I've just researched to know more about Mobiclip, and it seems that their video codec only supports videos up to 240 pixels wide.

https://wiki.multimedia.cx/index.php/Mobiclip_Video_Codec

I don't know if that's really the case, but if it's, I'm afraid the Mobiclip video codec may not be enough. Although, if that wiki is wrong, then maybe we can replace the video using a technique similar to the texture replacement, once it's fully implemented.

[vitor251093]

an idea , would it be possible to have a folder full of hd cutscenes and have a function or shader that calls those from the folder to play the video over the emulator when a prerendered cutscene is playing ¿ it would be a sortof hacky way of doing replacement seeing as it would just be playing video while the original game plays underneathe , but it might work

While that makes sense conceptually, I don't think it's possible to load a file from a shader. The shader can only perform calculations with whatever is given to it. Otherwise, it can't even access functions from the application itself.

With that being said, if I can provide the correct HD cutscene frame to it, it should be capable of replacing the original cutscene with it, but that leads us to the major problem: how to detect which cutscene is playing?

It may be possible to discover that using the same technique I used to figure out the memory address of the in-game camera menu option, but I haven't tried that yet.

Although, maybe we should move this discussion to #9 , so we have one problem per issue.

Moving the discussion to the pertinent issue.

[sandwichwater]

With that being said, if I can provide the correct HD cutscene frame to it, it should be capable of replacing the original cutscene with it, but that leads us to the major problem: how to detect which cutscene is playing?

possibly could use the ID and offset addresses of the cutscenes ¿ for example the ID of the file "802.mods" (the opening cutscene) has an ID of 0x5CF and an offset of 0x8985400 , if those can be detected for each cutscene then a system of "if this matches these addresses then overlay cutscene1.mp4" (or something similar) may work ¿

[vitor251093]

That's correct. If we manage to figure out when a cutscene starts playing, we can play the video as soon as the cutscene starts.

And, if we keep a video player preloaded at all times, it should load the video fast enough to keep them in sync.

We would also need to stop the cutscene if the user skips it, possibly by simply stopping the reproduction of the cutscene if the ID is no longer being played.

The next step would be evaluating if we can (and if we should) replace cutscenes that originally were scripted dialogues, but which received HD cutscenes. But that will be sorted out by a different ticket, in the future.

Anyway, in order to detect when a video starts playing: we need to figure in which memory address the game stores the video ID (or the video offset) so we can figure out which video is playing, and when exactly that happens.

We can name the cutscene files with the ID/offset (whatever we find in the memory) so we don't need to map each one of them manually.

[vitor251093]

for example the ID of the file "802.mods" (the opening cutscene) has an ID of 0x5CF and an offset of 0x8985400

Unfortunately, I couldn't find any of those while the opening cutscene was playing. At least during the execution of the cutscene. It's still possible that those may appear only when the cutscene is loaded into the memory though

[sandwichwater]

hmm yea thats where theyre stored in the memory , i didnt think about the ones while the scene is playing , i wonder if that would be easier

[ZoopOTheGoop]

i looked into the Days cutscenes in the Epic Games release, and while .mp4 files, they appear to have some sort of weird codec or missing header/metadata that makes them unplayable outside the game. Even ffprobe, which is usually extremely reliable, fails to load them. This is probably an incredibly easy fix to inject the proper metadata (scraping it from the executable maybe?), but that seems to be an added difficulty.

Edit: the moov atom isn't found, so they may need to be untruncated (e.g. https://github.com/ponchio/untrunc). My guess is the executable has it hardcoded how long the files are etc, so it doesn't need to have the metadata to play them.

[CpazR]

Since the emulator absolutely cannot supply these assets itself. We would need some sort of tool to grab these files from the pc port install (and modify them as needed) and bundle them in such a way that the emulator can use them.

At this point, it seems more likely that collaboration with OpenKH to help build this would be more phesable. I'd recommend hitting up the discord to see if anyone has done anything similar.

[vitor251093]

At this point, it seems more likely that collaboration with OpenKH to help build this would be more phesable. I'd recommend hitting up the discord to see if anyone has done anything similar.

That does make sense. Although, I think that, before doing that, we need a reliable way to detect which cutscene is playing in-game. Otherwise, we may reach the capability of playing the remastered cutscenes without a clue about when to play them.

[mirh]

udta   }meta       Oilst   G©too   ?data   
    TMPGEnc MPEG Smart Renderer 4 Version 4.0.15.50   "hdlr        mdirappl             Àfree 

This is how the files end, it's not encrypted or stuff. It's just.. somehow the mp4 "parts" order is reversed? https://www.exiftool.org/TagNames/QuickTime.html https://xhelmboyx.tripod.com/formats/mp4-layout.txt https://developer.apple.com/documentation/quicktime-file-format/chunk_offset_atom

Before that you have stuff like: tkhd, Sound Media Handler, mp4a And right before: moov, mvhd, tkhd, mdia, Video Media Handler, avc1

[mirh]

The collection just got released on steam. I don't have it (and neither does steamdb) but unless I missed some last minute change to wine/dxvk, they are probably shipping actually different video files altogether.

[sandwichwater]

although that is good to know , we still have the issue of not knowing how to get them to play ingame , obtaining the video files isnt really the problem , its getting them into the game . still , worth a look into the steam version though to see if theres any enhancements

[mirh]

Ok some gentleman must have gifted them the game, and they are definitively *at least* re-encoded. By the way, while I was at it I checked the ps4 files, and ironically enough they were plain mp4s (ftypisom, ftypmp42) all along. They play perfectly fine in VLC.

The days movies still pretty much end up with the same string and structures that I posted above.. So I can eventually just guess the real problem with the original pc release was just that they removed the header from all the files.

[sandwichwater]

are there any differences in the days cutscenes at all ¿ if there are i might get the steam version and rip them , if not then we still have the epic files

[vitor251093]

are there any differences in the days cutscenes at all ¿ if there are i might get the steam version and rip them , if not then we still have the epic files

According to what I learned recently, they are the same (Steam and Epic), but there is a bug across both versions that makes the cutscenes unplayable. @mirh may have just figured out why that is, so we may be able to fix the cutscenes upon loading them, as long as we have the needed headers

[vitor251093]

@mirh is the header the same across all files? If so, would you mind sharing its hex?

[sandwichwater]

oh thats not a problem , i already have the epic files in a playable format

[vitor251093]

oh thats not a problem , i already have the epic files in a playable format

But is there any way for us to make them playable programmatically? So we don't necessarily need to provide the cutscenes (only if the player doesn't have the collection)

[sandwichwater]

ohh , hmm good point , i didnt think about that🤔 that would probably be ideal yea

[ZoopOTheGoop]

Haven't cracked how they're being loaded yet, but table of addresses for movies, all are under the mv directory of the FAT filesystem of the ROM. The memory location is based on the US ROM:

https://gist.github.com/ZoopOTheGoop/7ba7bba3d0a99fe0345f7d973e904b90

Later we can match them up with their HD counterparts. It should be noted that this excludes honestly probably 70% of the scenes that are in the HD version (there are almost no clocktower scenes in this list, for instance).

Also, the only known mobiclip decoders don't like these files and either strip the audio or garble it heavily, so most of the descriptions are based off my memory of well-known scenes or the few bits of audio I could make out, they're just for reference purposes.

Next step I'm going to use desmume to watch these addresses to try and figure out how to tell if they're being loaded. I'd recommend making the other scenes a separate ticket, as it will involve a much more involved process of learning how scene transitions in general are handled and manually and laboriously playing through the game mapping Scene ID -> Cutscene.

@sandwichwater where did you get the ID? I'll add those to the table if needed. (E: I did a base conversion - It's just the fs-wide file ID, so that table starts at 1487 decimal and ends at 1533 decimal).

Also, I think using the opening cutscene is a mistake for this detective work, I set a breakpoint in desmume on that part of the ROM and at least the first bit of it seems to be preloaded when the game boots. The most useful would be to have a savestate during a mission before one of the later ones to test with.

[ZoopOTheGoop]

Also, it seems like the static effect that plays over the flashbacks of Sora's memories are done in-engine overlayed over the playing video - they're not encoded in the video files themselves, so we may have to contend with that if we attempt a replacement.

[ZoopOTheGoop]

Found it! Address 02093A4C holds the address of the playing prerendered cutscene. When not in a cutscene, it tends to be set to EA000000. As I suspected, the opening cutscene is different, when a New Game is started, the value is zeroed out - it's done in a different way. I can find the address in the memory from boot at 020DA2D8, but that value never seems to change even within a mission so it's probably a red herring.

Caveats: this value will be set when the scene begins to play, but is not set to another value until control of Roxas resumes, so even in dialogue after the cutscene, it will still hold the address of the previous prerendered cutscene even after scene transitions.

Whenever you switch between scenes (e.g. skipping a cutscene) the value appears to flicker to at least one other value, that could be relevant (e.g. a scene ID to load), but I'm not sure - I'm also still in the tutorials so it could be related to the tutorial prompts that show up.

E: More testing - 02093A4C appears to be some sort of game state transition related address, here's a log of values I've gotten after maybe 20 minutes of playing

00000000 - Title Screen, New Game, and Opening Cutscenes Cutscene Address - Prerendered cutscene is playing (except opening) F1165900 - Pause pressed, pause menu not loaded 01DA01B0 - Pause menu loaded 90239191- After Skippable Text Cutscene, or when closing pause menu EA000000 - Otherwise

[ZoopOTheGoop]

In terms of replacing the played videos, there's quite a lot of moving parts and would involve significant scripting on top of the game. Roughly, we'd have to hijack Melon when it detects a change to this address, manually play the cutscene in the window, and then basically feed an input to skip the original cutscene on after ours is done.

The raw HD videos also don't have subtitles burned in AFAIK so we might also have to do that on our own (which is a minor concern since we have to extract, fix, and possibly transcode them from the PC releases anyway).

The primary issue is letting users skip the cutscene. Days doesn't allow pausing prerendered cutscenes, only skipping, so as a first pass it wouldn't be too hard to just have any button skip it like Days already does, but anything beyond that (and I predict this is a feature that would need to be in due to the huge QOL upgrade) - we start having to manually handle input in a way that's not immediately fed to the DS emulation during cutscene mode. I suspect that's not the worst problem since there are already buttons that make UI elements appear and disappear that don't exist in the original release, but I want to make that step explicit.

The good news, is that 90% of this should still work for the cutscenes that replace in-engine cutscenes. When we learn how to detect those (which I'm working on) it's the exact same principle, it's just a different method to skip the original. So the setup for replacing the prerendered ones is effectively transferrable.

The bad news is that i dont think the HD cutscenes have a great 1:1 correlation with the DS ones, from a quick glance there are multiple cutscenes in one that are single cutscenes in the other and vice versa - and this probably gets worse when you start taking in-engine ones into consideration. This isn't a deathknell for the idea, merely annoying. The real hard work is going to be pulling in a video playback library and wiring it up to bypass the DS emulation on a memory trigger. (Hopefully the work you've already done re: UI and stuff helps with this to some degree).

There are some corner cases that my vote is "don't worry about it because it makes everything way more complicated for now" on - for instance, what if someone loads a savestate in the middle of one of the netherzones where the memory still points to a prerendered cutscene but its after skippig it. There'll almost certainly be other issues that crop up (replaceable cutscenes that arent skippable or whatever else), but better to leave those to the future.

[mirh]

According to what I learned recently, they are the same (Steam and Epic)

Yes, I guess they are the same. If you use the default windows branch. But on the steam deck (depot 2552437) they ship different movies.

is the header the same across all files? If so, would you mind sharing its hex?

It's very slightly different (of course? I mean, I think that's normal). But it's not hard to find the CUSA05786 files, and then extract it with fake pkg tools.

The raw HD videos also don't have subtitles burned in AFAIK so we might also have to do that on our own (which is a minor concern since we have to extract, fix, and possibly transcode them from the PC releases anyway).

I mean, I reckon you'd already need (say) avisynth or vapoursynth to cut and merge the cutscenes to match the DS ones.

[vitor251093]

The primary issue is letting users skip the cutscene. Days doesn't allow pausing prerendered cutscenes, only skipping, so as a first pass it wouldn't be too hard to just have any button skip it like Days already does, but anything beyond that (and I predict this is a feature that would need to be in due to the huge QOL upgrade) - we start having to manually handle input in a way that's not immediately fed to the DS emulation during cutscene mode. I suspect that's not the worst problem since there are already buttons that make UI elements appear and disappear that don't exist in the original release, but I want to make that step explicit.

Yeah, that won't really be that complicated. A d as a long term enhancement, we can implement the ability of pausing during cutscenes, by making the pause button pause the DS emulation instead, while the cutscene plays on top of it, and a fake pause screen is rendered on top of the cutscene. It would be a long term enhancement though.

for instance, what if someone loads a savestate in the middle of one of the netherzones where the memory still points to a prerendered cutscene but its after skippig it

Since the original game doesn't have save/load states as a feature, I think we can make it functional, but without diving too much into it. Like, always reproducing the cutscene from start if the save state is loaded.

But on the steam deck (depot 2552437) they ship different movies.

How different are them?

[mirh]

Well, they are a fourth the size for starters. Idk then, but given it's supposedly the only deck-specific fix they needed I'd expect at least a somewhat different file format.

[mirh]

Just for the absolute records some of this fixed the game on linux (and if I had to take a guess, it was specifically something here).. what to say? Lots of stuff, including more support for transforms, conversions and unsupported formats themselves. Checking that might give some random insight into how the videos are coded, but you'd still eventually need some serious coding skill to come out with something usuable.

So, really.. just use the SteamDeck=1 variable and copy the videos (who knows if their quality is better or worse than the ps4 ones?) hoping that square won't decide to take down the branch in the future.

[ZLGNF]

@ZoopOTheGoop Is it possible to detect the Recoded dialogue scenes?

[ZoopOTheGoop]

probably, but im a bit slammed and can't check right now. i outlined my method above if you want to give it a shot

On Wed, Jul 10, 2024, 09:00 ZLGNF @.***> wrote:

@ZoopOTheGoop https://github.com/ZoopOTheGoop Is it possible to detect the Recoded dialogue scenes?

— Reply to this email directly, view it on GitHub https://github.com/vitor251093/KHMelonMix/issues/9#issuecomment-2220910228, or unsubscribe https://github.com/notifications/unsubscribe-auth/AWIX57GRG2MTI6RCPU6B3WTZLVLA7AVCNFSM6AAAAAAZFZA2ZKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMRQHEYTAMRSHA . You are receiving this because you were mentioned.Message ID: @.***>

[ZLGNF]

@ZoopOTheGoop

I'm not sure if I'm doing this correctly at all but using the RAM Search on desmume it looks like these are the addresses that change when you are in and out of these scenes

[rarsneezes]

Clarification question: is this about replacing ALL cutscenes to be their HD Movie counterparts (including the textbox cutscenes), or just about replacing the like 12 already prerendered fully 3D/voiced cutscenes?

I'm only asking because I prefer the textbox cutscenes over their movie counterparts, personally. Sorry if this has been asked before! Can't find it, and I felt like opening a whole new issue about it was silly.

[vitor251093]

Clarification question: is this about replacing ALL cutscenes to be their HD Movie counterparts (including the textbox cutscenes), or just about replacing the like 12 already prerendered fully 3D/voiced cutscenes?

I'm only asking because I prefer the textbox cutscenes over their movie counterparts, personally. Sorry if this has been asked before! Can't find it, and I felt like opening a whole new issue about it was silly.

Replacing the DS cutscenes with HD counterparts.

In the future, if possible, we want to eventually replace the textbox ones as well, but either way, the code is being structured in a way that cutscenes will only be replaced if the HD replacement is available, so, if that ever happens, just keep the cutscenes whose name start with "hd8" in the assets/days/cutscenes folder, and delete the other ones.