search

elishacloud / Silent-Hill-2-Enhancements

A project designed to enhance Silent Hill 2 (SH2) graphics and audio for the PC. It also includes scripts to build or modify SH2 audio files (SFX, BGM and Dialog).
http://www.enhanced.townofsilenthill.com/SH2/
zlib License
567 stars 41 forks source link

[Silent Hill 2] Checklist of Remaining/Wishlist Fixes #13

Closed elishacloud closed 4 years ago

elishacloud commented 6 years ago

From @Polymega on October 17, 2017 23:59

(Note: This thread was made a few days before the creation of the SH2 Enhancements Repository)

I'm doing a play through of Silent Hill 2 again for the Enhanced Audio project and I was making mental notes of the remaining "wishlist" fixes and wanted to post them for posterity purposes.

Noted areas for improvement:

Discussion thread

Fixed!:

Copied from original issue: elishacloud/dxwrapper#9

Polymega commented 4 years ago

I love being wrong and learning new things. I'm very amateur when it comes to videography and video editing, so I'd like to ask about interlacing in a little more detail, if you don't mind:

Therefore a ~30i video actually carries information of 60 unique frames per second... interlaced together.

The way I've always understood it, and from what I've read elsewhere, is that an interlaced frame has half the data of two different frames that makes up a single frame. Because there's only half data for two different frames on a full frame at any given frame, I thought it can't be possible to turn something like 30i into 60p?

From what I've read:

60i on the other hand, can’t just be dropped into a 24p timeline and slowed down because it takes two fields to create one frame, meaning you really only have 30 frames to start with, not 60.

A 60i video camera recording leads to effective 30 frames per second because 2 fields are required for one full image frame.

What makes this more confusing for me then is that you said the videos are actually ~30i. So, from my understanding, to turn interlaced into progressive you'd need to halve the previous frame rate. (So something like 60i = 30p, 30i = ... 15p?)

... but then you said 480i60 and 480i30 are different names for the same thing! What a mind twist!

Smoothly deinterlacing 30i produces a reconstruction of sorts of the original two "subframes" buried within, i.e. 60p output.

I think my confusion lies in here, with how it's possible to turn 30i into 60p with having each frame's data fully restored, and not using half the data from a previous frame to make up/carry over to the next frame? Seriously... what a mind-bender!

angularGraphics commented 4 years ago

If for each interlaced frame, you take its odd fields to form subframe1 (line 1, 3, 5, ...) and its even fields (line 2, 4, 6, ...) to form subframe2, you now have twice as many frames as what you started with. And then you let your PC or device do (hopefully) smart things to them. :)

Polymega commented 4 years ago

Asking for clarification; not to be argumentative or anything:

So for subframe1 that uses odd lines (1, 3, 5), where do you obtain the even lines to complete the frame? Because the even lines on that particular frame make up subframe2 now?

If you grab the even lines from the previous (or) next frame to complete the full frame image of subframe1, and grab the odd lines from the previous (or) next frame to complete the full frame image of subframe 2, you've duplicated a frame's worth of data because you had to use that data from a previous (or) next frame to complete the full, deinterlaced frame image, correct? And because you've duplicated that data, you'd drop that previous/next frame from the deinterlacing process, which ought to halve the framerate when deinterlacing?

Unless...

And then you let your PC or device do smart things to them. :)

Do you mean let software "fill in the blanks," so to speak, through something like interpolation?

If so, instead of cutting the frame rate in half to create true, full frames from the data, you preserve the frame rate and use half-frames with interpolation to fill in the blanks? And at that point the question would become which of those two scenarios is the better outcome?

Sorry for the dumb questions here. I'll probably admit defeat after this one if I'm still scratching my head over the whole process. :-p

mirh commented 4 years ago

https://en.wikipedia.org/wiki/Deinterlacing https://en.wikipedia.org/wiki/Motion_interpolation

Polymega commented 4 years ago

Most deinterlacing techniques fall under three broad groups:

  1. Field combination deinterlacing
  2. Field extension deinterlacing
  3. Motion compensation deinterlacing

I think I understand my own confusion now, if someone can confirm this for me:

I was always under the impression that interlaced footage actually contained a full frame, but was split in half over two frames. For example: Frame 3's odd lines and frame 4's even lines could restore a full, true, complete frame.

But that's not the case? A single interlaced frame contains half of two different frames, but the previous or proceeding frame is not the other (alternating lines) half of that frame, but instead an entirely new/different frame? Meaning, there are only half frames of all given frames, and combining the previous/next frame's alternating lines wouldn't create the true other half of that frame?

I hope that made sense. If that's the case, then that's where my confusion has always lied.

Polymega commented 4 years ago

Yes, I think that was my confusion the whole time. As an example, for 30i, it'd be something like:

1.  First field (1/60s) + second field (2/60s) = 1st interlaced frame
2.  First field (3/60s) + second field (4/60s) = 2nd interlaced frame
...
30. First field (59/60s) + second field (60/60s) = 30th interlaced frame

From the above, no two fields share the same time (X/60s), so you cannot take fields from one frame and alternating fields from another frame to create a "true," full frame as the times aren't the same.

So angularGraphics you would instead strip the alternating fields, make them their own frames (doubling the count), and fill in the blanks using a method like interpolation. I think I got it now.

My incorrect, preconceived notions that the next frame's alternating fields held the same time stamp as the previous frame's fields for interlaced footage is what was throwing me off.

elishacloud commented 4 years ago

Should this list be updated? A lot of these have been outright fixed.

@Polymega, let me know which ones are fixed and I can mark them as fixed. I believe the Room 205 items have been fixed?

Also, I can close this one out if we are tracking these somewhere else.

Polymega commented 4 years ago

Also, I can close this one out if we are tracking these somewhere else.

We can close this one if need be. I've mitigated the list to an Excel sheet to make things easier, but honestly, I haven't been keeping that Excel sheet as up-to-date as needed, either. (Although I have been updating a personal text document tracking all the new features that are finished, which will be arriving in the next update.)

@angularGraphics If/when you'd like to discuss more about a "version 2.0" of your wonderful FMV pack, please feel free to open a dedicated ticket/post on the subject. I know we all have been waiting to try and make the game switch from 30 FPS (in-game) to 60 FPS (FMVs) correctly. We haven't made any strides on that yet, but it's still on the back-burner.

An ETA for it is undetermined at this time, but if you've been wanting to look back into this, perhaps the WIP files can be created for 60 FPS, but saved out as 30 FPS again in the meantime. Then, once we can get the game to switch to 60 for FMVs without issue, the 60 FPS versions can be readily swapped out? Only suggested if you've been itching to look back into it.

elishacloud commented 4 years ago

Thanks @Polymega. Closing this one. We can track the remaining issues separately.

For the 60 fps videos, we can open a new issue to track those. It would be great to support 60 fps for this game!

mirh commented 4 years ago

Putting aside I don't personally mind framerate.. I'd just like to share with you the state of art as of "FMV revamp" https://www.nexusmods.com/masseffect/mods/144?tab=description

Polymega commented 4 years ago

Putting aside I don't personally mind framerate.. I'd just like to share with you the state of art as of "FMV revamp" https://www.nexusmods.com/masseffect/mods/144?tab=description

That does remind me @angularGraphics: I have a copy of GigaPixel AI. So if you'd ever want to pass the video images through this at any point of an upscale process, I can do that and send them back to you.

Polymega commented 4 years ago

I'm playing around with this a little and I think the default 640 x 384 playback that's used on all versions is actually... stretched? Check these out when I change them to 600 x 384; it looks more correct (I'm using the demuxed PS2 PSS videos as a base here):

image image

^ Look at Maria's necklace pendant. It's more circular on 600 x 384.

image image

^ James' face is more long than wide at 600 x 384, which looks more natural/accurate.

EDIT: It should be AssumeTFF() (top field first), not BFF.

EDIT 2: Here's the final script I used:

    SetFilterMTMode ("QTGMC", 2)
    FFMPEGSource2("movie.m2v", atrack=-2)
    /*Levels(16, 1, 235, 0, 255, coring=false)*/
    AssumeTFF()
    QTGMC(preset="Slower", EdiThreads=1)
    Spline64Resize(600,384)
    Prefetch(2)

The line that's commented out would change the levels to 0-255.

Change EdiThreads and Prefetch based on your PC specs, going off this person's suggestions. For me, I did this conversion on my weak laptop, so I used low values.

Polymega commented 4 years ago

Posting these things for shits n' giggles. After using avisynth to extract a deinterlaced, lossless image sequence (folks are saying QTGMC is one of the best deinterlacers), I ran some of the frames from the intro cutscene through ESRGAN and Gigapixel, and Gigapixel produced much more desirable results:

toilet0152

It was upscaled to 1280 x 819 (from 600 x 384, which was originally 512 x 384) and then vertically cropped down to 720. This also hides the weird upscale line you see at the bottom right (at the sink tile).

image (Posting for posterity)

There may be even better adjustment settings within Gigapixel; right now I'm just fooling around with it. I've ran all the frames for the intro cutscene through this just to see what the full, end result would look like.

Cropped the image sequence in After Effects, re-saved out sequence, then used Bink's recommended settings for PS3 720 HD videos for the conversion/encoding.

image (Posting for posterity)

Those settings could also be changed to be better, of course.

Anyway, here's that movie at 59.94 FPS upscaled (didn't both with inserting the audio): https://drive.google.com/file/d/1kMEaXhIaCz7P5BSWeouJsCESJ96e2fIt/view?usp=sharing (You can play it with VLC Player)

Again: I need to stress I'm just playing around with this (mostly because I just built a new, faster PC and want to run it through its paces). Not trying to step on any toes or anything.

Polymega commented 4 years ago

After AI upscaling various FMVs I don't think it's ideal to use this approach/method for Silent Hill 2's videos. I think a big part of the problem is that the videos are natively a little too small (512x384) and the style of the FMVs doesn't lend well to AI upscaling.

What I mean by that is the FMVs are naturally soft-looking by artistic design, so the AI upscaler struggles on certain FMVs/scenes to make the images sharper. Whenever the AI upscaler does make a scene look sharper, things like a character's eye lid or mouth will become "shifty" as it plays out in motion... it's a rather unsettling effect, but not in a good way.

Here's some test files I did using Gigapixel AI: http://www.igotaletter.com/temp/sh2pc/FMV_AI_Upscale_Test.zip

Here are my notes on the test videos. I recommend viewing the test videos on an LCD monitor to see all the details I'm about to mention:

deai

Doesn't even look like it went through an AI upscaler at all. The video is still surprisingly soft-looking, considering it came from AI upscaling. Visually it doesn't look any different than what's currently provided in the FMV Enhancement Pack.

hei & hakaba

Just like deai, there are certain camera cuts where it doesn't even look like it was AI upscaled. But other camera cuts it looks decent. I think FMVs that take place outside during the day don't bode well to AI upscaling. Again: I think this is because all of the FMVs are naturally soft-looking by default/by design.

knife

Looks pretty good.

toilet

Looks good, but this is where the AI upscaler really starts to fall short. The AI upscaler cannot consistently produce uniform levels for dark backgrounds. Once James starts to bend over to look at himself in the mirror, look at the wall to the right of James. The black levels start to go crazy. This is because Gigapixel works on a per-image basis; it doesn't understand that all of these upscaled images are meant to be uniformally played out in video form.

gero & korosu_a

Looks... okay. But pay attention to black levels around things like the toilet bowl or the overall background. The levels of these areas just go crazy as the video plays out. Again: This is because the AI upscaler doesn't know to keep these levels consistent for video usage, as it only knows to upscale on a per-image basis.

korosu_b & saikai

These two are the worst. In korosu_b, whenever the camera is on James, there's a "halo" of faint light going all around James' body. This again is a result of the AI upscaler's process. It's very bad looking (and very noticeably on LCD displays). The same goes for saikai: The background behind James is all over the place with constantly shifting levels.

After doing these tests I don't think using an AI upscaler is the way to go for SH2's FMVs. I think the nature of how these FMVs were made and presented doesn't lend them well to an AI approach.

While AI upscaling can make some FMVs (or portions within FMVs) look really good, it does not make all of the FMVs look good. And, to keep the visual presentation consistent across the board, I think we shouldn't do the AI upscaler. I think the more simple resize method with some conservative filters applied to them is the lesser of two evils.

Topaz Labs has released a beta version of Gigapixel meant for videos: https://videoai.topazlabs.com/

There isn't a lot of reviews out for it yet but, from the ones I've read:

In the meantime, I took a deep dive into avisynth and made a version of the FMVs where they're scaled to horizontally fit a 16:9 aspect ratio, and vertically cropped. This removes the pillarboxing and doesn't stretch the FMVs. This method of scaling/cropping the FMVs is also used in the latest development builds of PCSX2, when playing the game using its Widescreen Patch feature, by the way.

I also partially remade the ending credits to make the text within them much more legible at higher resolutions.

The approach/methodology angularGraphic's used is what I took as well, with this revisit to the FMVs.

http://www.enhanced.townofsilenthill.com/SH2/files/SH2PC_Enhanced_FMV_Pack_1.5.0.zip

I made this version 1.5 so, if in the future whenever angularGraphics is ready, the 2.0 version number for the FMVs will be reserved for his work. I'll include this updated version of the FMVs with the next update release, but if you want, you can go ahead and download it now from the link right above.

angularGraphics commented 4 years ago

I haven't seen how they look yet, but very nice that you experimented!

I imagine the archive has Bink encoded files, right?

Unlike your above screenshot, I hope you used 64 allocated frames and 0 for audio compress level (max possible values).

If not, if you still have the source files, I would highly recommend re-encoding them with these settings. Bink is an old codec, and every bit helps.

Polymega commented 4 years ago

I did 0 for the audio compression, but I did not do 64 for the allocated frames (I used 12 based on Bink's recommendation for CGI videos). I have all the working files saved so I can redo them at 64 allocated frames. Give me a day or two and I'll update/replace the current download for it with the bumped allocated frames. 👍

Edit:

I imagine the archive has Bink encoded files, right?

Yes sir. The working files on my PC amount to many, many gigs worth of data, so only the download has the final Bink files. :)

Data rate: 2200000 (credits, end/ending, end_dog, end_wish, and water use 1200000 to keep their file sizes down) Keep peak data rate: Under a multiple of 3.0 Allocated frames: 12 64 Frame rate: 29.97 Audio compression: 0

Polymega commented 4 years ago

I updated the videos to have 64 allocated frames. Same link as before: http://www.enhanced.townofsilenthill.com/SH2/files/SH2PC_Enhanced_FMV_Pack_1.5.0.zip

angstsmurf commented 4 years ago

I wouldn't dismiss AI upscaling the videos just yet. The strength of ESRGAN and similar open source technology is that anyone (at least anyone with a very powerful GPU and lots and lots of free time) can train it to create custom models for exactly the kind of image they want to upscale.

In this case, the basic process would be something like a) find some HD video of very high quality in the same style as the Silent Hill 2 videos, b) take screenshots of this, shrink and compress them to give them the same size and kind of artefacts as the original SH2 videos, and c) let ESRGAN train on them (by comparing the compressed low-quality images to their high-quality counterparts) for a couple of days or more. Of course, each step has many pitfalls.

If anyone is interested, here is a good place to start: https://upscale.wiki

The field of image upscaling is obviously in heavy development, so the technology will likely be vastly more powerful and easy to use in a couple of years. Things like this look very promising.

Polymega commented 4 years ago

I should've worded my previous post better on the matter of AI upscaling. I personally haven't dismissed the method/process as a whole, but I did dismiss it through the use of Gigapixel (the AI upscaler tool I have at my disposal).

Aero_ and I tried ESRGAN as well, but, as you've mentioned, without properly training it to make a model to suit your specific needs, you won't get results you're happy with. I'd love to try and attempt to train such an algorithm, but I personally don't have the knowledge, nor can I currently devote any amount of serious time to learn more about it at this point, to do anything beneficial and worthwhile on this front.

The field of image upscaling is obviously in heavy development, so the technology will likely be vastly more powerful and easy to use in a couple of years. Things like this look very promising.

I think you said it best here. :) I'm excited to see how far the entire field of neural networking progresses as time goes on. It can only get better, easier, and faster to use for the common user which is nothing but a win, win, win.