Movie Support

[ghost]

Add the ability for the game to play movies and to be scripted accordingly.

• [x] Add a game state for when movies are playing
• [x] Implement movie audio/video rendering
• [x] Allow a movie to be skipped by some input like the escape key
• [x] Add scripting layer support
• [x] Document scripting API on the wiki

Issues:

• [x] (High) Start button doesn't work
• [x] High)Skip button(Esc) actually crashes the game
• [x] (Low) Some broken animation in startup (Higurashi Logo)
• [x] (Critical) StateMovie does not exit successfully
• [x] (Low) Volume level not consistent with BGM or whatever else makes sense

Describe DLL changes here in this thread.

[drojf]

Hey guys, I managed to get it working. Explanation/tidied up implementation to come later, but for now here is the .dll incase my computer dies:

Assembly-CSharp.zip

[enumag]

@drojf Wow, that's awesome news!

[irlPM]

Awesome work drojf! Thank you so much

[ghost]

You the man @drojf . The movie appears and that was awesome to see.

There is a NullPointerException in the logs between two steps of your setup, but the video surprisingly plays anyway. Probably good to check out though.

What remains with the movie state is when I try to leave the movie state early by hitting escape (for some reason escape takes 2 tries to work) it doesn't proceed with the rest of the script. I assume it's just not exiting the state properly. This likely has nothing to do with your changes since I've observed this before, but surprised it didn't seem to tell us what type it failed to load. But still open to any ideas on what could be wrong before I dive in. It could be that this also occurs when you let the movie run to completion.

This is one notable exception that I found in the logs; since it's occurring in the input handler, it's especially suspicious:

TypeLoadException: A type load exception has occurred.
  at Assets.Scripts.Core.State.StateMovie.InputHandler () [0x00000] in <filename unknown>:0 

  at Assets.Scripts.Core.GameSystem.Update () [0x00000] in <filename unknown>:0 

Other than that, MovieTexture does not yet work, which is required for linux support. An exception occurs on the setup and then subsequently on each update because the movie texture gets stored at the end of the setup block that threw that exception. This is the first time i'd even run that block of code so that's not surprising. Glad we got at least one method to work so far.

[ghost]

Also to-do: I hard-coded the volume level to 128, whereas the code it was ported from either put it through the BGM mixer or set it to the BGM volume. Low priority but would be good to have, especially since I often mixed Higurashi sound channels at low volumes because the released BGM was super compressed; having a movie play at full blast would probably be jarring.

Also I'm recording these as checklist items in the issue post.

[drojf]

I should mention that on that .dll that I had working, I added in a this.gameSystem.ExecuteActions(); just before the end of the public BurikoVariable OperationMODPlayMovie() function, because I thought that was required to get the background layer to show up when I called drawScene. I also modified the drawLayer function to create a mesh like this.mesh = MGHelper.CreateMesh(853, 480, this.alignment); when it normally doesn't - this was because the normal mesh was zoomed in for some reason. My changes should probably be applied to your old DLL/a fresh DLL to ensure I didn't screw something up.

Regarding linux - On the same day I did a version using Unity's inbuilt MovieTexture and got that to work (it taught me how to get the AvPro version to work), so that part should be fine (assume someone can test it actually works on linux). However if you're not aware, it can only play .ogv files. Does this mean that Da Capo has two copies of the movie in the game files? (possibly, the .ogv file is embedded in an archive and not visible, so you may not be able to see it).

I'll look into the Null pointer exception - right now i'm hijacking the bgLayer to playback the movie, but now that I know how it works it might not be necessary, and it may tidy everything up and fix the exception in the process.

Assembly-CSharp.zip (saving work 06-03-18)

Assembly-CSharp.zip (with linux/.ogv support 11-03-18)

[drojf]

Hi all,

Here's a list of instructions to convert the 'movie not working' version of the .dll to one where the movie playback works. I haven't tried to fix the other stuff (like going back to the script after the movie completes etc.). This version has both Windows and Linux playback, but on linux you must have an .ogv version of the video to play (with the same name). Also, I've left in my log statements as comments, but you can remove them if you want.

Instructions

1. In the 'Layer' (Assets.Scripts.Core.Scene.Layer) class, make the 'material' and 'meshRenderer' variables Public (right click -> edit field -> Set Access to Public) This breaks the logo - create functions 'GetMeshRenderer' and 'GetMaterial' instead. You'll have to use the rightclick->add method to do it since you can't compile that class directly.
2. Create a class which inherits from 'ApplyToMaterial' (RenderHeads.Media.AVProVideo.ApplyToMaterial) as follows:

   
   using System;

namespace RenderHeads.Media.AVProVideo { public class MODApplyToMaterial : ApplyToMaterial { public void OnEnable() { }

    public void OnDisable()
    {
    }
}

}

This is only to prevent the 'null pointer' error from occurring. You could also just directly edit the class.
You can also set the namespace such that the class is included in any namespace you would like, to keep things tidy. For example:

using System; using RenderHeads.Media.AVProVideo;

namespace Assets.Source.Scene.Objects { public class MODApplyToMaterial : ApplyToMaterial { public void OnEnable() { }

    public void OnDisable()
    {
    }
}

}

Will keep it together with the MovieEntity class.

3. Add the following function to the SceneController:

public Layer MODSetupBackgroundLayerForVideo() { Layer backgroundLayer = this.GetActiveScene().BackgroundLayer; backgroundLayer.ReleaseTextures(); backgroundLayer.DrawLayer("white", 0, 0, 0, null, 1f, false, 0, 0f, false); return backgroundLayer; }

This function prepares the background layer so that a movie can be drawn on it.

4. Recompile the DLL, so the changes are reflected

5. Replace the MovieEntity Class (Assets.Source.Scene.Objects.MovieEntity) with the following code (NOTE: you have to replace '.material' with 'GetMaterial()' and '.meshRenderer' with 'GetMeshRenderer()':
<details> 
<summary> **Click HERE Show Code** </summary>

using System; using System.IO; using Assets.Scripts.Core; using Assets.Scripts.Core.Scene; using RenderHeads.Media.AVProVideo; using UnityEngine; using UnityEngine.Events;

namespace Assets.Source.Scene.Objects { public class MovieEntity : MonoBehaviour { public void Update() { if (this.isAvPro) { this.AvProUpdate(); return; } if (this.www != null && this.www.isDone && !this.isStarted) { Logger.Log("Loading complete - Trying to use movietexture"); MovieTexture movie = this.www.movie; Logger.Log("Loading complete - Setting movie texture and audio clip"); this.LayerHandle.material.SetTexture("_Primary", movie); base.gameObject.AddComponent().clip = movie.audioClip; Logger.Log("Begin movieTexture video playback"); movie.Play(); base.GetComponent().Play(); this.isStarted = true; return; } }

    public void AvProUpdate()
    {
    }

    public void OnAvProVideoEvent(MediaPlayer mp, MediaPlayerEvent.EventType et, ErrorCode errorCode)
    {
        if (errorCode != ErrorCode.None)
        {
            Debug.Log("Encounted video error, stopping video playback.");
            this.Renderer.enabled = false;
            GameSystem.Instance.PopStateStack();
            return;
        }
        if (et - MediaPlayerEvent.EventType.ReadyToPlay <= 2)
        {
            this.Renderer.enabled = true;
            this.isStarted = true;
            return;
        }
        if (et != MediaPlayerEvent.EventType.FinishedPlaying)
        {
            return;
        }
        GameSystem.Instance.PopStateStack();
    }

    public void OnDestroy()
    {
        if (this.MediaPlayer != null)
        {
            this.MediaPlayer.Stop();
            UnityEngine.Object.Destroy(this.MediaPlayer.gameObject);
            return;
        }
        if (this.MovieTexture != null)
        {
            this.MovieTexture.Stop();
            Resources.UnloadAsset(this.MovieTexture);
        }
    }

    private static MovieEntity SetupForMovieTexture(string name)
    {
        Logger.Log("drojf - Get handle to background layer");
        Layer layerHandle = GameSystem.Instance.SceneController.MODSetupBackgroundLayerForVideo();
        Logger.Log("drojf - Create new gameObject with movieEntity component, and save a handle to the background layer");
        MovieEntity movieEntity = new GameObject().AddComponent<MovieEntity>();
        movieEntity.LayerHandle = layerHandle;
        Logger.Log("drojf - Begin Load video (rest of work done in 'update'");
        string text = "file:///" + Path.Combine(Application.streamingAssetsPath, "movies/" + name + ".ogv");
        Logger.Log("drojf - Trying to Load" + text);
        movieEntity.www = new WWW(text);
        movieEntity.isStarted = false;
        return movieEntity;
    }

    private static MovieEntity SetupForAvPro(string name)
    {
        Logger.Log("drojf - Try Draw Scene - old DrawSceneCustom3('bg_216', 0.5f);");
        Layer layer = GameSystem.Instance.SceneController.MODSetupBackgroundLayerForVideo();
        Logger.Log("drojf - Begin SetupForAvPro");
        Logger.Log("drojf - Create blank game object, add MovieEntity component (this class) and AvProMedia MediaPlayer componenent, and AudioSource");
        GameObject gameObject = new GameObject();
        MovieEntity movieEntity = gameObject.AddComponent<MovieEntity>();
        MediaPlayer mediaPlayer = gameObject.AddComponent<MediaPlayer>();
        gameObject.AddComponent<AudioSource>();
        Logger.Log("drojf - OpenVideoFromFile");
        mediaPlayer.OpenVideoFromFile(MediaPlayer.FileLocation.AbsolutePathOrURL, Path.Combine(Application.streamingAssetsPath, "movies/" + name + ".mp4"), true);
        Logger.Log("drojf - Add Listener");
        mediaPlayer.Events.AddListener(new UnityAction<MediaPlayer, MediaPlayerEvent.EventType, ErrorCode>(movieEntity.OnAvProVideoEvent));
        Logger.Log("drojf - ChangeMediaPlayer");
        gameObject.AddComponent<AudioOutput>().ChangeMediaPlayer(mediaPlayer);
        Logger.Log("drojf - Add AvPro ApplyToMaterial component to game object and set properties");
        MODApplyToMaterial modapplyToMaterial = gameObject.AddComponent<MODApplyToMaterial>();
        modapplyToMaterial._material = layer.material;
        modapplyToMaterial._texturePropertyName = "_Primary";
        modapplyToMaterial._media = mediaPlayer;
        Logger.Log("drojf - Using shader: " + modapplyToMaterial._material + " for Video Playback");
        Logger.Log("drojf - Save MediaPlayer and Renderer to MovieEntity object (this object)");
        movieEntity.MediaPlayer = mediaPlayer;
        movieEntity.Renderer = layer.meshRenderer;
        Logger.Log("drojf - Setup MediaPlayer for Video Playback");
        mediaPlayer.m_AutoOpen = true;
        mediaPlayer.m_AutoStart = true;
        mediaPlayer.m_Volume = 128f;
        movieEntity.isAvPro = true;
        return movieEntity;
    }

    public static MovieEntity CreateMovieEntity(string name)
    {
        if (Application.platform == RuntimePlatform.LinuxPlayer)
        {
            return MovieEntity.SetupForMovieTexture(name);
        }
        return MovieEntity.SetupForAvPro(name);
    }

    public MovieTexture MovieTexture;

    public MeshRenderer Renderer;

    public MediaPlayer MediaPlayer;

    public bool isStarted;

    public bool isAvPro;

    public Layer LayerHandle;

    public WWW www;
}

}

</details>
<br>

6. Save the module again. Ensure that the movies are in the `StreamingAssets/Movies` folder, with .mp4 and .ogv (for linux) extensions.

Here are the base and modified .DLLs. 
[movie_dll_files.zip](https://github.com/07th-mod/higurashi-dlls/files/1801246/movie_dll.zip)

#### Explanation

In short, to play back a movie, you need an in-game texture to play back on. Once the texture is specified, both methods will rapidly update the texture with the contents of the movie. 

I don't really know why the Da Capo method of creating a quad and using its texture doesn't work (probably due to how they setup the game engine in this game, and how they use a custom shader to render things). One reason it doesn't work is that most unity games have a 'mainTexture' which the movie is played back on, but I think this game uses a custom shader which has a '_Primary' and '_Secondary' texture. If you use the Da Capo method, I think it renders to 'mainTexture', but the shader doesn't ever use 'mainTexture', so you don't get any output. 

Since I knew the backgroundLayer (normally used for showing backgrounds in-game) texture was already visible, I decided to hijack that texture to play back the movie.

Anyway, here's what I did:

1. Make the draw the background (as if you were just using the normal 'draw background command' from script), with the width/height set appropriately such that the movie would play fullscreen. The background is a Layer object called backgroundLayer. 
2. Make the game return the background's material so it can be used to play back the video. 

Parts 1 and 2 are carried out by the `MODSetupBackgroundLayerForVideo()` function.

3. Make the video play back on the '_Primary' texture of the backgroundLayer. This is done differently depending if using the native video playback or avProVideo playback.
4. Play the movie.

For the Linux/native unity movieTexture method, I've basically followed the normal Unity tutorial, the only different part being that I use the '_Primary' texture instead of 'mainTexture'. If you want to understand how the playback is meant to work (for both versions), installing the old unity editor and doing the easy tutorial will help a lot.

For the avProVideo method, I used the ApplyToMaterial Class instead of the ApplyToMesh class because I couldn't figure out any other way to specify to use the '_Primary' texture is to be used.

idealpersona noted my initial release got a null pointer error in the ApplyToMesh class. This is because in the Unity editor, you can specify some inputs to classes by dragging and dropping them, which then sets the variables before the game even runs (or something like that). I don't know how to set those variables from the code, so it gave a null pointer error when the class is instantiated. However, the class was written so that it still works, as long as you set those variables later on. To fix this, I created a class which inherits from ApplyToMesh, but makes the OnEnable and OnDisable functions do nothing to avoid the null pointer error. You could also just directly modify the ApplyToMesh class and delete the contents of the two functions.

Also, I'm not entirely sure I garbage collected everything properly - I don't know if you have to explicitly delete the 'WWW' objects which have been created during the video playback process.

[enumag]

Today we've been testing a new DLL that I got from @irlPM. Posting here the results to not forget them:

The game played the video and with correct placement of the ModPlayMovie("mv01") call (at the end of BrandLogo section) it also loaded the menu afterwards. There are 3 known issues we need to fix:

1. Need to add skip function to the video.
2. Start button on the scenario screen doesn't work.
3. Higurashi logo animation is broken (video).

Also @Petsnew told me the player was unable to play 10-bit video at the moment. Does anyone know why? Would it be possible to fix that in the future?

[Petsnew]

Unlikely. AvPro Video doesn't support 10bit, so there's nothing we can do. Well, unless we actually modify the plugin itself, but I don't think anyone is willing to take such commitment.

Also, if we are to go that far, I'd rather try to write our own plugin from scratch, using ffmpeg/libav or libmpv as a base. They can decode pretty much everything and are cross-platform.

[ItaloKnox]

Is 10bit really necessary? I can't see how much we have to gain from a simple opening video, 8bit should be enough with basically no difference.

[Petsnew]

I think it is, well at least for me. 8bit has noticeably more banding and so I've to add grain to try to mitigate it. It's doesn't look anywhere near as good as a banding-free 10bit encode. But again, it's pretty hard to implement, so I don't think we should worry about it. Maybe in the later stages of the patch, when there's nothing much to do.

[irlPM]

Yea, I'll be working at the few bugs we have at the moment, at first the skip button is the top priority for me to worry about.

[enumag]

Agreed, we shouldn't worry about 10-bit for now. We can try later on.

[drojf]

I've managed to fix the issue where the background layer gets corrupted after movie playback, i'll just note it down here so it's recorded.

To re-initialize the background layer (Assets.Scripts.Core.Scene.Layer) object, you just need to call its Initialize() function after the movie is finished. This function is basically the constructor, and is called in the Awake() function.

However, to call this function from the MovieEntity class, you need to make the following changes:

1. Make Initialize() in Assets.Scripts.Core.Scene.Layer public (otherwise can't call it externally)
2. We need to keep a reference to the background layer so we can call Initialize() on it. To do this, add the property public Layer SceneControllerLayerReference; to the MovieEntity class. Then in the static function SetupForAVPro(), save a reference by doing movieEntity.SceneControllerLayerReference = layer;
3. Finally, we need to actually call Initialize(). Insert this in the function OnAvProVideoEvent(), after the movie has been played, but before GameSystem.Instance.PopStateStack();. This will make the last two lines of the function look as follows:

    this.SceneControllerLayerReference.Initialize(); //this resets the background layer
    GameSystem.Instance.PopStateStack();

---

Edit: it appears that the 'crash' when you try to destroy the video while it is playing, is fixed. The fix is similar to the above, in that you just have to call Initialize() on the layer at the right time:

In the MovieEntity class, in the OnDestroy() function, just after you call UnityEngine.Object.Destroy(this.MediaPlayer.gameObject); I have inserted this.SceneControllerLayerReference.Initialize();. Possibly this stops the engine trying to render a deleted texture or something similar, I'm not entirely sure how it fixes the bug.

[drojf]

So as far as I can tell, the issue with the logo being broken was due to me setting changing the access level of the Layer class's 'material' and 'renderMesh' variables to Public (they were Private). I do not understand how this is possible, but this seems to be the case. Attached is a DLL which I have re-created from the base .dll which has this fix applied.

However, you can use your existing DLL, if you do the following steps:

1. Add two functions in Assets.Scripts.Core.Scene:

>namespace Assets.Scripts.Core.Scene
+       public Material GetMaterial()
+       {
+           return this.material;
+       }
+
+       public MeshRenderer GetMeshRenderer()
+       {
+           return this.meshRenderer;
+       }

2. Change Assets.Source.Scene.Objects.MovieEntity to use the above two functions instead of the variables directly (change material to GetMaterial() etc.)

3. Change material and meshRenderer from public to private:

-       public MeshRenderer meshRenderer;
+       private MeshRenderer meshRenderer;
-       public Material material;
+       private Material material;

Here is my 're-created' DLL, with the logo fix: Assembly-CSharp_logo_fix.zip Note that this contains idealPersona's code which is hardcoded to play the video after the logo appears. I don't know if it was there before, but there is a transition effect which happens when the video starts playing... (like a diagonal fade in). I'm hoping that is a result of us hardcoding the video, and there being a fade in effect there at the time.

Here are the source code changes from original (as the patch is atm) -> broken logo version -> fixed logo version (my re-created version). Open with Git GUI or similar. Assembly-CSharp.zip

[drojf]

While I was trying to merge the AVProVideo code, I realized you can merge multiple CSharp files together (to avoid problems with inter-dependencies when importing into dnSpy). I made a script to do it in python, attached below, however it's simple enough that you can just make your own.

If you want to make a file which contains multiple class definitions in it, and put it in dnSpy, you just have to do the following:

1. Collect all the using statments of all the files you want to merge
2. Collect the bodies of all the files you want to merge (everything after the using statement)
3. Create the final merged file, which is all the using statements, followed by all the bodies.

When you import this merged file into dnspy, it will automatically put all the classes into the correct namespaces (because dnSpy doesn't know about 'files' anyway).

Python Script. Takes one argument - the folder containing the .cs files to merge https://gist.github.com/drojf/c630d4c82107b115e00173ca9ac529d0

edit: maybe you guys are/have already done this previously, but I thought it was worth mentioning

[ghost]

I'm in the middle of a major rewrite after getting a ton of help from drojf to get everything working. This DLL supports only Windows and Mac at the moment. Working on MovieTexture for Linux. Assembly-CSharp.zip

[ghost]

cc @irlPM Onikakushi DLL is available here. https://github.com/07th-mod/higurashi-dlls/raw/movie/Onikakushi/Assembly-CSharp.dll So far Petsnew has tested on Linux (MovieTexture implementation) successfully. If others test and it looks good we can do a prerelease and assuming no major issues come up we can start porting to the other arcs. It's on the movie branch. I probably won't be available at all tomorrow so @enumag or @ItaloKnox if you need the DLL to be on master for the release process to work feel free to merge it in. Many thanks to @drojf for providing the engine support and helping me whenever I got stuck.

[ghost]

I added a wiki page where we can start documenting the high-level design here. @drojf feel free to add or edit as needed since you know the code pretty well too.

[enumag]

@idealpersona I tried the latest dll and it seems to work alright. The audio is synced (on my PC at least), start button works, menu animation is fine, video can be skipped.

Just one minor thing, can we make the video skip-able with Enter to be consistent with other parts of the game?

[ghost]

@enumag updated accordingly - upon right or left ctrl key pressed or already held, or enter key pressed, the video will skip.

[enumag]

@idealpersona I was just checking something in Oni so I skipped the opening quickly and the audio kept playing while the game was loading menu. I was unable to reproduce it a second time though. If there is something simple you can do about this then do it, otherwise ignore this comment. It is a minor and rare issue as far as I can tell.

[ghost]

2 of the 3 people who tested said raise it a little more - at Italo's request I changed to 1.55x volume and plan to start porting to other arcs tomorrow.

[ghost]

Please don't kill me, but I just pushed a major refactor to the Onikakushi DLL. Plus, I polished it up so that it doesn't have that gray screen flash before playing the video when using AVPro. If anyone can test and give feedback I'd appreciate it.

[ghost]

@enumag @ItaloKnox @irlPM All arcs should be ported now; let me know of any issues.

[ghost]

Here are the steps i used to port

Add ModMaterial and ModMeshRenderer to Layer
Make Layer.Initialize() public
Merge AVProVideo library DLL
Copy Paste in this order
    MovieInfo
    MODApplyToMaterial
    IMovieRenderer
    AVProMovieRenderer
    TextureMovieRenderer
Add Movie value to GameState
Add StateMovie
Add ModPlayMovie to BurikoOperations
Add param lookup entry in OperationHandler.FillParamValues for ModPlayMovie
    OperationHandler.paramLookup.Add("ModPlayMovie", new OpType(BurikoOperations.ModPlayMovie, "s"));
Add BurikoScriptFile.OperationMODPlayMovie
Add switch entry for OperationMODPlayMovie to BurikoScriptFile.ExecuteOperation
            case BurikoOperations.ModPlayMovie:
                return this.OperationMODPlayMovie();

[DoctorDiablo]

Edit: List removed. See the updated list in my comment below.

Here's the list of all the OPs and where they go. We may want to move Side Effect, Tsuisou no Despair, and Place of Period somewhere else, but all the others have clearly established positions.

We also need to find high quality versions of Tsuisou no Despair, and Place of Period because they're not in the Sui archive. At worst, we can rip the YouTube versions.

Edit: Naegi_Makoto suggested this setup in discord, and I like it: In my personal opinion id place Side Effect in Meakashi, Place of Period in Himatsubushi and Tsuisu no Despair in Watanagashi and Tatarigoroshi.

[enumag]

Note for myself: There is a typo in the rtf, Complex Image is Matsuri OP2, not Matsuri OP1.

[enumag]

Also there are two more openings missing in your rtf which we should place.

https://www.youtube.com/watch?v=JlTPRiTxfJM&list=PLxB9p8WAdj0L5LT_UYU_fTmtGmqj8NdVW&index=9 https://www.youtube.com/watch?v=PIRxQQu-7K4&list=PLxB9p8WAdj0L5LT_UYU_fTmtGmqj8NdVW&index=10

[DoctorDiablo]

HigurashiOPsUpdated.zip

Ok, here's an updated list with the corrections mentioned here and the other 2 OPs added. (those are new for the switch version)

[ghost]

I believe we received good feedback so far so I'll close this issue.

The topic of webm support for better quality and the associated Unity VideoPlayer component came up many times in Discord. Essentially, we don't have webm support with our current libraries and the VideoPlayer which seems to be superior at a glance is not supported by Higurashi's version of Unity even as of Tsumi. It appears MG has refused to upgrade because around the same version Unity dropped DX9 support. If later arcs get released with an upgraded version we may be able to use it but I would not count on it.