Future development of Higurashi patches

enumag commented 7 years ago

cc @ItaloKnox, @DoctorDiablo, @Grelo, @jwgrlrrajn

Current problems

Installing a patch requires too many steps.
More complications if I want both PS3 and MG sprites.
Graphics take too much disk space.
Voice directories contain unused files.
Some voices or images may not work on Linux because of case sensitivity.

Features and goals

Long story short this tool already deals with all the issues above. However it works fine for my needs only and requires more work and some changes to our development porcess if we want to make things better for everyone.

Single repository using Git LFS

The thing is we can create a single repository for each Higurashi episode and have it contain all voices graphics and scripts directly. We're not doing that at the moment because Git has isseues with large repositories but we should be able to solve this with Git LFS. It would be more convenient for development in my opinion.

The details might get a bit long and needs discussion so I'll create a separate issue for this.

Graphics compression

Full HD graphics are awesome but the graphics patch for each episode is extremely large - about 2.5GB per episode with Watanagashi taking it to the extreme with 3.6GB. The thing is that all the PNG files are unnecesarily large because they're not using the highest compression level PNG supports. This compiler cuts them down to 1GB for Watanagashi and around 700MB for the others.

Voices filtering

You know how we need about 3 of the s## directories from the voices zip for each episode? Some episodes even require the same directories meaning that none episode needs all the files. To solve this the guide recommends symlinks to save space but that complicates the installation process even further.

This tool parses the scripts and deletes all the voice files that are not actually used for the episode. No need for any symlinks, the patch contains just what is needed for the episode.

Using the graphics compression and voices filtering I managed to cut down the Watanagashi patch (PS3 graphics + voices + MG patch) from 4.9GB down to 1.6GB.

MG Sprites

With the current patches the games only work if the "Updated Character Sprites" option is enabled. "Original Character Sprites" don't work because the game will crash in some cases when a sprite is missing. Also who would want to use the old ugly sprites anyway?

On the other hand there are the new sprites created by MangaGamer which some people might prefer over the PS3 ones (I do because MG Rika looks way more cute than PS3 Rika). Of course some character sprites are missing which means making the patches fully compatible with MG Sprites requires some work. I don't think we should commit ourselves to do this for every episode. On the other hand there are already awesome patches made by @jwgrlrrajn for Onikakushi, Watanagashi and Tatarigoroshi so why not include these as well? (Of course only if he decides to give us a permission to do so.)

The whole thing should work like this: All the PS3 graphics will go into the CG directory. If the player chooses the original sprites it will be the PS3 ones. If the player chooses updated sprites the game will use the MG sprites in CGAlt directory but if they don't exist the PS3 sprites will be used as a fallback for characters that don't have MG sprites or if any sprite like the zoom or weather variant is missing in CGAlt. We would not necessarily add all these sprites ourselves but we could easily include any sprites contributed by other people.

This tool already copies all the PS3 sprites into the CG directory and then copies the MG patch to the CGAlt directory.

Fixing Linux issues

While parsing the files this tool also updates the scripts so that all the files references match the file names exactly. The resulting patch should be fully compatible with Linux systems.

Summing up

The main goals here are to make the installation process easier by providing (ideally) just a single zip file with voices, scripts, PS3 sprited and (possibly) MG sprites included while making the patch as small as possible without any unused resources.

So what do you guys think? Feel free to disagree with any of my proposals and make your own suggestions.

Grelo commented 7 years ago

@enumag Sorry for the big size of the pictures im reducing it right now and will upload whole question arcs here later today. It looks like it will be cca 5x smaller same quality from the beginning of the proces. I could reduce it 20x with slightly reduced quality but i think 5x is enough and i like it more.

P.S. i didnt touch the unused ones so you can delete them later.

enumag commented 7 years ago

@Grelo Yep, that's about the same compression this tool is able to achieve. And I agree with you that we should keep the quality.

jwgrlrrajn commented 7 years ago

Cool. Sounds like a plan.

DoctorDiablo commented 7 years ago

Streamlining the installation process sounds awesome, and reducing disk usage is great too (as long as we're not sacrificing any quality, which it sounds like).

I'm a little hesitant about including the MG sprites though. I'm sure people would like that feature, but I wouldn't want to force someone to switch art styles 4 chapters in, so we'd be committing to doing all that extra work for each new release. If we do though, I think the PS3 sprites should go in CGAlt and MG in CG, which would make PS3 graphics the default option for new users.

jwgrlrrajn commented 7 years ago

Makes sense, although I think the reason for the PS3 sprites being in CG instead of CG Alt was so that characters with PS3 sprites but no MG sprites (e.g. Chie-sensei prior to episode 3, if MG hadn't given her sprites at all) would still show up. Not entirely sure how that works though.

jwgrlrrajn commented 7 years ago

I'm the one who's mostly been compiling the MG sprites as an extra patch to offer additional options to the PS3 sprites, so I don't mind if that responsibility stays with me. You guys can keep doing what you're doing for the PS3 end of things, and I can make sure everything like the zoomed-in sprites are properly represented on the MG side whenever possible.

ItaloKnox commented 7 years ago

Hey, man, that's rude. I like the old Ryukishi sprites. 😆

My proposal is to reduce the workload on the user's machine and instead use this compiler to prebuild the patch. If we take updates in account, the worst offender is the graphics patch since we can't update it directly without the git workaround. I mean, for free.

Voices and scripts are small, so we can keep updating it normally with new versions since we rarely miss a lot of files. So coming back to the proposal, maybe can't we use the compiler to slim down the patch and build the binaries from there? Github can host ~2gb binaries for free without any restrictions, if we make one zip for graphics and another one for voices, then it's easiest thing ever to set up an installer that downloads, installs and updates the patch. This way we can cut more than half of the installation time and updating the patch would be pretty much painless since the update list goes server-side.

I did a bunch of these before, but only for Windows. It seems there are a substantial number of Mac users around, so we would also need to take a look at that as well. But at least for Windows I think it's a solid solution and easy to implement.

Anyway, the clock is ticking so hopefully we can reach a solution that works for everyone before next friday. My best case scenario is to use the new installation in Meakashi and then update the previous chapters after that.

Grelo commented 7 years ago

For everyone who wants it, i just leave it here. Graphic mod with reduced size. Deleting the unused BGs should result in even smaller size.....if there are some,

https://mega.nz/#!FFcAxJpR!2E8htX7kDUaqgbh9XVvVObqAtjL5_u1DW6oXv8AN_aA

ItaloKnox commented 7 years ago

Nice, it seems to include all chapters. If we can get the voice files now (or a list of them), I think I can make something based on my suggestion. For now I'll be waiting for everyone's opinion on that.

Grelo commented 7 years ago

@ItaloKnox If you know how to do it and it would be easier than what we have now (its easy anyway at least for me) + if it would be automatic with less interaction than @enumag described in main page then definitely go for it. But im not good in these type of things so wait for other opinions.

enumag commented 7 years ago

@Grelo Great work with the images as always! Thank you! :-)

@ItaloKnox @Grelo Oh I never meant for this tool to be used by the end users - it's way too complicated for that. All along this was meant as a tool for us to discover and fix potential problems before making a release and to transfer all previous chapters to the new way. Sorry for the confusion.

An installer for windows is something I wanted to suggest anyway - it will be necessary if we use Git LFS anyway (I'll explain why in a separate thread). So yeah, definitely 👍 .

@ItaloKnox Should we make a Ryukishi sprites patch for you? 😆

@DoctorDiablo Having MG sprites in CG and PS3 sprites in CGAlt is unfortunately not possible because the CG directory works as a fallback if a file is not in CGAlt, but not the other way around. As I said I don't think we should commit ourselves to make the MG patches, I just don't see why not to include them if someone else makes them. We would need to add a note to the installation readme that it's highly recommended to change the settings to the "original sprites" (PS3) because the updated sprites may be incomplete or unavailable for future episodes. I don't think it's a problem to force them to the other sprites later - we can write it into known issues and if someone complains anyway we will just post a link to a page describing our policy that "we're not making MG patch but if you can contribute it we will happily include it". What do you guys think, @ItaloKnox and @Grelo ?

Grelo commented 7 years ago

@enumag Im personally using only PS3 sprites, if others are ok with it then do it. Im happy with how things are now but if it will be easier for others or demand is big enough for MG Sprites...thats the question.... you should make some survey and ask people xD

ItaloKnox commented 7 years ago

I like this, I think including the MG graphics would be nice, though I have two concerns about this:

Wouldn't it further increase the development time? Since the MG sprites needs to be updated manually with the weather filters, right?
Wouldn't it also increase the patch size?

Point 2 can be easily fixed since we can make it optional, I just have to set up two scripts (one with the usual CGAlt installation and another one using the new CG installation).

Point 1 however worries me a bit. Himatsubushi still doesn't have a MG patch, so if we are committing to include them, then it's partially our responsibility to include them. But, I think there's no issue in providing our usual patch as always and then updating the installer to include the option to install the MG patches as well after they're released. I think that's the best scenario for us.

If that's okay for everyone, then it's also ok for me. We can do that.

enumag commented 7 years ago

@ItaloKnox For Watanagashi (the largest patch) we're talking about 250MB which is not that much (most disk space are the PS3 backgrounds which are needed always). And as you said it can be optional in the installation process. I don't think there is an issue with the develoment time either if we do it the way you just said (which is what I've been suggesting from the start).

@jwgrlrrajn You're ok with us including your MG patches, right?

jwgrlrrajn commented 7 years ago

I'm working on Himatsubushi, I just haven't been able to do it as quickly as I'd like due to finals coming up.

jwgrlrrajn commented 7 years ago

@enumag I'm more than okay with that.

ItaloKnox commented 7 years ago

@enumag I'm thinking about starting it asap, can you get me a list of all the voice files needed for Onikakushi?

@jwgrlrrajn I didn't mean it in a bad way or anything like that, just take your time. Updating the installer is pretty much painless.

enumag commented 7 years ago

Here you go, captain. 😄

onikakushi.txt

ItaloKnox commented 7 years ago

I tried to work through the voices locally but it's such a pain the ass that I tried your compiler. I'm running it in a clean Ubuntu 16.04, with the dependencies installed.

There's an issue, though. When it finishes downloading the zips, I get this error:

[Symfony\Component\Debug\Exception\FatalThrowableError]
  Class 'ZipArchive' not found

I tried to run it again with sudo but the same thing happens. Any ideas?

Edit: oh, well, I think I know what's wrong. I don't have the game files in there, guess I'll need to install steamcmd for that.

Edit 2: I'm fucking blind. Windows was a prerequisite all along. 😆 Anyone around with good upload speeds to make the voice patch? Since I can't do it with my Linux server, there's not much I can do. My upload is around 100kb/s, so that's not much...

enumag commented 7 years ago

ZipArchive class should work after you run sudo apt-get install php7.1-zip. There might be some more extensions needed, php on linux is way too limited in default installation.

The windows prerequisite comes in later when cleaning up which files are used and which are not. However I just updated onikakushi to fix case sensitivity so if you update both the onikakushi patch and the compiler (I removed the exception thrown on linux later on) it should work.

EDIT: As for the steam files they're only needed to detect which sprites are missing but otherwise they're not necessary. I updated the compiler to throw a warning but not actually fail if the game files are missing. You will get false-positive messages about missing CG files in the end - don't worry though, it won't affect the voices.

I also removed the CG compression thing. You don't need it since @Grelo already uploaded his pack,

I recommend running the higurashi:make command with | tee output.txt so that you can give me the whole output it something goes wrong.

If it doesn't show any error check the output - there should be a message Deleted 18581 unused voice files. - if it is not there or if it has a different number it means I screwed up the linux compatibility.

If this doesn't work I'll write something simple for you just to clear the voices.

Unfortunately my upload speed sucks too and it's not about to get better any time soon. :-(

ItaloKnox commented 7 years ago

Man, that wasn't hard but it took a long time because I had to make some workarounds. I made a new EC2 instance just to compile the patch but it seems it didn't have enough HDD for everything. Dummy download files solved the problem, I guess.

After comparing to the list you sent me earlier, it seems to be fine. All 5118 files are there. I just finished setting up a batch script to do the whole job, so the only thing left is to test and try to implement it inside something. It's easy to have the installer run the script in a cmd window, but I still haven't found a way to output the cmd windows inside the installer. Would it be ideal to output it inside or letting the cmd window open would not be a problem?

enumag commented 7 years ago

Some people might be confused about a cmd window opening but I don't think it's serious enough to bother with it. I mean it's just a cosmetic problem, right?

ItaloKnox commented 7 years ago

Yep, that's all. I feel more safe about running an external script instead of letting the installer run the commands itself, this also allows me to pull the newest script without changing the installer itself.

I can also have the installer warn the user about the upcoming cmd window (the script also shows what step it's doing the entire time). Since we don't have much time until Meakashi, I'll try to at least have it working with minimum user interaction. After that I can try to make it pretty.

Update: took me a while but the installer seems to be working properly now. Of course it works on my machine, so if anyone feels compelled to test it, please give me some feedback!

Github doesn't support .exe attachments, so I zipped it. The installation should be pretty straightforward, just open the file, set the correct path and hit the install button. After that a CMD windows will pop up and download/install the files (you can click the "finish" button in the installer). I did an insane amount of testing and it finally installs without any issues here, so hopefully it'll work nice everywhere else (the script is super messy, tho).

Onikakushi PS3 Voice and Graphics Installer.zip

DoctorDiablo commented 7 years ago

Just tried the new installer. At first, my antivirus was giving me hell because it didn't recognize the executable, so I temporarily disabled antivirus and it started up fine. The installer launched the cmd window and it ran for a while and made it through all the downloads, but it started giving me an error about maximum path lengths once it tried installing the graphics. My installation directory is "D:\Program Files (x86)\SteamLibrary\steamapps\common\Higurashi When They Cry" which is a little long on top of all the nested folders being set up during the install, so I guess it can't handle it. I'll try installing it again somewhere else.

Does the installer spit out a log file somewhere that I can post for you?

Edit: After a brief fistfight with steam, I was able to get it installed at "D:\Steam\steamapps\common\Higurashi When They Cry" and I tried the installer again. It worked perfectly this time, I'm able to launch it and see the new graphics. It might help if we change the script to extract the files somewhere lower in the tree like \HigurashiEp01_Data rather than all the way in \HigurashiEp01_Data\StreamingAssets\temp

ItaloKnox commented 7 years ago

I need to enable logging first, so I'll start doing that by the next version. But I think I know what's wrong, I'll do some more testing and upload a new version as soon as possible.

ItaloKnox commented 7 years ago

Ok, I guess the bug is "fixed". Moving on Windows is pretty shitty so I had to make a workaround to copy and delete the source, but it seems to work. Did some testing using the same path as you do and it looks fine.

You don't have to redownload the installer, just run the file Onikakushi Patch Installer.exe that is inside the temp folder in your game directory. If you deleted it, no problem, just use the installer again.

By the way, about the antivirus issue I really don't know what I should do. Perhaps it's overreacting because it runs a batch, but I'll try to look into it.

ItaloKnox commented 7 years ago

Hey @enumag, I've been trying to get started with the remaining chapters but it seems to do the whole process excluding the most important part, the voices. It seems to move the voices but after it finishes the work, he cleans the entire SE folder.

Can I guess that the problem is the same Onikakushi did have? My script seems to be at a comfortable position right now so it would be nice if we could release the installers and update the wiki before Meakashi is released.

enumag commented 7 years ago

You're still running it on linux right? Then I need to fix the scripts of the remaining chapters. I'll do it right away.

ItaloKnox commented 7 years ago

Yep, unfortunately that's my only reliable choice. Thanks for doing that, if everything goes okay I'll have everything ready tonight.

enumag commented 7 years ago

Well you're doing most of the work again. This is the least I can do for you - no problem at all!

Wata and Tata are fixed. Hima will take a few more minutes because as I need to add the compilation script for it first and then fix any issues I find with that.

EDIT: Hima is fixed too. You'll need to pull a new version of the compiler as well.

ItaloKnox commented 7 years ago

Nah, this time I might be doing the whole thing but it wouldn't exist without everyone in this thread. If anything, I'm very grateful to be in a team that works so hard (and we still have a long way before finishing the novel).

But anyway, back to business, all the installers are properly compiled and working. There might be a bug or two I left behind, but it seems to be functional. I won't release them today because I need to take some time to prepare the release announcement and update the wiki, but tomorrow is the best day to do it since Meakashi is comming the day after.

For now let's enjoy the little time we have left before the madness begins again. After we're done with Meakashi, I hope we can come back to this to find a solution for Linux and OSX and try to make the installer a bit more complete.

enumag commented 7 years ago

@ItaloKnox I see the new releases and the Wiki update. Good job! Well actually that's an understatement.

EDIT: I removed the note about @jwgrlrrajn's patch from Himatsubushi release - it doesn't apply there yet.

ItaloKnox commented 7 years ago

Thanks, mate! I guess we're ready to tackle on Meakashi now. Also thanks for removing the Himatsubushi note, my finger was itching a lot to click the release button. 😆

Now let's hope the installer will work well for everyone, I tested every chapter on my machine and it seems to work well and the path bug was fixed, so there's nothing more I couldn't find to fix. Anyway, the work ends for today but tomorrow the real thing starts. Let's hope for a smooth development without tons of voice lines missing!

enumag commented 7 years ago

I'm going to read the Wiki and give it some finishing touches (like I removed nonexistent download links to Meakashi patches). I'll also test the installers tomorrow on my machine and let you know how it works.

Now I have an idea for Meakashi although I'm not sure if it's good or bad. As you know the issues with the automatic voice inserter are probably unfixable since the japanese text is different so I doubt I can achive much there. However I might be able to write a detection which lines should have voices (using the 「 and 」 characters in japanese text) and which should not. Then add empty placeholders like PlaySE(4, "", 128, 64); for voices everywhere where the script thinks there should be a voice line but could not find it. It might be a bit of a double-edged sword though because some voice lines have multiple text lines so some placeholders would be wrong. What do you think? Should I attemt to implement it? I'm honestly not sure if it would be good for you or not.

ItaloKnox commented 7 years ago

The Meakashi links were placeholders but I guess I forgot to comment them so they would not show.

And about your idea, I really like it. I think it takes more time to deal with lines without this solution than with lines using it. I'm totally in favor of that.

enumag commented 7 years ago

Alright then. I'll look into it tomorrow. (It's almost midnight for me now.) Hopefully it will make your work a bit easier this time. :-)

EDIT: I changed the headlines on https://github.com/07th-mod/guide/wiki/Part-1---Voice-and-Graphics-Patch. New operating system from Apple is no longer called OSX so I changed all references to Mac. Also it's now a bit more clear that the manual installation can be used on Windows as well. (Someone might want to use it if they don't trust our installers or have some issues with it.) Hope you're alright with the changes. If not I can revert them.

DoctorDiablo commented 7 years ago

Detecting lines that need manual correction sounds pretty helpful. I did something like that on Umineko by looking at the numbers in the filenames to identify gaps (blahblah17 followed by blahblah19, check if blahblah18 is missing). You solution sounds good too.

Grelo commented 7 years ago

I can confirm the installer works and everything is here, good job.

P.S. Are you sure that all voices in use are there? Is it possible that some got deleted through installer? Just asking not that anything is missing. And is installer downloading scripts from master file?

enumag commented 7 years ago

@Grelo Not 100% sure (there is always the possibility that I made a mistake somewhere in the script that deletes the unused voices) but about 97% sure?

Grelo commented 7 years ago

@enumag Ok thanks, that is good success percentage.

enumag commented 7 years ago

@ItaloKnox I implemented the placeholders thingy for the voice insertion script. I tried it on Himatsubushi and it inserts about 13 placeholders for every 100 inserted voice lines. For the whole Himatsubushi chapter it inserted over 2200 voice lines and 300 placeholers. That's a bit more placeholders than I expected to be honest so I'm worried about false positive placeholders. But then again those should be easier to deal with than no placeholers at all, right?

ItaloKnox commented 7 years ago

I think it should be ok, the number doesn't sound so far from what I fix manually. Actually it feels like I fix more lines than that, so it should fine. I usually have to double check the entire script, seems to be easy to catch false positives after I finish the whole script. Let's test it like that, if it doesn't work we can revert to the old method.

jwgrlrrajn commented 7 years ago

I've run into a bit of a snag getting a proper MG replacement for oisi2_8_0.png. If it was any one of his existing expressions, I could just swap the face out for the correct one, then resize, but since the expression doesn't match any existing ones, I've had to come up with my own.

This is mostly a placeholder until MG makes a proper version of this sprite, although that may never happen, so I need some feedback on this. Plus, if anyone wants to make their own version, should they be more talented at this kind of thing than I am, I'd appreciate it. oisi2_8_0 placeholder

enumag commented 7 years ago

@jwgrlrrajn Good enough replacement in my opinion. :-)

@ItaloKnox I tried the installers today. They all work fine. All the issues I found are very very minor.

Windows Defender thought they were dangerous and blocked them the first time I tried to launch them.
In the input where I select the installation directory I can't move the cursor to the end of the text when it is longer than the input.
The same input has weird looking red background in Oni and Tata installers.
The installers should probably delete the useless CGAlt/Alternate Chie-sensei sprites directory from the MG patch. Although maybe the better solution is to not have these directories at all in the MG patch and just use all available MG sprites by default (cc @jwgrlrrajn).

@DoctorDiablo Out of curiosity how did you manage to port Someutsushi to the PC engine? Did you do that manually or do you have some easy way to port a chapter (untranslated)? Awesome work with Someutsushi!

ItaloKnox commented 7 years ago

Thanks for the input, @enumag. I'll keep these in mind for Meakashi and update the others. Though I'm still not sure about fixing the antivirus issue, it seems to red flag it just because it has an .exe (that runs a batch).

@jwgrlrrajn If you hadn't told me it was made by you I would never know. Looks great. 👍

DoctorDiablo commented 7 years ago

@ItaloKnox For the antivirus issue, I think we can just add a note to the installation instructions that says that the exe might be flagged by antivirus and users can bypass that by saying they trust the file, or temporarily disabling their virus protection.

@enumag Porting Someutsushi was a fair amount of work. I have the PS3 script file so I was able to write some code that parses it and spits out all the text and voice commands in the proper format for the PC engine. But I had to insert all the sprites, backgrounds, and sound effects by hand because those are apparently not stored in the original script.

ItaloKnox commented 7 years ago

Yep, the wiki instructions actually note that this might happen. The ideal scenario would be to not need it all, and that's what I'm aiming for. Maybe not for now but for a better version of the patch.

I still have to check Someutsushi but I've also included it in our wiki. Hopefully I'll be able to play it after I finish Meakashi. If I can keep up with the current pace, the scripts should be finished in two more days.

jwgrlrrajn commented 7 years ago

I've updated the .zip for Watanagashi, and added a couple of different versions of the Himatsubushi patch. Since Rika and Mion are supposed to be younger in that one, I figured it would make sense to have one patch that's as pure MG as possible, then another patch that has the chibi versions of Rika and Mion instead of the MG counterparts. https://github.com/jwgrlrrajn/higurashi-steam-sprite-mods/releases/tag/1.1.0

I'm working on one that replaces the PS3 chibis' hair color with the shade from the MG sprites for consistency with their MG designs, but that's not a huge priority at the moment.

enumag commented 7 years ago

@ItaloKnox If you got my last comment (which I deleted) by email, please ignore it. That was my own mistake, no need to fix anything.

enumag commented 7 years ago

@DoctorDiablo Maybe we should post some note to https://www.reddit.com/r/visualnovels/ or somewhere that we're looking for translators to help with the console arcs? I mean as awesome as your work on Someutsushi is it is probably unknown to most people and I consider it unlikely for some translator to pick up on your note in the readme. What do you think?

07th-mod / higurashi-patch-compiler