Current status and the future

[enumag]

cc @ItaloKnox @DoctorDiablo @Grelo @jwgrlrrajn @irlPM @InochiPM @Nozoki17

So as you might have noticed, P-Chang left us. He left suddenly without giving us guidelines how to so the things he did for us until now while also thoroughly deleting all resources he made in the past. I'm creating this issue to summarize what he was doing and the consequences of him leaving us.

First of all with the efforts of some of our members, things are already moving very fast now. Keep in mind that some of the problems I'm posting here might already be solved by the time I finish writing this or a few hours later.

Current status

DLL hacking

So the main thing P-Chang was doing for us was DLL hacking. Over time he made many changes in the DLL which I'll try to summarize here.

1. Global flags GADVMode and GLinemodeSp for NVL/ADV mode. This fixed several ADV-mode issues we had in our old releases.
2. A hotkey (F1) to switch between NVL/ADV mode.
3. Fixes to Line_* modifiers in the scripts in order to fix voice sync. This allowed the game to be playable in auto-mode without continuing while a voice was playing and also fixed other issues we had.
4. ModEnableNVLModeInADVMode() function to force a part of a game to be in NVL mode even for ADV mode users. Used in Tatarigoroshi epilogue.
5. Fixed save screen.
6. Fixed position of buttons.
7. Global flag GCensor for content censoring level along with hotkey.
8. Hotkeys for Quick Save / Quick Load.
9. Hotkeys to increase / decrease voice volume.
10. In game help of available flags and hotkeys.
11. Fixed the <i> tag so that we can use italics in the game without them looking awful.
12. Many other flags, hotkeys and changes. Most of them were just his ideas and we never actually used them. An incomplete list is available here.
13. And finally of course the LipSync which is available in out Onikakushi 5.0 pre-release.

So yeah that's a really big list of important features and bugfixes. And it's not even a complete list. I'm pretty sure I forgot something important here. All of these features are lost for future chapters unless we figure out how to decompile, edit and recompile the DLLs. Italo is already busy trying to achieve that, I'm sure he will post his findings soon. The latest news I have now is that he was able to decompile the DLL so we should be able to make diff between vanilla DLL and the last hacked DLL.

Note: I believe P-Chang was using this decomiler.

EDIT: And apparently he also used this tool.

LipSync

LipSync was not even implemented for other chapters than Onikakushi yet and in Onikakushi it was experimental. Also the LipSync involved spectrum files which made the LipSync truly awesome - it basically stopped the mouth movement when the voice paused for a moment. It works really good in the game. Sadly we have lost the spectrum files for PS3 voices. We still have spectrum files for PS2 voices available which should mostly be the same as PS3 but we're not yet sure what kind of effect this will have on the LipSync feature and we're missing the spectrum files for all PS3-specific voices. EDIT: PS3 spectrum files were saved thanks to @EliEron. We have no idea whatsoever how the spectrum files were made either - P-Chang wrote some analyzer for that but never shared it with us. Anyway this is all irrelevant unless we figure out how to decompile the LipSync feature from his last DLL and put it into the other chapters.

New UI

Another thing is that we don't know is how to unpack and repack the assets files that contain the UI images. I have no idea what he used to do this. Any takers?

EDIT: Apparently P-Chang was using this editor. thx @InochiPM

New graphics patches

Most of P-Chang's work on renaming the backgrounds and sprites for the new graphics patches has been lost as well. I don't remember which chapters he worked on but we'll find that out soon. Anyway this should be easy to remake even although a bit time consuming. Italo has some program to do it quickly though.

Advanced script changes

P-Chang was also able to do some advanced changed in the game scripts like moving backgrounds. These changes required deeper knowledge of the available functions. It should be doable with the decompiled DLL but still it won't be easy for us.

New policy

All the features we were working on are hereby postponed until we solve this. Feel free to post your ideas and WIP files here on GitHub but be careful to make a separate issue for everything. I promise we'll get back to it after we sort this out but we might easily skip anything that isn't in a separate issue.

Some of you might remember that I mentioned somewhere here on GitHub that I want us to document everything we're doing. It was exactly because I was paranoid enough to expect something like this to happen sooner or later. Well apparently I failed to execute this initiative in time - I prioritized LipSync and new graphics patches instead and this is the result. The plan was to document everything while developing Tsumihoroboshi patch. In a way P-Chang left us at the worst time possible - right before we could put this initiative into motion.

Of course this initiative to make the project future proof and independent on any individual (myself and Italo included) is now more important than ever. Obviously this has start with Italo and myself - we need to write down everything we know about making these patches - voice insertion, installers, all the scripts I have in this repo etc. These guides will be further improved while we work on Tsumihoroboshi (note that Tsumi patch might also be delayed too because of this depending on when it will be released).

This also means some changes to our policy with accepting ideas and changes. The proposal is to only accept changes that can be reproduced later without the original author. Anyone can still come up with and suggest anything they want of course but from now on I think we should only accept changes with all the PSD files, photoshop filters, fonts, scripts and any other resources used to achieve the changes. In more complicated cases a written guide how to use those resources could be needed as well. There should be no exceptions to the new policy, not even for Italo or myself.

Please give us your feedback and ideas about this proposed new policy and how strict we should be with the rules in your opinion. The goal is for anyone to be able to continue our work in case we're unwilling or unable to continue for whatever reason.

We also want to ask you to send us any files related to your previous contributions that you have on your harddrives and share your notes how to use them.

Future

Aside from documenting everything the top priority now is to recreate what P-Chang was doing until now, mainly all the important DLL hacks. The goal is to have decompiled source code with documentation about all the hacks as well as a written guide on hot to decompile and recompile the dlls. If you think you can help with this task or know someone who could teach us how to decompile and recompile the C# library, please let us know.

Another thing I want to make is that I think we should write down a full list of features our patch provides compared to the vanilla version (by now it will be quite long). This list should include credits to people who contributed or later improved the change and link to guide how to make the change.

Please give us your feedback on this and your support. We need all the motivation we can get from your support and all the help you can give us. Many thanks to all contributors, we're looking forward to working with all of you in the future. We believe that together we can overcome this and make our patches better than ever.

Thanks for reading.

[jwgrlrrajn]

Do we want to create a new repo for documentation? I, for one, think I really should get to work uploading my notes on how to reproduce the MG sprite changes.

[enumag]

ping @EliEron @yagamisempai You guys might want to read this too and give us your feedback

@jwgrlrrajn Not sure yet. Either we will use the wiki on this repo or create a new repo for it. You can post it anywhere for now, we'll put it in the right place when it's decided. Thank you!

[Nozoki17]

I'm used to replace the UI so maybe I'll be able to help with that.. As for the files I modified I should be able to provide documentation and psd files on how I did it.

[drojf]

Probably not your biggest problem right now, but I'm willing to re-write the voice spectrum generator thing to generate the spectrum files (assuming it's what I think it is. I think I was there for the initial discussion on Discord).

I understand you still need to extract the lipsync code from the .dll before you can even use the spectrum files at all, but I'm just putting my hand up now for when/if the time comes.

edit: Just re-read what you wrote - are the spectrum files already generated/saved for ALL voices for past and future games? then what I said isn't really needed...

[enumag]

@Nozoki17 Great. We'll need you or @InochiPM to write the guide how to extract and edit the assets. What PSDs we need depends on what version of the UI we will use in the end but it certainly wouldn't hurt if you can upload what you have.

@drojf Thanks for offering. We actually managed to salvage all of the spectrum files P-Chang previously uploaded. Yes it should contain spectrum files for all PS2 and PS3 chapters. According to what @ItaloKnox and @DoctorDiablo posted on Discord it seems we should be able to use them. So it is not really necessary to write a new generator now. In my opinion we should make new releases with what we have available and then have you look at it. There is certainly some room for improvement in the spectrum files though so if you think you can write a better generator, it might be good in the future. For now it's not a high priority in my opinion.

[InochiPM]

@enumag I can upload all of the PSD files i have on hand for the Logos as well as the font list we already have on here with the fonts i uploaded. As for the UI Files,My UI patch needs some tweaks because of the transparency issue with the recompiler, i think i just need to change the mode lol, ill upload all of the PSD files for you all for those once i get them all completely done.

We also still need to decide on a font file for the final text images for the game i just remembered.

[EliEron]

@enumag I agree that being more stringent about documentation is an important and necessary future step. Also I feel that all important files should have multiple backups that are maintained by different people. To prevent one account deletion taking down a multitude of files like what happened this time.

Somebody should also have local copies of most of the files and documentation, this entire project could be taken down one day and losing all of the development files just from that would be terrible.

I haven't really contributed any features to the patch, so I sadly can't help with the documentation too much.

[InochiPM]

Also! @Nozoki17 Do you happen to know why some of the textures that are imported don't appear transparent but the others do. Perhaps its the way it was saved in photoshop but the Dialog_Delete variations dont take to kindly to transparency can you see if you can get them to work right?

I'm also about to upload all of my files to my few accounts so i always have backups if we need them.

[Nozoki17]

Weird, can you give me the psd files of the stuff that causes issues?

[InochiPM]

Archive.zip @Nozoki17 I put the two files inside, the file with the issue is dialog_areyousure and the ones that work fine are dialog_data.

[Grelo]

@enumag The other problem is graphic patch for Tsumihoroboshi chapter once it is released. I could do it like with previous chapters but the problem is with so many changes you did with P-Chang to the script i think i would only make your future polish on it difficult since my knowledge about this stuff is poor. So what now....

[enumag]

@Grelo Yeah. That's actually what I'd like to discuss with you on Discord. I need to know more about your approach when making the graphics patch.

[Nozoki17]

@InochiPM I fixed it, when you import a texture with alpha you need to change the texture format to "ARGB32" in the Asset bundle extractor

[Grelo]

@enumag But there is hardly anytihing to discuss because my steps are simple. Wait for ItaloKnox to make voice patch then ready my vita, open scripts in notepad++ and do the work. Both versions have almost identical sprite positions so when there is something wrong i just swap them etc with correct counterparts, same with backgrounds. In case of adding stuff i use basic commands already in the scripts DrawBustShot, DrawScene, etc..so it is not that hard. And in the end i have full 1:1 graphic patch. It's only time consuming. If that is Ok with you i can do it. I booted Dicsord but realized i don't have fckin microphone...

[DoctorDiablo]

For Tsumihoroboshi, I think we should plan to add the voices first and put together a voices-only release, then have @Grelo make his graphics changes on top of that like usual and package a 1.0 release that runs on the vanilla dll. Then we can make the changes necessary to get it working on the hacked dll on top of that as a separate 2.0 release later down the road.

[enumag]

@Grelo Actually we're using Discord as chat. I don't think we ever used it for audio calls. We're slightly off topic here with Tsumihoroboshi so I'll create a new issue about the graphics patch and ping you there.

@DoctorDiablo Yep that's exactly the plan. Actually I believe this is exactly what we did with Meakashi.

[Grelo]

@enumag What is the channel?

[enumag]

@Grelo #higurashi_dev

[ItaloKnox]

@Grelo We don't really use the voice chat since everyone's around the world and it would be quite difficult to communicate like in text. 😆

I think I'll set up the new repo for the dev wiki tomorrow. We need to scavenger a bit to detail the functions properly, so it should take some time. I'll take care of the writing and organization, so don't worry about these two. It would be really helpful if someone could compile a small list of the common functions and how to use them (PlayVoice, DrawBurst yadda yadda, that kind of thing). Doesn't need to be a full explanation of them, but the syntax and a summary will be enough, I can work through the rest.

[InochiPM]

@Nozoki17 Strange i did that and the background still ended up being white. Hmm ill have to try it again today!

[Grelo]

@enumag Doesn't work it wants some URL invite.

[DoctorDiablo]

I may have done it. Here's the latest Watanagashi dll + all the lipsync code changes. Instructions on what I did can be found here for now, but should probably be moved to the new dev guides repository later on.

Assembly-CSharp_WataLipsync.zip

Now we need to set up all the scripts and spectrum stuff so we can actually test it.

[Grelo]

Regarding Watanagashi and censor branch now with P-Chang gone is the uncensored version still in game because i think i was intended to be switchable and i would not like to have censored text as granted.

[jwgrlrrajn]

Okay, so I have an early draft of my sprite modifications written up. I've got a few batch actions, psd's and sprite bases I need to upload though. Where do we want to put things like that?

[ItaloKnox]

@jwgrlrrajn you can drop the zip here and I'll organize it in the wiki later.

@Grelo here: https://discord.gg/acSbBtD

[enumag]

@Grelo As far as I know there are censor branches in wata but uncensored is default.

[enumag]

I'm not sure if everyone who saw this is following Discord so just to summarize recent events.

Firstly in the end we have pretty much all the resources we need. Nothing irreplaceable has been lost. Secondly Italo, DoctorDiablo and djorf are making a good progress with DLL hacking. This should pretty much take care of mitigating any long-term damage.

We're still prioritizing finishing the LipSync over the documentation a little but I believe we can get started on documenting things this weekend. As far as DLL hacking goes DoctorDiablo already wrote down his notes on the topic.

Meanwhile I finished the compiler for new wata patch which Italo compiled here. LipSync should work but it's not yet tested. Missing CGAlt archive and PS3 poem are in progress and should follow soon.

Some of our members also have some new ideas about features we could add this year. Stay tuned!

Overall I'm very pleased about our progress. Everyone pretty much doubled their efforts and we're doing really well as a result. Thanks everybody, you're all awesome!