search

WalkerMx / AutoCrispy

Automatically apply AI Upscaling on Dumped Textures
GNU General Public License v3.0
100 stars 9 forks source link

Feature request PCSX2EX support #9

Closed NoUserNameForYou closed 3 years ago

NoUserNameForYou commented 3 years ago

Hello!

I love what you're doing, thanks a lot!

pcsx2ex creates a textures folder, then game id folder and dumps r8g8b8a8 dds files in there and it needs r8g8b8a8 or b8r8g8a8 dds files to replace as well.

It reads from gameid.yaml file from txtconfig folder. Its structure is basically this:

ProcessTEX:
  0xCBF51E7B: face.dds

no spaces ProcessTEXcomma enter tab or double space 0casesensitivextexture'snamecomma space editedtexture.dds

note: notepad++ creates four spaces with tab and the yaml file needs two

Can you automate this process somehow? Thank you

WalkerMx commented 3 years ago

@NoUserNameForYou I'll look into this. Right now, It looks like PCSX2-EX is exclusively dumping and loading DDS files, which don't have the support that PNG does.

Also, does PCSX2-EX build the YAML file, or is that done manually?

NoUserNameForYou commented 3 years ago

It's done manually. However, I don't think you need to implement it as I'm guessing its dev is going to make it himself.

We just need dds availabilty in the softwares you use and automatically mapping of the 0xoldtex: newtex.dds files in the yaml file.

WalkerMx commented 3 years ago

@NoUserNameForYou Well, handling the YAML shouldn't be too difficult in the short-term. It's unfortunate that the software only supports directly mapping the files.

Adding DDS support to each of the backends would be up to each of those individual developers, but I'll see about adding some sort of converter inside AutoCrispy. The only drawback to this method is that there would be considerable overhead converting from and to DDS in the chain.

NoUserNameForYou commented 3 years ago

I have asked the png support question in his discord.

WalkerMx commented 3 years ago

@NoUserNameForYou Quick question, which release of PCSX2-EX are you using? Also, if possible, could you attach a YAML file that for sure works?

Thanks

NoUserNameForYou commented 3 years ago

@NoUserNameForYou Quick question, which release of PCSX2-EX are you using? Also, if possible, could you attach a YAML file that for sure works?

Thanks

Mate I'm sorry, I was being dumb. It was late at night and I told you that the yaml file's name would be gameid (which would be something like SCES50760) but it was another id which is the same name with the pnach cheat files. I don't know the nomenclature for that, HEX id maybe?

I'm using 3.1 from his github with this plugin: https://cdn.discordapp.com/attachments/808298057992372234/853060967376879636/GSdxEX-2.7z (from his discord)

Adding yaml file as well. PCSX2-EX tex and yaml for ico.zip

The zip contains vanilla textures as well as edited textures and the yaml file as a drag and drop folder structure for ICO eu.

Changed files are options/button config square triangle cross and circle buttons' colors. Hit f10 to turn on and off.

WalkerMx commented 3 years ago

@NoUserNameForYou Thanks for this! I'd drummed up a program to manage the YAML, and it wasn't working - turns out, it works just fine, with the plugin you attached!

This is a piece of the program, for testing: ExtraCrispy.zip

Note that your Upscales folder must be set up just like the @DUMPS folder, with the game CRC folders inside it. It scans for matching textures, and automatically maps them. Unmatched textures are left blank in the file.

Also, back up your YAML files, because it overwrites them!

NoUserNameForYou commented 3 years ago

Right it does!

Thank you so much.

jlnhlfan commented 3 years ago

Note that your Upscales folder must be set up just like the @Dumps folder, with the game CRC folders inside it. It scans for matching textures, and automatically maps them. Unmatched textures are left blank in the file.

Yeah, okay. Then why didn't the yaml file update when I put a new texture into the @COM folder I have already created?

ghost commented 3 years ago

Huh. Would you look at that.

@WalkerMx is there a source for that ExtraCrispy program? Is it merged or in any sort of branches that I can see? Just as a precaution. People are getting virus alerts and I would like to whitelist this in my server if possible.

WalkerMx commented 3 years ago

@Topaz-Reality There isn't any source uploaded yet, because I was intending to merge this functionality into AutoCrispy ^^'

Actually, I didn't know anyone else was following this, I was just letting @NoUserNameForYou test it out. Do you want a zip of the scratch solution I'm using ("ExtraCrispy")?

WalkerMx commented 3 years ago

@jlnhlfan I'm not 100% sure what you mean. Basically, inside your "@COM" folder, the folder structure needs to be identical to that of the @DUMP folder. even the DDS file names must match. Otherwise, it doesn't work?

Also, since this is just a scratch version, it doesn't have any realtime processing, you have to run it each time to rebuild.

jlnhlfan commented 3 years ago

The folders are structured the same, and the DDS files have the same name.

WalkerMx commented 3 years ago

@jlnhlfan I don't mind taking a look at this. Could I get a screenshot of the two folders, and maybe a copy of the YAML?

jlnhlfan commented 3 years ago

pics and yam.zip Here's the pictures. Uptaed the yaml because of the lack of realtime processing.

WalkerMx commented 3 years ago

@jlnhlfan Huh.. In your "@COM\82CA153C" folder, I count 5 files, and there are 5 mapped files in your YAML file..

All the rest are blank, because there are no matches. Shouldn't that be correct?

jlnhlfan commented 3 years ago

Yeah, that's true. This is still a WIP. explorer_Tb7KXrNe6L Meanwhile, here is a new screenshot of the @com folder.

WalkerMx commented 3 years ago

@jlnhlfan Right. Eventually, these routines will just be added into AutoCrispy, after I get them all debugged.

After you add another texture into the folder, does manually updating the YAML add the texture to the YAML?

jlnhlfan commented 3 years ago

Yes, it does.

WalkerMx commented 3 years ago

@jlnhlfan Then, so far so good! That's all the build I posted here does.

I'm about 2/3 of the way done programming the conversion portion, so that AutoCrispy can automatically upscale the DDS files.

jlnhlfan commented 3 years ago

Alright. And make sure it doesn't get flagged as malware.

WalkerMx commented 3 years ago

@NoUserNameForYou , @Topaz-Reality , @jlnhlfan

I've got the bulk of the programming worked out. If you want to try batch upscaling while I'm integrating these features into AutoCrispy, I'm posting the testing version here along with the source. Note, that it requires the nVidia DDS Utilities to be installed:

DDS Utilities ExtraCrispy.zip ExtraCrispy_Source.zip

The "DDS to PNG" button makes two folders, a "_RAWPNG" folder and a "_SCALEDPNG" folder. It spits out PNGs in the "_RAWPNG" folder when you push the button. Set AutoCrispy to use "_RAWPNG" as the input, and "_SCALEDPNG" as the output. When the upscaling is done, use the "PNG to DDS" button. When it finishes, use the "Make YAML" button.

It might be a little bit before I have this feature integrated into AutoCrispy. I'm going to start cleaning up the code, and I want to eliminate the need for a third-party DDS library (I'm already manually reading the Targa files the nVidia tools spit out ><).

As always, back up your files before testing any of this.

NoUserNameForYou commented 3 years ago

It doesn't output any png files here.

Is DDS utils locked to certain NV hardware?

WalkerMx commented 3 years ago

@NoUserNameForYou It shouldn't be. The most recent version seems to have been released in 2007. The only limitation I can think of right now would be that it only works on 64-bit systems (this was just me being lazy though).

Are the Targa files appearing in the folder in "@DUMP"?

NoUserNameForYou commented 3 years ago

No they are not.

I set the dump folder and I set the upscale folder. When I click on dds to png, the folders are created. And nothing else happens. Let me restart for a good measure.

Edit: Restart didn't fix the issue. Tried removing @ incase nvidia cmdline app didn't like it, it didn't fix it either.

WalkerMx commented 3 years ago

@NoUserNameForYou Let me know if your DDS Utilities is installed in:

"C:\Program Files (x86)\NVIDIA Corporation\DDS Utilities"

Also, which game were you testing with? (Was it ico?)

NoUserNameForYou commented 3 years ago

Okay it works, it's the space in my file path. Used e:\dump and e:\upscale and it works.

Edit: tried to quote the path in your app but it crashes.

WalkerMx commented 3 years ago

@NoUserNameForYou Aha!

Okay, I'll put that on the list of "things to fix".

ghost commented 3 years ago

@Topaz-Reality There isn't any source uploaded yet, because I was intending to merge this functionality into AutoCrispy ^^'

Actually, I didn't know anyone else was following this, I was just letting @NoUserNameForYou test it out. Do you want a zip of the scratch solution I'm using ("ExtraCrispy")?

Please do provide. The program looks dope but it shows as a virus. I need to verify a false positivity.

ghost commented 3 years ago

@NoUserNameForYou , @Topaz-Reality , @jlnhlfan

I've got the bulk of the programming worked out. If you want to try batch upscaling while I'm integrating these features into AutoCrispy, I'm posting the testing version here along with the source. Note, that it requires the nVidia DDS Utilities to be installed:

DDS Utilities ExtraCrispy.zip ExtraCrispy_Source.zip

The "DDS to PNG" button makes two folders, a "_RAWPNG" folder and a "_SCALEDPNG" folder. It spits out PNGs in the "_RAWPNG" folder when you push the button. Set AutoCrispy to use "_RAWPNG" as the input, and "_SCALEDPNG" as the output. When the upscaling is done, use the "PNG to DDS" button. When it finishes, use the "Make YAML" button.

It might be a little bit before I have this feature integrated into AutoCrispy. I'm going to start cleaning up the code, and I want to eliminate the need for a third-party DDS library (I'm already manually reading the Targa files the nVidia tools spit out ><).

As always, back up your files before testing any of this.

Thanks. After a look through I have seen that this program is safe to use. It is now whitelisted.

WalkerMx commented 3 years ago

@Topaz-Reality Perfecto.

Thanks!

WalkerMx commented 3 years ago

It wasn't really a bug, but it was annoying me. The PNGs were being unwrapped inside out, and I was wasting processor cycles fixing it with the DDS Utilities, so here:

Slightly updated, with no more inside out PNGs: ExtraCrispy.zip ExtraCrispy_Source.zip

WalkerMx commented 3 years ago

@NoUserNameForYou , @Topaz-Reality

I've finished the preliminary integration into AutoCrispy. It can now convert to and from DDS files in pretty much any format, courtesy of Microsoft's Texconv Utility. I switched from nVidia's tools, for the sake of portability and compatibility. Just place it in the folder with AutoCrispy, or in it's own folder in AutoCrispy's folder. (ExtraCrispy was also switched to Texconv. It must be in the same folder as ExtraCrispy!)

To set it up for DDS files, the first item in your chain must be Texconv set to "DDS Input", and the last item will need to be Texconv set to "DDS Output". Currently, for PCSX2-EX, the format should be set to BGRA; if the plugin is updated, the format must match the format it uses.

The source code for AutoCrispy is updated, but I've still got a little housekeeping to do before I do a full release. This is a preview of the next release:

AutoCrispy.zip

Now, regarding the YAML portion, I've decided it's outside the scope of AutoCrispy to have a feature so application-specific. It is my hope that the plugin for PCSX2-EX will eventually only need a YAML for custom-named textures, and will detect correspondingly named textures automatically.

That being said, I made a small update to ExtraCrispy to let it update the YAML over time (adjustable between 1s ~ 60s): ExtraCrispy.zip ExtraCrispy_Source.zip

Good luck, and enjoy ^^

NoUserNameForYou commented 3 years ago

Thanks mate :)

WalkerMx commented 3 years ago

This feature is complete and has been added as of AutoCrispy 1.0.3.0.