search

speige / WC3MapDeprotector

Warcraft 3 Custom Map deprotector
https://www.youtube.com/@ai-gamer
MIT License
17 stars 2 forks source link

Send new list file entries discovered to an API #2

Open speige opened 7 months ago

speige commented 7 months ago

This allows users to benefit from each other because the entries can be included in the next release

speige commented 6 months ago

Find a free online service where we can host the listfile.txt - To avoid maintaining custom server on AWS. A simple example is a shared google drive document, but we have to tie it to an account, and there's a risk of that account user deleting the file. And then we have to think about security. We need the exe to edit the file via API, but not grant write access to random anonymous web users. And we don't want the file to disappear from the cloud in a few years, so it should be a reputable company. Google drive may be the best option, but do some research first before implementing.

Luashine commented 6 months ago

Neat idea but I think we can get a head start by doing something different and offline-only.

One major part of name discovery is the automatic extraction from current map resources, this includes object files (.doo, SLK and friends) but map scripts too (unless obfuscated). Maps cannot hide 99% of these. Though one piece of the puzzle iirc are MDX (model files) that have texture paths hardcoded in them. I don't think anything currently extracts from there?

Let's assume automatic extraction works to the best extent possible. We have a huge lake of maps to draw data from, the biggest websites hosting maps are:

My idea is to run the automatic extraction over all public maps we can get our hands on in the hopes of extracting 99.9% of used names from this entire corpus of maps. Then this listfile could be used to uncover new maps. Considering the extent of resource reusage, maps that were not protected at one point, maps that has an unprotected release by mistake -- this should provide us with excellent coverage.

As you see, this wouldn't even take any online services and probably help to reduce the search namespace for bruteforcing yet unknown names. One ad-hoc way to "forget" about a real-time online solution is to present users with a "listfile diff" that consists of new auto-detected names to be submitted upstream (to the authors of the program). A simple message box and Discord/Github links would do. Speaking of Github, automate issue/PR creation with API? :) Should be possible too. People don't usually extract API keys to abuse them.

I can run such an offline auto-detection effort if you make a tool to run recursively over all maps in a folder. 400 GB should be a good starting point ;) Of course if you add strict asserts, that map pool and me can act as your test pipeline on the way to figure out the various protections there are.

speige commented 6 months ago

I agree with all your points here. I consider this online listfile task "phase 2", if it ever gets done. My primary goal is to improve the unknown file detection so that we don't need a listfile at all. Then we could cancel the idea of keeping a combined online listfile.

In the future, I love the idea of running over every map file & doing batch deprotection for testing.

Right now with manual testing, it's fairly easy to find bugs, so there's no reason to do batch testing yet. & I want to look at other options for unknown file recovery before I invest in making a 100% comprehensive list file for every map.

I think the starting point is running a web spider to download all the maps & group them by map name (Dota/Enfos/etc, so each folder contains different versions of the same map). If you have time to do that, it would be awesome. I can set up an FTP folder for you to send all the files to me once you have them downloaded. Even though we're not ready for batch processing yet, this would be a HUGE help.

I'm not ready to release a public version of the app for batch processing yet. The tool is too fragile still (breaks on some maps) & also I don't want a rogue actor to deprotect every map in the world & publish them somewhere. For now, I want to keep it 1 map at a time, by hand, so the human can verify everything deprotected correctly & they can consider the ethics of if that map really should be released unprotected (if the author has abandoned development, for example).

Luashine commented 6 months ago

I think the starting point is running a web spider to download all the maps

I already have the 400 GB mentioned of EpicWar as I was afraid of the site disappearing one day. It's not up-to-date, I'd need to make a cron job to fetch the new maps as they appear. Some porn maps :) are deleted fairly actively (that was when I realized they exist).

I'm not ready to release a public version of the app for batch processing yet.

I didn't think of batch deprotection, but other parts of processing and testing. Like testing the parsers. I had made that triggerdata-converter parser extra strict and found a couple mistakes as a result. I don't know how strict War3Net is with parsing/asserts.

Also I love your focus on ethics, that warning window I saw in your video is great! Every so little obstacle is psychologically important (well I want to believe so).

speige commented 6 months ago

P.S. I think a lot of tools extract the texture file names from the MDX files, including mine. I don't know if any actually "parse" the MDX, but they just do a regex across it. The downside to regex is it's going to be imperfect and possibly miss some of the filenames.