search

psthrn42 / SCVI_Extract

19 stars 6 forks source link

Fix: Extract everything on full extract #5

Closed FullLifeGames closed 2 years ago

FullLifeGames commented 2 years ago

Previously, I just used the last entry in the data array and stored that in the "extracted" file. This was obviously wrong. Now I create a folder and store every file of the trpak with their hexed hash.

FullLifeGames commented 2 years ago

Tagging @psthrn42 since it is a merge to master.

bWFpbA commented 2 years ago

Quick request - can you make the Oodle library an argument or something? This hardcodes it to search for "oo2core_6_win64.dll", but I believe the latest is "oo2core_9_win64.dll". I don't think the API for OodleLZ_Decompress has changed at all, and so it should function the same.

FullLifeGames commented 2 years ago

Quick request - can you make the Oodle library an argument or something? This hardcodes it to search for "oo2core_6_win64.dll", but I believe the latest is "oo2core_9_win64.dll". I don't think the API for OodleLZ_Decompress has changed at all, and so it should function the same.

Good idea, I will get to it in the evening (going out for the day)

bWFpbA commented 2 years ago

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

FullLifeGames commented 2 years ago

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

Tagging @psthrn42 @Jack-Rickwood since they compiled the list.

bWFpbA commented 2 years ago

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

Tagging @psthrn42 @Jack-Rickwood since they compiled the list.

In my personal opinion, I think the list should just be removed, and directories should just be named by the key value from the trpfd file. I've gone through the list, and I've spotted multiple mistakes that don't resolve back to the original paths, For example: arc/messagedat/JPN_KANJI/commonzukan_comment_B.tbl.trpak should instead be: arc/message/dat/JPN_KANJI/common/zukan_comment_B.tbl.trpak And if you take an fnv1a64 hash of it excluding the arc/ prefix and .trpak suffix, like so: fnv1a64('message/dat/JPN_KANJI/common/zukan_comment_B.tbl') You'll get the result 0x2400B5B48C5C141A, which can be found in the trpfd hash table.

FullLifeGames commented 2 years ago

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

Tagging @psthrn42 @Jack-Rickwood since they compiled the list.

In my personal opinion, I think the list should just be removed, and directories should just be named by the key value from the trpfd file. I've gone through the list, and I've spotted multiple mistakes that don't resolve back to the original paths, For example: arc/messagedat/JPN_KANJI/commonzukan_comment_B.tbl.trpak should instead be: arc/message/dat/JPN_KANJI/common/zukan_comment_B.tbl.trpak And if you take an fnv1a64 hash of it excluding the arc/ prefix and .trpak suffix, like so: fnv1a64('message/dat/JPN_KANJI/common/zukan_comment_B.tbl') You'll get the result 0x2400B5B48C5C141A, which can be found in the trpfd hash table.

For now, I built a workaround until we have a new list: https://github.com/psthrn42/SCVI_Extract/pull/5/commits/24d9696fe417ea1f2b9c39948291e926dad8a398 We can of course also just remove that and just use the values, but the overview will be drastically worse.

Quick request - can you make the Oodle library an argument or something? This hardcodes it to search for "oo2core_6_win64.dll", but I believe the latest is "oo2core_9_win64.dll". I don't think the API for OodleLZ_Decompress has changed at all, and so it should function the same.

Made this more flexible with https://github.com/psthrn42/SCVI_Extract/pull/5/commits/4a2182e52b0fa5380c811903b3dbdc6ec71fe939

psthrn42 commented 2 years ago

To be honest, @FullLifeGames I'm wondering about the usefulness of full_extract at all. Can't really see a usecase for fully extracting all of the trpaks at once, and that would take a very long time i think (did you measure?). Seems like in most cases someone would just want to extract individual trpaks they're interested in (i added trpak_extract to my right click menu for convenience). If you still think it's useful though, I'm not opposed to merging.

Also, if you still want to go ahead, might look cleaner to call the functions already in trpfs_extract and trpak_extract rather than having duplicate code. Those files both already correctly extract (not just the last entry).

As for the list, I didn't actually compile it, someone on discord did. They have made a new list and it's much better, just hadn't bothered to push yet. Naming via the key value would probably be better, yeah, but this tool is horribly written (I don't even really know python) in the first place, and i'm sure it will become obselete the minute the game comes out and other people release their tools. And tbh i can't be bothered coding a better solution, if someone else does i'll merge.

psthrn42 commented 2 years ago

And I guess a problem: KeyError: 'arc/applidatabtl_teambtl_team_select_00.arc.trpak' I'm guessing the tool doesn't like the 1.0.1 data? Probably something to do with names_changed.txt and names_original.txt not having the new keys.

Yes this would be the issue

psthrn42 commented 2 years ago

(This PR does contain a bunch of other good changes so you're free to merge regardless)

FullLifeGames commented 2 years ago

To be honest, @FullLifeGames I'm wondering about the usefulness of full_extract at all. Can't really see a usecase for fully extracting all of the trpaks at once, and that would take a very long time i think (did you measure?). Seems like in most cases someone would just want to extract individual trpaks they're interested in (i added trpak_extract to my right click menu for convenience). If you still think it's useful though, I'm not opposed to merging.

As for the list, I didn't actually compile it, someone on discord did. They have made a new list and it's much better, just hadn't bothered to push yet. Naming via the key value would probably be better, yeah, but this tool is horribly written (I don't even really know python) in the first place, and i'm sure it will become obselete the minute the game comes out and other people release their tools. And tbh i can't be bothered coding a better solution, if someone else does i'll merge.

Don't have real numbers, but definitely took me over an hour. For me (even when it was working wrongly), it was helpful to just get a grasp on how everything looked and was convenient, when you have the time to spare and have a full extraction run => therefore this script.

But I get your concern, that this is really not efficient.

Also, if you still want to go ahead, might look cleaner to call the functions already in trpfs_extract and trpak_extract rather than having duplicate code. Those files both already correctly extract (not just the last entry).

Yeah, right now, this is all a bit of a mess, maybe in the evening, I'll try to cleanup after myself.