Update to new (2.7.0.58+) vromfs/blk encoding

MuhAbroomz commented 3 years ago

I tried to unpack 2.7.0.56 blk files, but i couldn't due to an error. UnicodeDecodeError: 'utf-8' codec can't decode byte 0xb5 in position 2: invalid start byte Cap 2021-06-18 02-58-07-208 I have no idea what this mean... I need help

klensy commented 3 years ago

Lines from errors looks wrong, you tried latest (https://github.com/klensy/wt-tools/releases/tag/v0.2.2.8) version?

MuhAbroomz commented 3 years ago

Yes, it was the latest version. Re-download the tool, tried again and got another error Cap 2021-06-18 03-19-00-894

klensy commented 3 years ago

Hmm, perhaps non english path to files?

klensy commented 3 years ago

First error about attachable.blk says that file ~~already unpacked~~ not packed, strange.

MuhAbroomz commented 3 years ago

There is no non-english path to files. pre-2.7.0.58 version files are successfully unpacked. post-2.7.0.58 version files are failed. Seems like something's changed with 0.58 update. Edit: Just realized that the game version I'm trying to unpack was 0.58, not 0.56.

klensy commented 3 years ago

Here https://github.com/gszabi99/War-Thunder-Datamine/commits/master version 2.7.0.56 unpacked at least. I'll try to update game and check.

klensy commented 3 years ago

Yeah, i see error, blk format changed.

MuhAbroomz commented 3 years ago

Will there be an update?

gszabi99 commented 3 years ago

From 2.7.0.58 onwards vromfs unpacking sometimes fails, but even if it's successful, blk unpacking always fails anyway.

klensy commented 3 years ago

Some vromfs changed too.

Keksilton commented 3 years ago

The only major difference I have spotted in BLK files is the enumeration scheme. Type ids look the same.

klensy commented 3 years ago

Can anyone upload last working aces.vromfs.bin?

v3ntore commented 3 years ago

1606devaces.vromfs.bin.zip

Here's the last working aces file I've got, version 2.7.0.55 (version that was 3,5 GB and with leaked vehicles).

klensy commented 3 years ago

Restored vromfs unpacking, and i think know how to fix blk unpacking.

OshidaBCF commented 3 years ago

Something that you may really really really appreciate https://www.reddit.com/r/Warthunder/comments/o5r4v1/heads_up_on_vrom_and_blk_format_changes_for_data/

gszabi99 commented 3 years ago

Are there any updates on your progress that you can share with us?

klensy commented 3 years ago

@gszabi99 Yeah, vromfs unpacking seems fixed, working on blk's. Switched main branch to dev to show actual progress, instead of old master one.

OshidaBCF commented 3 years ago

FUCK YEAH

gszabi99 commented 3 years ago

Thanks! We really appreciate your efforts! :)

AidenTKZ commented 3 years ago

Any update on the state of BLK unpack ?

wtrabbithole commented 3 years ago

Hello. Is it possible to restore the unpacking of the War Thunder game resource files? It's not working right now. I am using the latest version of wt-tools from the dev branch. Below I attach a piece of the log.

orig Здравствуйте. Возможно ли восстановить распаковку файлов ресурсов игры War Thunder? Сейчас она не работает. Использую последнюю версию wt-tools из ветки dev. Ниже прикладываю кусочек лога. log.txt

OshidaBCF commented 3 years ago

Hello. Is it possible to restore the unpacking of the War Thunder game resource files? It's not working right now. I am using the latest version of wt-tools from the dev branch. Below I attach a piece of the log.

orig Здравствуйте. Возможно ли восстановить распаковку файлов ресурсов игры War Thunder? Сейчас она не работает. Использую последнюю версию wt-tools из ветки dev. Ниже прикладываю кусочек лога. log.txt

in case you didn't know, gaijin updated the format of the files, which broke the tools. Klensy is already working on updating them

wtrabbithole commented 3 years ago

I know the file format has changed. It looks like the aces.vromfs.bin file is not unpacked either. Here is the log, can it help with what

orig text: Я знаю что формат файлов изменился. Похоже что и файл aces.vromfs.bin не распаковывается. Вот лог, может чем поможет... aces.txt

OshidaBCF commented 3 years ago

I know the file format has changed. It looks like the aces.vromfs.bin file is not unpacked either. Here is the log, can it help with what

orig text: Я знаю что формат файлов изменился. Похоже что и файл aces.vromfs.bin не распаковывается. Вот лог, может чем поможет... aces.txt

... if you know that the format change, why do you expect the aces to work?

Keksilton commented 3 years ago

if you know that the format change, why do you expect the aces to work?

No need to be toxic around here. I'm sure there is enough of it in game. No such thing as a stupid question.

OshidaBCF commented 3 years ago

if you know that the format change, why do you expect the aces to work?

No need to be toxic around here. I'm sure there is enough of it in game. No such thing as a stupid question.

sorry I didn't wanted to be toxic.

Keksilton commented 3 years ago

@klensy idk if you had a chance to look into it. Increasing max_output_size for zstd compression seems to work for vromfs ( I set it to 200_000_000). Considering the addition of the name map, maybe it would be better to include blk unpacker as part of vromfs unpacker

AidenTKZ commented 3 years ago

Are there any updates on your progress that you can share with us ?

kotiq commented 3 years ago

I have a prototype of a codec of the new blk format, restored using a public algorithm. It builds an intermediate representation of the tree as a multi-dictionary of names and values and it is very slow.

OshidaBCF commented 3 years ago

how much slow ?

kotiq commented 3 years ago

i5-4670 @ 1 core in: tmpfs, out: hdd

pytest -vvs --durations=0 test_unpack_dir.py
========================================================================================================================== test session starts ==========================================================================================================================
platform linux -- Python 3.9.6, pytest-6.2.4, py-1.10.0, pluggy-0.13.1 -- /home/kotiq/.virtualenvs/blk/bin/python
cachedir: .pytest_cache
rootdir: /media/shared/src/games/WarThunder/blk, configfile: pytest.ini
collected 2 items                                                                                                                                                                                                                                                       

test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u] PASSED
test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u] PASSED

=========================================================================================================================== slowest durations ===========================================================================================================================
145.11s call     tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.05s call     tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
0.00s setup    tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
0.00s setup    tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[aces.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_fat_dir[game.vromfs.bin_u]
===================================================================================================================== 2 passed in 145.18s (0:02:25) =====================================================================================================================

OshidaBCF commented 3 years ago

145s for the call but 0 second for the teardown ?

i would have understood the opposite, well as long as it works.

also 145 second for aces is like, good

AidenTKZ commented 3 years ago

Aces.vromfs generally takes a lot of time to unpack

try with Char.vromfs.bin, with the old files/unpacker it takes me around 11.3 seconds

Edit : old Aces with old unpacker takes 59 seconds, so well never mind

kotiq commented 3 years ago

aces.vromfs.bin v 2.7.0.46 under the same conditions:

time python3 -m blk_unpack --format=strict_blk aces.vromfs.bin_u/
...
real    0m41.039s
user    0m38.300s
sys     0m0.823s

gszabi99 commented 3 years ago

@kotiq Will there perhaps be a public beta test of your program?

kotiq commented 3 years ago

I will need to simulate the interfaces so that your scripts at a higher level work as before. If you write the wrappers yourself, I will publish the library. There are examples of unpacking in the tests.

kotiq commented 3 years ago

@AidenTKZ

pytest -vvs --no-header --durations=0 test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
============================================================================ test session starts ============================================================================
collected 1 item                                                                                                                                                            

test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u] PASSED

============================================================================= slowest durations =============================================================================
13.76s call     tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
0.00s setup    tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
0.00s teardown tests/test_blk/test_binary/test_constructor/test_unpack_dir.py::test_unpack_slim_dir[char.vromfs.bin_u]
============================================================================ 1 passed in 13.78s =============================================================================

char.vromfs.bin v 2.5.1.125

time python3 -m blk_unpack --format=strict_blk char.vromfs.bin_u/
real    0m3.336s
user    0m3.238s
sys     0m0.074s

AidenTKZ commented 3 years ago

interesting

kotiq commented 3 years ago

Для ознакомления с новым форматом repo. Оцените указанные риски при возможной смене порядка в текстах, если будете их заливать в свои репозитории. По-хорошему, гляньте код, если пост на reddit был запутанным, и закодируйте сериализацию правильно.

gszabi99 commented 3 years ago

@kotiq Have you tried unpacking the new char vromfs? I always get a zstd decompression error. Is that something you can help with perhaps?

kotiq commented 3 years ago

@gszabi99 1 2 I used 5_000_000 for unpacking. @AidenTKZ wt_tools/blk_unpack.py:123,126,131. The specified lines contain ensure_ascii=False.

kotiq commented 3 years ago

The scheme for the Color was changed. Now RGBA color is stored as BGRA for raw bytes in a binary file

b'AABBCCDD' -> [0xCC, 0xBB, 0xAA, 0xDD]

L-Mustang commented 3 years ago

@kotiq For me blk_unpack_demo.py crashes on aces.vromfs.bin_u\config\tanksights.blk. Do you perhaps know what I'm doing wrong? All files before this specific file are unpacked correctly.

Traceback (most recent call last):
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 152, in <module>
    main()
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1137, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1062, in main
    rv = self.invoke(ctx)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 1404, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\site-packages\click\core.py", line 763, in invoke
    return __callback(*args, **kwargs)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 148, in main
    process_dir(path, out_type, is_sorted)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 115, in process_dir
    process_slim_dir(dir_path, names, out_type, is_sorted)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 132, in process_slim_dir
    process_slim_dir(entry.path, names, out_type, is_sorted)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 134, in process_slim_dir
    process_file(entry.path, names, out_type, is_sorted)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 76, in process_file
    serialize_text(root, ostream, out_type, is_sorted)
  File "D:\Programming\datablock\blk\tests\demo\blk_unpack_demo.py", line 19, in serialize_text
    return jsn.serialize(root, ostream, out_type, is_sorted)
  File "D:\Programming\datablock\blk\src\blk\json\serializer.py", line 124, in serialize
    json.dump(mapper.map(root), ostream, cls=NoIndentEncoder, ensure_ascii=False, indent=2, separators=(',', ': '),
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\json\__init__.py", line 180, in dump
    fp.write(chunk)
  File "C:\Users\user\AppData\Local\Programs\Python\Python39\lib\encodings\cp1252.py", line 19, in encode
    return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 1-3: character maps to <undefined>

tanksights.zip

kotiq commented 3 years ago

@Lord-Mustang The file contains lines in Russian. I didn't explicitly specify utf8 encoding for all text writer's instances. Fixed. Please, update the file.