Project-Path-of-Exile-Wiki / PyPoE

Collection of Python Tools for Path of Exile
GNU General Public License v3.0
24 stars 21 forks source link

Experimental DAT64 use in exporter #96

Closed zao closed 1 year ago

zao commented 1 year ago

Abstract

As 3.20 removed 32-bit DAT files we need to support DAT64 (or DATL/DATL64) files instead in the schema and the exporters.

Action Taken

The DAT exporter is adapted to map all .dat extensions into .dat64 without any choice. The fixed-size datatype ulong previously used for foreign keys which used to be 8 bytes (4+4 actually) turns into ref|out which is 4+4 or 8+8 bytes depending on bitness. The variable-size datatype ref|generic previously used for keys referencing other rows in the same table is renamed to ref|self to communicate its purpose and remains 4 or 8 bytes depending on bitness. Wiki exporters are adapted to use DAT64 files at the outermost level while any indirection via foreign fields naturally use 64-bit files as long as the initial file was 64-bit.

Caveats

There may be some exporters that dynamically build _files lists or use RelationalReader, DatReader or DatFile in some other way and which hasn't been covered by this PR. This is all quite hacky and should probably be thought through whether a more elegant solution exists or if the generality of supporting both 32-bit and 64-bit files should be stripped out. Another alternative would be to use DATL files (UTF-32LE rather than UTF-16LE) which still ship with the game files.

This has been tested on 3.19.2b with:

pypoe_exporter wiki items item id Metadata/Items/DivinationCards/DivinationCardTheDoctor -p
pypoe_exporter dat json dat.json --file BaseItemTypes

FAO

@Journeytojah

zao commented 1 year ago

Superseded by #97 as it incorporates the commits from this PR.