.pak format update

[LunaTheArcticFox]

Chucklefish has changed the .pak format in the unstable branch. I'm not sure how much has changed, but at the very least the ID needs changing from "Assets1" to "Assets2" (which seems to work on my end, though we haven't received an update to the .pak yet).

[wizzomafizzo]

If the change turns out to be pretty simple, can we make StarDB backwards compatible (like.. just accept both IDs)? It would be nice to support this now but also not break it for stable users.

[LunaTheArcticFox]

Assuming today's update added new files to the .pak, it seems to not have any problems being read with the current library. https://github.com/ohgoditburns/starbound/commit/930d0584d15875d859077012aca80d67ad0b44c3 has the assets changes for this patch. I'll be holding off until the next patch that adds assets to push an update for the Java implementation of this, but I'm expecting it to work at this point with just the ID change/addition.

[McSimp]

When the format changes, all that needs to be done is create a new class like the one in https://github.com/McSimp/StarDB/blob/master/stardb/databases/assetdatabase.py, change the identifier from 'Assets1' to 'Assets2', then implement whatever functionality is needed.

An exception will be raised if you try to open a database with an invalid identifier (sorry, I haven't had time to implement proper exception classes so it's just a plain old Exception) so I guess you could use that to get the right one.

It also may be worth looking at https://github.com/blixt/py-starbound from @blixt which seems pretty well written and may make handling different versions a bit easier.

[debugman18]

What are the next steps to get it working without waiting for a patch, assuming I change Assets1 to Assets2 in assetsdatabase.py? Or is it a more complicated fix?

[LunaTheArcticFox]

That's all you need to do, I think. I haven't tested it all that much, but it appears to work.

[debugman18]

Hmm. It gives me this error when that is done.

return struct.unpack(fmt, buff)[0]

struct.error: unpack requires a bytes object of length 1 Segmentation fault

[McSimp]

Could you give us a stack trace and the file you were trying to open?

[debugman18]

I wasn't trying to open any particular file, just trying to run starcheat.py.

Traceback (most recent call last): File "/home/debaggu/Documents/starcheat/build/starcheat.py", line 70, in main() File "/home/debaggu/Documents/starcheat/build/starcheat.py", line 67, in main gui.mainwindow.MainWindow() File "/home/debaggu/Documents/starcheat/build/gui/mainwindow.py", line 83, in init for species in self.assets.species().get_species_list(): File "/home/debaggu/Documents/starcheat/build/assets.py", line 234, in species return Species(self) File "/home/debaggu/Documents/starcheat/build/assets.py", line 704, in init self.humanoid_config = self.assets.read("/humanoid.config", self.assets.vanilla_assets) File "/home/debaggu/Documents/starcheat/build/assets.py", line 198, in read db.open() File "/home/debaggu/Documents/starcheat/build/stardb/btree/btreedatabase.py", line 103, in open self.readRoot() File "/home/debaggu/Documents/starcheat/build/stardb/btree/btreedatabase.py", line 69, in readRoot self.rootIsLeaf = unpack('?', rootData.read(1)) File "/home/debaggu/Documents/starcheat/build/stardb/util.py", line 5, in unpack return struct.unpack(fmt, buff)[0] struct.error: unpack requires a bytes object of length 1 Segmentation fault

[wizzomafizzo]

blixt is totes old school and uses python 2. help us @McSimp you're our only hope :(

[McSimp]

@debugman18 The function it's crashing in has something going on that I haven't fully explored yet (the unknownBool) so that might be part of it. I may have done the maths wrong on the extra seek.

You could try changing Line 60 to rootData = self.blockStorage.readUserData(28, 15) or maybe Line 66 to rootData.seek(7, 1), but really I'm just guessing and can't say for sure until I actually have a proper look at it.

[debugman18]

@McSimp Changing Line 60 in btreedatabase.py seemed to do the trick; StarCheat launches now. I'll be sure to yell if it blows up again.

Well, trying to rebuild the index results in:

Traceback (most recent call last): File "/home/debaggu/Documents/starcheat/build/gui/utils.py", line 240, in rebuild_db rebuild = build_assets_db(self.dialog) File "/home/debaggu/Documents/starcheat/build/gui/utils.py", line 46, in build_assets_db asset_files = assets_db.find_assets() File "/home/debaggu/Documents/starcheat/build/assets.py", line 112, in find_assets vanilla_assets = self.scan_asset_folder(vanilla_path) File "/home/debaggu/Documents/starcheat/build/assets.py", line 135, in scan_asset_folder index = [(x, pak_path) for x in db.getFileList()] File "/home/debaggu/Documents/starcheat/build/stardb/databases/assetdatabase.py", line 25, in getFileList self.fileList = unpackStringList(indexData) File "/home/debaggu/Documents/starcheat/build/stardb/util.py", line 25, in unpackStringList strings.append(stream.read(strLen).decode('utf-8')) UnicodeDecodeError: 'utf-8' codec can't decode byte 0x8f in position 7: invalid start byte

[blixt]

Two things have changed recently in the .pak files:

1. SHA-256 hashes now work correctly
2. The file index for Assets2 is now a new kind of list, consisting of path string + space character + 32 bytes (SHA-256 hash of the file contents of path) instead of just the path string

py-starbound already supports the new format and can use old and new versions of the packed.pak file interchangeably.

[wizzomafizzo]

@blixt have you ever tested py-starbound after shooting it through 2to3?

[wizzomafizzo]

and @McSimp is it a pain for you to keep maintaining this? i'm not against switching if that makes things easier

[McSimp]

@wizzomafizzo Yeah I really don't have the time at the moment to maintain this, and it was never fully completed in the first place.

[wizzomafizzo]

that's cool man, I'll see what I can do to change over

YOU'RE STILL MY MAIN BAE THO NEVER FORGET THAT <3

On 28/02/14 08:14, Will Donohoe wrote:

@wizzomafizzo https://github.com/wizzomafizzo Yeah I really don't have the time at the moment to maintain this, and it was never fully completed in the first place.

— Reply to this email directly or view it on GitHub https://github.com/McSimp/StarDB/issues/3#issuecomment-36308339.