Add saving

[blixt]

This library only reads right now. Being able to save would be a great boon. The biggest hurdle to completing this task is to be able to expand the B-tree database.

[Qix-]

Hey - writing a re-implementation of this library and running into a similar challenge. The way I'm seeing the implementation is that the BTree database works on top of the SBBF format, and expanding blocks is basically the following process (correct me if I'm wrong):

Free block is not -1:

1. Read the free block's next free block ID (last 4 bytes) and update the next free block pointer found in the header
2. Convert the free block to either a leaf or index block (using the header bytes)
3. Consume the block

Free block is -1:

1. ftruncate (resize) the file block_size bytes further
2. Initialize with the proper header bytes
3. Consume the new block

And the alternate (freeing a block; when a file is shortened down enough to lose n blocks):

1. Set the newly freed block's next free block ID to the current next free block pointer found in the header
2. Set the newly freed block's type header (first two bytes) to 'FF'
3. Set the next free block pointer in the header to the newly freed block's ID

Is this correct?

[wizzomafizzo]

@Qix- sorry to hijack this but does starfuse reading already work? Or do you know if updating py-starbound to support the new format is easy enough? I am really out of my depth on this one and starcheat is kinda screwed without it working. I think @blixt is out of commission atm

[wizzomafizzo]

I'd be more interested in reading from the database directly, as cool as mounting it with FUSE is. Not sure how cross-platform FUSE is without adding dependencies.

[Qix-]

@wizzomafizzo reading from starfuse works, yes. Writing doesn't yet. FUSE is cross-platform sans Windows, and is very stable and mature on Unix systems.

FUSE allows you to read directly from the database itself without extracting anything. That's the whole point of the FUSE filesystem being glued to the Starfuse libs.

As well my library differs from Blixt's because I use mmap() instead of read/write buffers, allowing a bit more direct modification as if the entire library was addressed in-memory. Reduces latency a bit more, too. I'm also in the process of refactoring some of the pieces I borrowed from this library to be better structured for when write access is implemented.

[wizzomafizzo]

Thanks for getting back so quick. That's really good news. I had a quick check through your repo and yeah that looks good to me. I will give it a test and let you know how it goes

Unfortunately Windows is the no 1 platform for starcheat so no go there. It's pretty cool nonetheless

Sent from my iPhone

On 17 May 2016, at 11:49 AM, Josh Junon notifications@github.com wrote:

@wizzomafizzo reading from starfuse works, yes. Writing doesn't yet. FUSE is cross-platform sans Windows, and is very stable and mature on Unix systems.

FUSE allows you to read directly from the database itself without extracting anything. That's the whole point of the FUSE filesystem being glued to the Starfuse libs.

As well my library differs from Blixt's because I use mmap() instead of read/write buffers, allowing a bit more direct modification as if the entire library was addressed in-memory. Reduces latency a bit more, too.

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub

[Qix-]

So for starcheat you could use everything but the FUSE portions - the library itself isnt specific to FUSE operations and even when writing functionality is added it won't be coupled with FUSE ops at all. Without FUSE, starfuse is basically just py-starbound with mmap-ed files and soon to be writing capability.

[blixt]

Hey guys! Great to hear Starbound file format projects are still alive and breathing. :) Unfortunately, I have no time these days to work on this (starting a company does that) but I do want to revisit all these projects once Starbound finally hits 1.0 and file formats (presumably) won't change as much.

Regarding saving, the closest I got to doing anything here was with repair.py but that just rebuilds the file from scratch, attempting to keep the indexes balanced. That process might actually be useful down the line to speed up the loading of old worlds, but I presume Starbound already does this automatically in the background.

[Qix-]

@blixt I was going to ask - what is the point of the alternating root nodes? Is that for transaction-based modification, or am I misinterpreting the code?

[blixt]

@Qix- My memory of the Starbound internals is a bit faded now, but yes I believe it's intended for being able to modify the on-disk representation while the game is running (and reading the world). I remember a common issue for broken worlds would be an incomplete write on the current root node and sometimes switching to the other one would fix it (but point at older blocks, some of which have been repurposed resulting in glitches).

You could possibly ask @kyren on Starbound's IRC about details as well.

[Qix-]

@blixt Awesome, thanks :D

[wizzomafizzo]

Good luck with your company dude!

Sent from my iPhone

On 18 May 2016, at 6:00 AM, Josh Junon notifications@github.com wrote:

@blixt Awesome, thanks :D

— You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub