Saving: Added Region Save Mode

[Nebual]

A very scalable system. Sectors (8x8x8 blocks) are stored 4x4x4 in headerless files, 2 bytes per block. Position data is not stored, but inferred based on the byte offset within the file.

Whole regions (32x32x32 blocks) are read at once when world._show_sector asks. Sectors can be written to individually, though without a dirty_sectors system (TODO) it currently writes all sectors in memory to disk during a save.

[ghost]

Saving is still very slow...

[BertrandBordage]

@avelanarius: I started working on #40 today. For the moment, creating a new world with this future SQL solution is slow (~ 30% slower than current creation), but any other save/load takes less than 0.5s. And it's done with SQLite on the same process than the game. When it'll be on another process, it'll be much faster. And if we still have performance issue, we can consider using PostgreSQL, which should solve anything.

Every SQL save takes more than twice the size of @Nebual's current system. Compressing with 7z divides it by 6. A compressed 100-sized world therefore takes less than 500 kB. So size will never be a problem. A really big world will never reach 250 MB if compressed.

However, it's still very experimental and we need @Nebual's solution.

[Nebual]

I believe loading 64 sectors (1 whole region file, or 32768 blocks) takes between 0.0005s and 0.004s under my Region system.

Theres a few odd behaviors with the Queuing system that I believe are slowing down sector loading considerably. And I notice that if you keep the urgent queue full (by constantly flying and thus loading new chunks) the hide_sector calls don't occur until its totally empty. I also don't think the world.delete_opposite_task aspect is working, though its a good idea.

[BertrandBordage]

I have extreme lag since this pull request. Less than 40 FPS when I'm moving (sometimes even 15 FPS), the game freezes every second and there are dozens of bugs, including a ground in the sky, some crashes with KeyError, falling from every unloaded sector, screen update glitches, exposed blocks not calculated properly, etc. And sector updates are slower than a snail, even when the files were already generated in a previous game.

However, I must admin that opening an already created world is fast.

[BertrandBordage]

I also see an increase in RAM usage. I expected the opposite :(

[Nebual]

Odd, thats significantly worse than me. I'd be at 300fps standing still, dropping down to 120fps while loading chunks, the queueing prevented much framedrop I'm guessing. This is in pure python mode.

The "ground in the sky" is because my simple terrain generator is just blindly creating a layer of dirt at the bottom of all new Regions, including those in the sky. "exposed blocks not calculated properly"? "screen update glitches"?

Increase in RAM is both because more of the world is loaded now than before, and I'm not currently unloading sectors, so RAM usage will climb as you explore, but since theres nothing to explore and it resets when you reopen the game, I wasn't going to delay releasing this for that.

Edit: Also, you can switch back to the previous save if you need to.

[BertrandBordage]

That huge FPS drop I got is measured without Cython on an ASUS G74SX. So this is definitely not my computer that has a problem, though it's per-core power is limited.

You can also randomly see blocks that shouldn't be exposed: stone and ore blocks. In some games, I could see lots of "ore walls" under the ground, while waiting for sector updates. This is increasing the number of polygons to show, which decreases performance on computers with less GPU power.

The screen update glitches I saw (only a tiny square of the screen is refreshed) are maybe due to the chaotic freezes. I never got such glitches before, and no other OpenGL application was running at the same time.

[ronmurphy]

@BertrandBordage Quote:"You can also randomly see blocks that shouldn't be exposed: stone and ore blocks. In some games, I could see lots of "ore walls" under the ground, while waiting for sector updates. "

I saw this also. If you remove an Invisible block, then it, and it's neighbors, show up. Interesting.

[BertrandBordage]

I also note that the lazy queue, which is responsible for hiding sectors, is also extremely slow. This really shouldn't be.

[BertrandBordage]

@ronmurphy: It's normal that the "neighbours" blocks show up when you remove or place a block. It's thanks to World.check_neighbours. It's a good way to only load in your graphic card the blocks that should be visible. Minecraft works the same.

[ronmurphy]

@Bert I am trying to do something useful for once, I am trying to make a new sub from the existing World.initilize, at the land generation, that does +16 to current position to generate new terrain.

Not to sure how well it will work, or even if it will, but i will post results once i get it working.

[BertrandBordage]

"+16 to current position"?

[ronmurphy]

yes, up to 16 blocks away from current position should be generated, not more ... this could be changed, but i think that it will keep terrain generation time down, say, from +32 to current position.

[BertrandBordage]

OK. This is exactly what I'm currently doing in my SQL experiment ^^

[ronmurphy]

ok :D I had to stop as how i did so much not-working code i had to do a revert... Just more to learn more python!