read/write region into/from ram completely

[DonRudiX]

Reading und writing a whole region in one single I/O process is, as far as I understand the code, not possible at the current state of the library. That results in 1024(32x32 chunks) single read- and write-processes if you want to edit every chunk in a region. Can you add a class 'regionRAM' or something alike thats works similar to the current region-class, but has a method to read and write the whole region at once? I am aware that this variant uses a lot more ram but for editing large worlds it would be a really good time saver.

[LB--]

I could make Region a base class and have DiskRegion and MemoryRegion derivations - would that work for you?

[DonRudiX]

I think that would work for me as long as reading and writing chunks from/into a MemoryRegion doesn't produce any I/O activity to the HDD.

[LB--]

Yes, with a MemoryRegion the entire region file will be read into memory upon construction and all operations on the region will operate on the memory.

[DonRudiX]

Sounds perfect.

[LB--]

@MasterRudi I added the MemoryRegion class in d5d55c1cbc92d914ddf02f5f77873cdbf143f3bb - it is completely untested currently. If you want you can try it out and let me know of any issues. If you construct it with Region.ChunkCompression.None then the chunks will be fully uncompressed in memory while you work with them.

[DonRudiX]

Unfortunatly it's not working. This code produces an exception for ChunkCompression.None

test class

package test;

import java.io.File;

import com.lb_stuff.mcmodify.minecraft.IDs;

import com.lb_stuff.mcmodify.minecraft.Chunk;
import com.lb_stuff.mcmodify.minecraft.MemoryRegion;
import com.lb_stuff.mcmodify.minecraft.Region.ChunkCompression;

public class TestMemoryRegion
{
    public static void main(String[] args)
    {
        String inputFile = "data/minecraft/src/flatworld_master_r.0.0.mca";
        String outputFile = "data/r.0.0.mca";

        try
        {
            System.out.println("reading region");
            MemoryRegion mr = new MemoryRegion(new File(inputFile), ChunkCompression.None);
            //MemoryRegion mr = new MemoryRegion(new File(inputFile), ChunkCompression.GZip);
            //MemoryRegion mr = new MemoryRegion(new File(inputFile), ChunkCompression.Zlib);
            System.out.println("region read to memory");

            System.out.println("editing chunks");
            for (int i = 0; i < 32; i++)
            {
                for (int j = 0; j < 32; j++)
                {
                    Chunk c = mr.getChunk(i, j);
                    c.BlockID(1+i*32, 10, +j*32, IDs.Cobblestone);
                    mr.setChunk(i, j, c);
                }
            }
            System.out.println("editing chunks done");

            System.out.println("saving region to disk");
            mr.saveToFile(new File(outputFile));
            System.out.println("region saved to disk");
        }
        catch (Exception e1)
        {
            e1.printStackTrace();
            System.exit(0);
        }

        try
        {
            System.out.println("thread sleeping to estimated memory usage");
            Thread.sleep(5000);
        }
        catch (Exception e)
        {
            // none
        }
    }
}

Exception:

java.lang.IllegalArgumentException: There is no Id for compression scheme None
    at com.lb_stuff.mcmodify.minecraft.Region$ChunkCompression.getId(Region.java:65)
    at com.lb_stuff.mcmodify.minecraft.MemoryRegion.saveToFile(MemoryRegion.java:90)
    at test.TestMemoryRegion.main(TestMemoryRegion.java:39)

Changing the compression to to gzip or zlib does not produce the exception but the outputfile will only be like 9KB big for a flatworld inputfile.

[DonRudiX]

Sry, wrong button. I didn't want to close the Issue

[LB--]

Did you try either of the other two compression options? I'll start debugging this in an hour or so.

Changing the compression to to gzip or zlib does not produce the exception but the outputfile will only be like 9KB big for a flatworld inputfile.

What happens when Mincraft loads the region? How big was the original input file?

EDIT: I'm fixing some of the obvious issues with my implementation - stand by.

[LB--]

OK, I'm still not in an environment where I can test the code, but try 97d92e883293db2e8f8103b254b895a7dfc5e079 - the saveToFile() method now takes a Region.ChunkCompression as its second parameter (don't use None obviously).

[DonRudiX]

It still doesn't work. The IllegalArgumentException was removed but when using the edited region in a minecraft client the output console says:

[00:48:21] [Server thread/ERROR]: Chunk file at 1,7 is in the wrong location; relocating. (Expected 1, 7, got 1, 6) [00:48:21] [Server thread/ERROR]: Chunk file at 14,2 is in the wrong location; relocating. (Expected 14, 2, got 1, 6) [00:48:21] [Server thread/ERROR]: Chunk file at 14,4 is in the wrong location; relocating. (Expected 14, 4, got 1, 3) [00:48:21] [Server thread/ERROR]: Chunk file at 14,6 is in the wrong location; relocating. (Expected 14, 6, got 1, 5) [00:48:21] [Server thread/ERROR]: Chunk file at 14,5 is in the wrong location; relocating. (Expected 14, 5, got 1, 5) [00:48:22] [Server thread/ERROR]: Chunk file at 14,8 is in the wrong location; relocating. (Expected 14, 8, got 2, 8) [00:48:22] [Server thread/ERROR]: Chunk file at 14,9 is in the wrong location; relocating. (Expected 14, 9, got 3, 8) [00:48:23] [Server thread/ERROR]: Chunk file at 15,4 is in the wrong location; relocating. (Expected 15, 4, got 1, 5) [00:48:24] [Server thread/ERROR]: Chunk file at 2,7 is in the wrong location; relocating. (Expected 2, 7, got 7, 7) [00:48:27] [Server thread/ERROR]: Chunk file at 10,6 is in the wrong location; relocating. (Expected 10, 6, got 9, 6) [00:48:27] [Server thread/ERROR]: Chunk file at 6,6 is in the wrong location; relocating. (Expected 6, 6, got 9, 6) [00:49:45] [Server thread/WARN]: Wrong location! (2, 7) should be (7, 7), afs['Slime'/360947, l='flat', x=47,89, y=4,42, z=112,04]

Seems like something with the location in the chunks isn't constructed correctly. For testing I used my example from above but changed the the saveToFile into mr.saveToFile(new File(outputFile), ChunkCompression.Zlib);

ChunkCompression.GZip doesn't work either, both produce a 4101KB file when the inputfile is 4104KB (flatworld, fully explored region, generated by minecraft client)

[LB--]

A difference of 3KB is normal - there will be differences in compression as well as the padding of empty sectors depending on how Minecraft implements it differently. Specifically, the last sector in the region file doesn't get padded with my implementation whereas it does with Minecraft's.

The server log you posted is related to an already-known issue not related to the MemoryRegion code - see #5 for details. If you try with the 0,0 region file it should work without issue.

[DonRudiX]

I don't think it is related to #5 because all of my tests so far took place on a 0,0 region

[LB--]

Hm, I will do some investigating then.