Enable off heap access for arrays through proxies

[KIC]

First of all I really like openHFT, thanks for sharing!!

I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!

See this quick and dirty example: Hello World

public class HftTest {
    public static void main(String[] args) {
        ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class,
                IArray.class).create();

        TestArray ta1 = new TestArray();
        ta1.setdata(new double[]{1, 2, 3, 4, 5});

        TestArray ta2 = new TestArray();
        ta1.setdata(new double[]{5, 6, 7, 8, 9});

        proxyMap.put(1, ta1);
        proxyMap.put(2, ta2);

        IArray using = proxyMap.newValueInstance();

        System.out.println(Arrays.toString(proxyMap.getUsing(1, using).getdata()));
        System.out.println(Arrays.toString(proxyMap.getUsing(2, using).getdata()));
    }
}

Interface

public interface IArray extends Serializable {
    double[] getdata();
    void setdata(double[] data);
}

Implementation

public class TestArray implements IArray {
    private double[] data;

    @Override
    public double[] getdata() {
        return data;
    }

    @Override
    public void setdata(double[] data) {
        this.data = data;
    }
}

[RobAustin]

You can get array like access using this technique, see below :

@Test public void bondExample() throws IOException, InterruptedException {

ChronicleMapBuilder builder = ChronicleMapBuilder.of(String.class, BondVOInterface.class)
        .averageKeySize(10);

try (ChronicleMap<String, BondVOInterface> chm = newInstance(builder)) {
    BondVOInterface bondVO = chm.newValueInstance();
    try (MapKeyContext wc = chm.acquireContext("369604103", bondVO)) {
        bondVO.setIssueDate(parseYYYYMMDD("20130915"));
        bondVO.setMaturityDate(parseYYYYMMDD("20140915"));
        bondVO.setCoupon(5.0 / 100); // 5.0%

        BondVOInterface.MarketPx mpx930 = bondVO.getMarketPxIntraDayHistoryAt(0);
        mpx930.setAskPx(109.2);
        mpx930.setBidPx(106.9);

        BondVOInterface.MarketPx mpx1030 = bondVO.getMarketPxIntraDayHistoryAt(1);
        mpx1030.setAskPx(109.7);
        mpx1030.setBidPx(107.6);
    }

}

}

package net.openhft.chronicle.map.fromdocs;

import net.openhft.lang.model.constraints.MaxSize;

public interface BondVOInterface { /* add support for entry based locking */ @Deprecated() void busyLockEntry() throws InterruptedException;

@Deprecated()
void unlockEntry();

long getIssueDate();

void setIssueDate(long issueDate);  /* time in millis */

long getMaturityDate();

void setMaturityDate(long maturityDate);  /* time in millis */

long addAtomicMaturityDate(long toAdd);

boolean compareAndSwapCoupon(double expected, double value);

double getCoupon();

void setCoupon(double coupon);

double addAtomicCoupon(double toAdd);

String getSymbol();

void setSymbol(@MaxSize(20) String symbol);

// OpenHFT Off-Heap array[ ] processing notice ‘At’ suffix
void setMarketPxIntraDayHistoryAt(@MaxSize(7) int tradingDayHour, MarketPx mPx);

/* 7 Hours in the Trading Day:
 * index_0 = 9.30am,
 * index_1 = 10.30am,
 …,
 * index_6 = 4.30pm
 */

MarketPx getMarketPxIntraDayHistoryAt(int tradingDayHour);

/* nested interface - empowering an Off-Heap hierarchical “TIER of prices”
as array[ ] value */
interface MarketPx {
    double getCallPx();

    void setCallPx(double px);

    double getParPx();

    void setParPx(double px);

    double getMaturityPx();

    void setMaturityPx(double px);

    double getBidPx();

    void setBidPx(double px);

    double getAskPx();

    void setAskPx(double px);
}

}

On 23 Apr 2015, at 16:23, KIC notifications@github.com wrote:

First of all I really like openHFT, thanks for sharing!!

I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!

See this quick and dirty example: Hello World

public class HftTest { public static void main(String[] args) { ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, IArray.class).create();

    TestArray ta1 = new TestArray();
    ta1.setdata(new double[]{1, 2, 3, 4, 5});

    TestArray ta2 = new TestArray();
    ta1.setdata(new double[]{5, 6, 7, 8, 9});

    proxyMap.put(1, ta1);
    proxyMap.put(2, ta2);

    IArray using = proxyMap.newValueInstance();

    System.out.println(Arrays.toString(proxyMap.getUsing(1, using).getdata()));
    System.out.println(Arrays.toString(proxyMap.getUsing(2, using).getdata()));
}

} Interface

public interface IArray extends Serializable { double[] getdata(); void setdata(double[] data); } Implementation

public class TestArray implements IArray { private double[] data;

@Override
public double[] getdata() {
    return data;
}

@Override
public void setdata(double[] data) {
    this.data = data;
}

} — Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21.

[KIC]

Hmmm what is @MaxSize needed for or what is its impact? Is it safe to set max size to some really big number say: 2,147,483,647 (max int) since I only know the real size at runtime.

[KIC]

Hmm even @MaxSize(1000) is not working for doubles ... I am thinking about using java tools to compile a class at runtime with the correct sized annotation. But therefore I would need to know how I can create bigger arrays then 10. Lets say something in the rage 10'000 to 100'000. Do you have an Idea how I could create such big arrays?

[peter-lawrey]

At the moment we simulate arrays through indexed access.

public interface IArray { double getDataAt(@MaxSize(16) int n); void setDataAt(int n, double data); }

This avoid dealing with arrays directly or the garbage overhead. It not not as natural as what you propose of course. Note: the indexed entries can themselves be generated data types from interfaces.

On 23 April 2015 at 16:23, KIC notifications@github.com wrote:

First of all I really like openHFT, thanks for sharing!!

I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!

See this quick and dirty example: Hello World

public class HftTest { public static void main(String[] args) { ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, IArray.class).create();

    TestArray ta1 = new TestArray();
    ta1.setdata(new double[]{1, 2, 3, 4, 5});

    TestArray ta2 = new TestArray();
    ta1.setdata(new double[]{5, 6, 7, 8, 9});

    proxyMap.put(1, ta1);
    proxyMap.put(2, ta2);

    IArray using = proxyMap.newValueInstance();

    System.out.println(Arrays.toString(proxyMap.getUsing(1, using).getdata()));
    System.out.println(Arrays.toString(proxyMap.getUsing(2, using).getdata()));
}

}

Interface

public interface IArray extends Serializable { double[] getdata(); void setdata(double[] data); }

Implementation

public class TestArray implements IArray { private double[] data;

@Override
public double[] getdata() {
    return data;
}

@Override
public void setdata(double[] data) {
    this.data = data;
}

}

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21.

[peter-lawrey]

Instead of using the generated classes to do this, if you need plain double[] you can implement this yourself.

public class DoubleArray implements Byteable { private static int LENGTH = 0; // assume a 32-bit size. private static int CAPACITY = LENGTH + 4; // assume a 32-bit size. private static int BASE = CAPACITY + 4;

private Bytes bytes;
private long offset;

public DoubleArray(int capacity) {
    bytes = DirectStore.allocate(BASE + capacity * 8L).bytes();
    offset = 0;
}

@Override
public void bytes(Bytes bytes, long offset) {
    this.bytes = bytes;
    this.offset = offset;
}

@Override
public Bytes bytes() {
    return bytes;
}

@Override
public long offset() {
    return offset;
}

@Override
public int maxSize() {
    return BASE + length() * 8;
}

public int length() {
    return bytes.readInt(LENGTH + offset);
}

public int capacity() {
    return bytes.readInt(CAPACITY + offset);
}

public double getDataAt(int index) {
    if (index < 0 || index >= length()) throw new

ArrayIndexOutOfBoundsException(); return bytes.readDouble(BASE + offset + index * 8L); }

public void setDataAt(int index, double d) {
    if (index < 0 || index >= capacity()) throw new

ArrayIndexOutOfBoundsException(); if (length() <= index) setLength(index + 1); bytes.writeDouble(BASE + offset + index * 8L, d); }

public void setLength(int length) {
    if (length < 0 || length >= capacity()) throw new

IllegalArgumentException(); bytes.writeInt(LENGTH + offset, length); }

public void addData(double d) {
    int index = length();
    if (index >= capacity()) throw new IllegalStateException();
    bytes.writeInt(LENGTH + offset, index + 1);
    bytes.writeDouble(BASE + offset + index * 8L, d);
}

public void setData(double[] doubles) {
    if (doubles.length > capacity()) throw new IllegalArgumentException();
    bytes.writeInt(LENGTH + offset, doubles.length);
    for (int index = 0; index < doubles.length; index++)
        bytes.writeDouble(BASE + offset + index * 8L, doubles[index]);
}

public int getDataUsing(double[] doubles) {
    int length = Math.min(length(), doubles.length);
    for (int index = 0; index < length; index++)
        doubles[index] = bytes.readDouble(BASE+offset+index*8L);
    return length;
}

}

You can create a DoubleArray by its capacity alone or by doing a get(key) (takes a copy) or getUsing(key, doubleArray) (uses the data in the Map) on a Map.

On 23 April 2015 at 17:18, KIC notifications@github.com wrote:

Hmm even @MaxSize https://github.com/MaxSize(1000) is not working for doubles ... I am thinking about using java tools to compile a class at runtime with the correct sized annotation but therefore I would need to know how I can create bigger arrays then 10. Lets say something in the rage 10'000 to 100'000. Do you have an Idea how I could create such big arrays?

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-95638107 .

[KIC]

Hmm tanks for this class! I really appreciate your help, sadly I can not get this work by myself. I truly have tried to figure this out by myself but this C style memory calculations are to wired for me :-) It will throw a IllegalArgumentException at:

    public void setData(double[] doubles) {
        if (doubles.length > capacity()) throw new IllegalArgumentException(); <<<< --- here
        bytes.writeInt(LENGTH + offset, doubles.length);
        for (int index = 0; index < doubles.length; index++)
            bytes.writeDouble(BASE + offset + index * 8L, doubles[index]);
    }

for:

public TestArray(double[] data) {
        this.data = new DoubleArray(data.length);
        this.data.setData(data);
    }

The problem is that capacity() always returns 0. I am not sure if the constant LENGTH=0 is the problem.

[peter-lawrey]

I was forgetting to set the capacity which is why it was always zero. I have added a unit test

The LENGTH, CAPACITY and BASE are the offset of those fields.

On 24 April 2015 at 12:05, KIC notifications@github.com wrote:

Hmm tanks for this class! I really appreciate your help, sadly I can not get this work by myself. I truly have tried to figure this out by myself but this C style memory calculations are to wired for me :-) It will throw a IllegalArgumentException at:

public void setData(double[] doubles) {
    if (doubles.length > capacity()) throw new IllegalArgumentException(); <<<< --- here
    bytes.writeInt(LENGTH + offset, doubles.length);
    for (int index = 0; index < doubles.length; index++)
        bytes.writeDouble(BASE + offset + index * 8L, doubles[index]);
}

for:

public TestArray(double[] data) { this.data = new DoubleArray(data.length); this.data.setData(data); }

The problem is that capacity() always returns 0. I am not sure if the constant LENGTH=0 is the problem.

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-95896881 .

[KIC]

Thanks! I have just found the class but not a unit test, have you pushed the test?

https://github.com/OpenHFT/Chronicle-Map/search?utf8=%E2%9C%93&q=DoubleArray

This class is slightly different but still does not work. The cpacity() method relies only on two variables the constant CAPACITY and offset:

    public int capacity() {
        return bytes.readInt(CAPACITY + offset);
    }

But we never change either CAPACITY nor offset (not in the constructor and not in the setters)

public DoubleArray(int capacity) {
        bytes = DirectStore.allocate(BASE + capacity * 8L).bytes();
        offset = 0;
    }
public void setData(double[] doubles) {
        if (doubles.length > capacity()) throw new IllegalArgumentException();
        bytes.writeInt(LENGTH + offset, doubles.length);
        for (int index = 0; index < doubles.length; index++)
            bytes.writeDouble(BASE + offset + index * 8L, doubles[index]);
    }

So what we will get is (regadless of our construction parameter): bytes.readInt(4 + 0);

[peter-lawrey]

Correct, I have now uploaded the classes I sent as attachments. I guess they didn't come through this group.

https://github.com/OpenHFT/Chronicle-Map/tree/master/src/test/java/net/openhft/lang/values

When you do bytes.readInt(offset) you are providing where to find the value you need. I have move the CAPACITY to the start as this is a bit more logical.

On 25 April 2015 at 09:15, KIC notifications@github.com wrote:

Thanks! I have just found the class but not a Test, have you pushed the test?

https://github.com/OpenHFT/Chronicle-Map/search?utf8=%E2%9C%93&q=DoubleArray

This class is slightly different but still does not work. The cpacity() method relies only on two variables the constant CAPACITY and offset:

public int capacity() {
    return bytes.readInt(CAPACITY + offset);
}

But we never change either CAPACITY nor offset (not in the constructor and not in the setters)

public DoubleArray(int capacity) { bytes = DirectStore.allocate(BASE + capacity * 8L).bytes(); offset = 0; }

public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); }

So what we will get is (regadless of our construction parameter): bytes.readInt(4 + 0);

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-96157073 .

[KIC]

Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.

Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException
    at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60)
    at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195)
    at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39)
    at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134)
    at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97)
    at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169)
    at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191)
    at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140)
Caused by: java.lang.NullPointerException
    at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47)
    at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43)
    at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58)
    ... 12 more

        DoubleArray a = new DoubleArray(10);
        a.setData(new double[]{1, 2, 3, 4, 5});
        DoubleArray b = new DoubleArray(10);
        b.setData(new double[]{5, 6, 7, 8, 9});
        ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create();
        proxyMap.put(1, a);
        proxyMap.put(2, b);
        System.out.println(proxyMap.get(1));
        System.out.println(proxyMap.get(2));

[RobAustin]

I suggest you use an interface like DoubleArray10 example below interface DoubleArray10 {

double getDoubleAt(int i);

void setDoubleAt(@MaxSize(10) int i, double value);

}

@Test public void test() {

ChronicleMap<Integer,DoubleArray10> proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray10
        .class).create();

final DoubleArray10 a = proxyMap.newValueInstance();

a.setDoubleAt(1,1);
a.setDoubleAt(2,2);

final DoubleArray10 b = proxyMap.newValueInstance();
b.setDoubleAt(1,1);
b.setDoubleAt(2, 2);

proxyMap.put(1, a);
proxyMap.put(2, b);

System.out.println(proxyMap.get(1));
System.out.println(proxyMap.get(2));

Assert.assertEquals(proxyMap.get(1), proxyMap.get(2));

}

On 28 Apr 2015, at 12:35, KIC notifications@github.com wrote:

Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.

Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more DoubleArray a = new DoubleArray(10); a.setData(new double[]{1, 2, 3, 4, 5});

    DoubleArray b = new DoubleArray(10);
    b.setData(new double[]{5, 6, 7, 8, 9});

    ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create();
    proxyMap.put(1, a);
    proxyMap.put(2, b);

    System.out.println(proxyMap.get(1));
    System.out.println(proxyMap.get(2));

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97028016.

[KIC]

But this brings us back the fact that I have to know the size at compile time. But I only know the size at runtime. So back to field one and compile interfaces at runtime using java tools and the MaxSize annotation?

[peter-lawrey]

I have added a test to DoubleArrayTest. For now I have added a hack to gives a DoubleArray a notional size if it hasn't been initialised. I will see if there is a more natural way to say what you expect the average size of an entry to be.

On 28 April 2015 at 12:35, KIC notifications@github.com wrote:

Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.

Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more

    DoubleArray a = new DoubleArray(10);
    a.setData(new double[]{1, 2, 3, 4, 5});

    DoubleArray b = new DoubleArray(10);
    b.setData(new double[]{5, 6, 7, 8, 9});

    ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create();
    proxyMap.put(1, a);
    proxyMap.put(2, b);

    System.out.println(proxyMap.get(1));
    System.out.println(proxyMap.get(2));

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97028016 .

[KIC]

Yes this works great!! Thanks a very lot!

        System.out.println(proxyMap.getUsing(2, using).getDataAt(2));

[peter-lawrey]

By using getUsing, you should be able to work garbage free. If you are going to resize the capacity of the entry you need to perform a put () On 29 Apr 2015 8:42 am, "KIC" notifications@github.com wrote:

Yes this works great!! Thanks a very lot!

    System.out.println(proxyMap.getUsing(2, using).getDataAt(2));

— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97335610 .