Closed KIC closed 9 years ago
You can get array like access using this technique, see below :
@Test public void bondExample() throws IOException, InterruptedException {
ChronicleMapBuilder builder = ChronicleMapBuilder.of(String.class, BondVOInterface.class)
.averageKeySize(10);
try (ChronicleMap<String, BondVOInterface> chm = newInstance(builder)) {
BondVOInterface bondVO = chm.newValueInstance();
try (MapKeyContext wc = chm.acquireContext("369604103", bondVO)) {
bondVO.setIssueDate(parseYYYYMMDD("20130915"));
bondVO.setMaturityDate(parseYYYYMMDD("20140915"));
bondVO.setCoupon(5.0 / 100); // 5.0%
BondVOInterface.MarketPx mpx930 = bondVO.getMarketPxIntraDayHistoryAt(0);
mpx930.setAskPx(109.2);
mpx930.setBidPx(106.9);
BondVOInterface.MarketPx mpx1030 = bondVO.getMarketPxIntraDayHistoryAt(1);
mpx1030.setAskPx(109.7);
mpx1030.setBidPx(107.6);
}
}
}
package net.openhft.chronicle.map.fromdocs;
import net.openhft.lang.model.constraints.MaxSize;
public interface BondVOInterface { /* add support for entry based locking */ @Deprecated() void busyLockEntry() throws InterruptedException;
@Deprecated()
void unlockEntry();
long getIssueDate();
void setIssueDate(long issueDate); /* time in millis */
long getMaturityDate();
void setMaturityDate(long maturityDate); /* time in millis */
long addAtomicMaturityDate(long toAdd);
boolean compareAndSwapCoupon(double expected, double value);
double getCoupon();
void setCoupon(double coupon);
double addAtomicCoupon(double toAdd);
String getSymbol();
void setSymbol(@MaxSize(20) String symbol);
// OpenHFT Off-Heap array[ ] processing notice ‘At’ suffix
void setMarketPxIntraDayHistoryAt(@MaxSize(7) int tradingDayHour, MarketPx mPx);
/* 7 Hours in the Trading Day:
* index_0 = 9.30am,
* index_1 = 10.30am,
…,
* index_6 = 4.30pm
*/
MarketPx getMarketPxIntraDayHistoryAt(int tradingDayHour);
/* nested interface - empowering an Off-Heap hierarchical “TIER of prices”
as array[ ] value */
interface MarketPx {
double getCallPx();
void setCallPx(double px);
double getParPx();
void setParPx(double px);
double getMaturityPx();
void setMaturityPx(double px);
double getBidPx();
void setBidPx(double px);
double getAskPx();
void setAskPx(double px);
}
}
On 23 Apr 2015, at 16:23, KIC notifications@github.com wrote:
First of all I really like openHFT, thanks for sharing!!
I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!
See this quick and dirty example: Hello World
public class HftTest { public static void main(String[] args) { ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, IArray.class).create();
TestArray ta1 = new TestArray(); ta1.setdata(new double[]{1, 2, 3, 4, 5}); TestArray ta2 = new TestArray(); ta1.setdata(new double[]{5, 6, 7, 8, 9}); proxyMap.put(1, ta1); proxyMap.put(2, ta2); IArray using = proxyMap.newValueInstance(); System.out.println(Arrays.toString(proxyMap.getUsing(1, using).getdata())); System.out.println(Arrays.toString(proxyMap.getUsing(2, using).getdata())); }
} Interface
public interface IArray extends Serializable { double[] getdata(); void setdata(double[] data); } Implementation
public class TestArray implements IArray { private double[] data;
@Override public double[] getdata() { return data; } @Override public void setdata(double[] data) { this.data = data; }
} — Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21.
Hmmm what is @MaxSize needed for or what is its impact? Is it safe to set max size to some really big number say: 2,147,483,647 (max int) since I only know the real size at runtime.
Hmm even @MaxSize(1000) is not working for doubles ... I am thinking about using java tools to compile a class at runtime with the correct sized annotation. But therefore I would need to know how I can create bigger arrays then 10. Lets say something in the rage 10'000 to 100'000. Do you have an Idea how I could create such big arrays?
At the moment we simulate arrays through indexed access.
public interface IArray { double getDataAt(@MaxSize(16) int n); void setDataAt(int n, double data); }
This avoid dealing with arrays directly or the garbage overhead. It not not as natural as what you propose of course. Note: the indexed entries can themselves be generated data types from interfaces.
On 23 April 2015 at 16:23, KIC notifications@github.com wrote:
First of all I really like openHFT, thanks for sharing!!
I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!
See this quick and dirty example: Hello World
public class HftTest { public static void main(String[] args) { ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, IArray.class).create();
TestArray ta1 = new TestArray(); ta1.setdata(new double[]{1, 2, 3, 4, 5}); TestArray ta2 = new TestArray(); ta1.setdata(new double[]{5, 6, 7, 8, 9}); proxyMap.put(1, ta1); proxyMap.put(2, ta2); IArray using = proxyMap.newValueInstance(); System.out.println(Arrays.toString(proxyMap.getUsing(1, using).getdata())); System.out.println(Arrays.toString(proxyMap.getUsing(2, using).getdata())); }
}
Interface
public interface IArray extends Serializable { double[] getdata(); void setdata(double[] data); }
Implementation
public class TestArray implements IArray { private double[] data;
@Override public double[] getdata() { return data; } @Override public void setdata(double[] data) { this.data = data; }
}
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21.
Instead of using the generated classes to do this, if you need plain double[] you can implement this yourself.
public class DoubleArray implements Byteable { private static int LENGTH = 0; // assume a 32-bit size. private static int CAPACITY = LENGTH + 4; // assume a 32-bit size. private static int BASE = CAPACITY + 4;
private Bytes bytes;
private long offset;
public DoubleArray(int capacity) {
bytes = DirectStore.allocate(BASE + capacity * 8L).bytes();
offset = 0;
}
@Override
public void bytes(Bytes bytes, long offset) {
this.bytes = bytes;
this.offset = offset;
}
@Override
public Bytes bytes() {
return bytes;
}
@Override
public long offset() {
return offset;
}
@Override
public int maxSize() {
return BASE + length() * 8;
}
public int length() {
return bytes.readInt(LENGTH + offset);
}
public int capacity() {
return bytes.readInt(CAPACITY + offset);
}
public double getDataAt(int index) {
if (index < 0 || index >= length()) throw new
ArrayIndexOutOfBoundsException(); return bytes.readDouble(BASE + offset + index * 8L); }
public void setDataAt(int index, double d) {
if (index < 0 || index >= capacity()) throw new
ArrayIndexOutOfBoundsException(); if (length() <= index) setLength(index + 1); bytes.writeDouble(BASE + offset + index * 8L, d); }
public void setLength(int length) {
if (length < 0 || length >= capacity()) throw new
IllegalArgumentException(); bytes.writeInt(LENGTH + offset, length); }
public void addData(double d) {
int index = length();
if (index >= capacity()) throw new IllegalStateException();
bytes.writeInt(LENGTH + offset, index + 1);
bytes.writeDouble(BASE + offset + index * 8L, d);
}
public void setData(double[] doubles) {
if (doubles.length > capacity()) throw new IllegalArgumentException();
bytes.writeInt(LENGTH + offset, doubles.length);
for (int index = 0; index < doubles.length; index++)
bytes.writeDouble(BASE + offset + index * 8L, doubles[index]);
}
public int getDataUsing(double[] doubles) {
int length = Math.min(length(), doubles.length);
for (int index = 0; index < length; index++)
doubles[index] = bytes.readDouble(BASE+offset+index*8L);
return length;
}
}
You can create a DoubleArray by its capacity alone or by doing a get(key) (takes a copy) or getUsing(key, doubleArray) (uses the data in the Map) on a Map.
On 23 April 2015 at 17:18, KIC notifications@github.com wrote:
Hmm even @MaxSize https://github.com/MaxSize(1000) is not working for doubles ... I am thinking about using java tools to compile a class at runtime with the correct sized annotation but therefore I would need to know how I can create bigger arrays then 10. Lets say something in the rage 10'000 to 100'000. Do you have an Idea how I could create such big arrays?
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-95638107 .
Hmm tanks for this class! I really appreciate your help, sadly I can not get this work by myself. I truly have tried to figure this out by myself but this C style memory calculations are to wired for me :-) It will throw a IllegalArgumentException at:
public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); <<<< --- here bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); }
for:
public TestArray(double[] data) { this.data = new DoubleArray(data.length); this.data.setData(data); }
The problem is that capacity() always returns 0. I am not sure if the constant LENGTH=0 is the problem.
I was forgetting to set the capacity which is why it was always zero. I have added a unit test
The LENGTH, CAPACITY and BASE are the offset of those fields.
On 24 April 2015 at 12:05, KIC notifications@github.com wrote:
Hmm tanks for this class! I really appreciate your help, sadly I can not get this work by myself. I truly have tried to figure this out by myself but this C style memory calculations are to wired for me :-) It will throw a IllegalArgumentException at:
public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); <<<< --- here bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); }
for:
public TestArray(double[] data) { this.data = new DoubleArray(data.length); this.data.setData(data); }
The problem is that capacity() always returns 0. I am not sure if the constant LENGTH=0 is the problem.
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-95896881 .
Thanks! I have just found the class but not a unit test, have you pushed the test?
https://github.com/OpenHFT/Chronicle-Map/search?utf8=%E2%9C%93&q=DoubleArray
This class is slightly different but still does not work. The cpacity() method relies only on two variables the constant CAPACITY and offset:
public int capacity() { return bytes.readInt(CAPACITY + offset); }
But we never change either CAPACITY nor offset (not in the constructor and not in the setters)
public DoubleArray(int capacity) { bytes = DirectStore.allocate(BASE + capacity * 8L).bytes(); offset = 0; } public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); }
So what we will get is (regadless of our construction parameter): bytes.readInt(4 + 0);
Correct, I have now uploaded the classes I sent as attachments. I guess they didn't come through this group.
https://github.com/OpenHFT/Chronicle-Map/tree/master/src/test/java/net/openhft/lang/values
When you do bytes.readInt(offset) you are providing where to find the value you need. I have move the CAPACITY to the start as this is a bit more logical.
On 25 April 2015 at 09:15, KIC notifications@github.com wrote:
Thanks! I have just found the class but not a Test, have you pushed the test?
https://github.com/OpenHFT/Chronicle-Map/search?utf8=%E2%9C%93&q=DoubleArray
This class is slightly different but still does not work. The cpacity() method relies only on two variables the constant CAPACITY and offset:
public int capacity() { return bytes.readInt(CAPACITY + offset); }
But we never change either CAPACITY nor offset (not in the constructor and not in the setters)
public DoubleArray(int capacity) { bytes = DirectStore.allocate(BASE + capacity * 8L).bytes(); offset = 0; }
public void setData(double[] doubles) { if (doubles.length > capacity()) throw new IllegalArgumentException(); bytes.writeInt(LENGTH + offset, doubles.length); for (int index = 0; index < doubles.length; index++) bytes.writeDouble(BASE + offset + index * 8L, doubles[index]); }
So what we will get is (regadless of our construction parameter): bytes.readInt(4 + 0);
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-96157073 .
Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.
Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder. (SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder. (ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more
DoubleArray a = new DoubleArray(10); a.setData(new double[]{1, 2, 3, 4, 5}); DoubleArray b = new DoubleArray(10); b.setData(new double[]{5, 6, 7, 8, 9}); ChronicleMapproxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create(); proxyMap.put(1, a); proxyMap.put(2, b); System.out.println(proxyMap.get(1)); System.out.println(proxyMap.get(2));
I suggest you use an interface like DoubleArray10 example below interface DoubleArray10 {
double getDoubleAt(int i);
void setDoubleAt(@MaxSize(10) int i, double value);
}
@Test public void test() {
ChronicleMap<Integer,DoubleArray10> proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray10
.class).create();
final DoubleArray10 a = proxyMap.newValueInstance();
a.setDoubleAt(1,1);
a.setDoubleAt(2,2);
final DoubleArray10 b = proxyMap.newValueInstance();
b.setDoubleAt(1,1);
b.setDoubleAt(2, 2);
proxyMap.put(1, a);
proxyMap.put(2, b);
System.out.println(proxyMap.get(1));
System.out.println(proxyMap.get(2));
Assert.assertEquals(proxyMap.get(1), proxyMap.get(2));
}
On 28 Apr 2015, at 12:35, KIC notifications@github.com wrote:
Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.
Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more DoubleArray a = new DoubleArray(10); a.setData(new double[]{1, 2, 3, 4, 5});
DoubleArray b = new DoubleArray(10); b.setData(new double[]{5, 6, 7, 8, 9}); ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create(); proxyMap.put(1, a); proxyMap.put(2, b); System.out.println(proxyMap.get(1)); System.out.println(proxyMap.get(2));
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97028016.
But this brings us back the fact that I have to know the size at compile time. But I only know the size at runtime. So back to field one and compile interfaces at runtime using java tools and the MaxSize annotation?
I have added a test to DoubleArrayTest. For now I have added a hack to gives a DoubleArray a notional size if it hasn't been initialised. I will see if there is a more natural way to say what you expect the average size of an entry to be.
On 28 April 2015 at 12:35, KIC notifications@github.com wrote:
Interestingly one will get a NPE when trying to use the DoubleArray as value in a Map (even as property of another object). Interestingly because this still happens if I provide a default constructor setting some size. I really have no idea how ChronicalMap is calling the method "length" without using a instantiated object.
Exception in thread "main" java.lang.IllegalStateException: java.lang.NullPointerException at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:60) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller$Default.(ByteableMarshaller.java:195) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.of(ByteableMarshaller.java:39) at net.openhft.chronicle.map.SerializationBuilder.configureByDefault(SerializationBuilder.java:134) at net.openhft.chronicle.map.SerializationBuilder.(SerializationBuilder.java:97) at net.openhft.chronicle.map.ChronicleMapBuilder.(ChronicleMapBuilder.java:169) at net.openhft.chronicle.map.ChronicleMapBuilder.of(ChronicleMapBuilder.java:191) at com.mindbusters.mql2.test.hft.HftTest.main(HftTest.java:47) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:497) at com.intellij.rt.execution.application.AppMain.main(AppMain.java:140) Caused by: java.lang.NullPointerException at com.xxx.mql2.test.hft.DoubleArray.length(DoubleArray.java:47) at com.xxx.mql2.test.hft.DoubleArray.maxSize(DoubleArray.java:43) at net.openhft.chronicle.hash.serialization.internal.ByteableMarshaller.initSize(ByteableMarshaller.java:58) ... 12 more
DoubleArray a = new DoubleArray(10); a.setData(new double[]{1, 2, 3, 4, 5}); DoubleArray b = new DoubleArray(10); b.setData(new double[]{5, 6, 7, 8, 9}); ChronicleMap proxyMap = ChronicleMapBuilder.of(Integer.class, DoubleArray.class).create(); proxyMap.put(1, a); proxyMap.put(2, b); System.out.println(proxyMap.get(1)); System.out.println(proxyMap.get(2));
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97028016 .
Yes this works great!! Thanks a very lot!
System.out.println(proxyMap.getUsing(2, using).getDataAt(2));
By using getUsing, you should be able to work garbage free. If you are going to resize the capacity of the entry you need to perform a put () On 29 Apr 2015 8:42 am, "KIC" notifications@github.com wrote:
Yes this works great!! Thanks a very lot!
System.out.println(proxyMap.getUsing(2, using).getDataAt(2));
— Reply to this email directly or view it on GitHub https://github.com/OpenHFT/Chronicle-Map/issues/21#issuecomment-97335610 .
First of all I really like openHFT, thanks for sharing!!
I try to follow the example using the off heap direct access to primitives ( https://github.com/OpenHFT/Chronicle-Map#off-heap-storage-and-how-using-a-proxy-object-can-improve-performance ). Sadly this seems not to work with arrays. But this would be a very cool thing. Even better would be if we could act on an specific index of an array directly off heap!
See this quick and dirty example: Hello World
Interface
Implementation