Pointer support

[vemana]

Are there any plans to extend support to "pointer-like" members ? It allows C/C++ structs to be used more smoothly with Java.

Some thing like (by no means thought-through; but happy to discuss further.)

interface A {

  // Hide the pointer completely.
  // stores a long address and converts to B transparently using PointerBytesStore.
  // C++ equivalent of "B *bPtr;"
  @Pointer 
  B getB();
  void setB(B b);  // Stores to current "address" and error if address is NULL.
  void setB(B b, Allocator allocator); // Uses allocator to allocate memory if address is NULL.
}

Supporting pointers raises many questions including "Deep Copying" and "Who/Where allocates memory" for writes.

I am using Values in Java for interacting with a C++ app writing structs to shared memory. The pointed data changes over the app's lifecycle and I would like to be able to make a deep copy of those structs in Java before they change. It would be great if Chronicle does it without need for application code.

[peter-lawrey]

One of the problems with using an object reference is it is typically 32 bit even on a 64 bit JVM. So what we do is wrap an address with a proxy object which implements an interface. This way it looks like a java object even though the data is stored in native memory.

Peter. On 4 Jul 2016 22:15, "vemana" notifications@github.com wrote:

Are there any plans to extend support to "pointer-like" members ? It allows C/C++ structs to be used more smoothly with Java.

Some thing like (by no means thought-through; but happy to discuss further.)

interface A {

// Hide the pointer completely. // stores a long address and converts to B transparently using PointerBytesStore. // C++ equivalent of "B *bPtr;" @Pointer B getB(); void setB(B b); // Stores to current "address" and error if address is NULL. void setB(B b, Allocator allocator); // Uses allocator to allocate memory if address is NULL. }

Supporting pointers raises many questions including "Deep Copying" and "Who/Where allocates memory" for writes.

I am using Values in Java for interacting with a C++ app writing structs to shared memory. The pointed data changes over the app's lifecycle and I would like to be able to make a deep copy of those structs in Java before they change. It would be great if Chronicle does it without need for application code.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/OpenHFT/Chronicle-Values/issues/11, or mute the thread https://github.com/notifications/unsubscribe/ABBU8bPylRODy0d83XVGDvfkYtu2SnKHks5qSXf5gaJpZM4JEp4q .

[vemana]

Not sure I follow the 32 bit part. I am assuming B is itself a Values interface. So, any addresses will be the addresses provided by B's ByteStore. I assume these addresses are valid for any C++ readers.

I understand if Pointers isn't a direction for Values to focus on, but just want to make the issue clear with a sketch that conveys what I am thinking better than I can write in words:

// This long field "bAddress" is auto-generated by Values upon noticing @Pointer.
long getBAddress();
void setBAddress(long address);

// In the generated native implementation:
B getB(B b) {
  B b = Values.newNativeReference(B.class);
  BytesStore bs = BytesStore.nativePointer();
  bs.set(getBAddress(), b.maxSize());
  b.bytesStore(bs, 0, b.maxSize());
  return b;
}

// set is trickier with deep/no-deep/allocation etc.
void setB(B b) {
  setB(b, false); // no deep copy by default.
}

void setB(B b, boolean isDeepCopy) {
  BytesStore bs = b.bytesStore();

  if (bs.isNative()) {
    if (isDeepCopy) {
      // Do a deep copy
      B copiedB = Values.newNativeReference(B.class);
      long bAddress = Memory.allocate(copiedB.maxSize());
      setBAddress(bAddress); // remember our allocated memory.
      copiedB.copyFrom(b); // memcpy.
    } else {
      // Just a shallow copy: copy the pointer and not the pointed data.
      setBAddress(bs.address(0));
    }
  } else {
    // Non-native bytesStore; error ?
    throw ...
  }
}

I currently use PointerBytesStore and can interpret C++'s 64 bit pointers fine. However, the main pain point is C++ struct doesn't map neatly to a single Java interface.

struct A {
  B *bPtr; 
  int x;
} 

becomes

interface A { 
  long getBAddress();
  void setBAddress(long bAddress);

  long getX();
  void setX(long x);
} 

and

interface RealA { // application is written against this interface
  B getB();
  int getX();
}

and

class ACompanion implements RealA {
  A delegate;
  B getB() {
    // use PointerBytesStore on delegate.getAddress() to retrieve B.
    ....
  }
  int getX() {
    return delegate.getX();
  }
  void copyFrom(A that) {
     delegate.copyFrom(that);
     // also copy <that>'s bPtr data.
   }
}

[peter-lawrey]

While B is not a pointer in the sense that void * is. B can be a "pointer" java object which wraps native memory and makes it available in Java.

I think I see what you are saying, you want a field of another type which is actually a pointer and for Chronicle-Values to understand that rather than return a long address.

On 5 July 2016 at 07:49, vemana notifications@github.com wrote:

Not sure I follow the 32 bit part. I am assuming B is itself a Values interface. So, any addresses will be the addresses provided by B's ByteStore. I assume these addresses are valid for any C++ readers.

I understand if Pointers isn't a direction for Values to focus on, but just want to make the issue clear with a sketch that conveys what I am thinking better than I can write in words:

// This long field "bAddress" is auto-generated by Values upon noticing @Pointer. long getBAddress(); void setBAddress(long address);

// In the generated native implementation: B getB(B b) { B b = Values.newNativeReference(B.class); BytesStore bs = BytesStore.nativePointer(); bs.set(getBAddress(), b.maxSize()); b.bytesStore(bs, 0, b.maxSize()); return b; }

// set is trickier with deep/no-deep/allocation etc. void setB(B b) { setB(b, false); // no deep copy by default. }

void setB(B b, boolean isDeepCopy) { BytesStore bs = b.bytesStore();

if (bs.isNative()) { if (isDeepCopy) { // Do a deep copy B copiedB = Values.newNativeReference(B.class); long bAddress = Memory.allocate(copiedB.maxSize()); setBAddress(bAddress); // remember our allocated memory. copiedB.copyFrom(b); // memcpy. } else { // Just a shallow copy: copy the pointer and not the pointed data. setBAddress(bs.address(0)); } } else { // Non-native bytesStore; error ? throw ... } }

I currently use PointerBytesStore and can interpret C++'s 64 bit pointers fine. However, the main pain point is C++ struct doesn't map neatly to a single Java interface.

struct A { B *bPtr; int x; }

becomes

interface A { long getBAddress(); void setBAddress(long bAddress);

long getX(); void setX(long x); }

and

interface RealA { // application is written against this interface B getB(); int getX(); }

and

class ACompanion implements RealA { A delegate; B getB() { // use PointerBytesStore on delegate.getAddress() to retrieve B. .... } int getX() { return delegate.getX(); } void copyFrom(A that) { delegate.copyFrom(that); // also copy 's bPtr data. } }

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenHFT/Chronicle-Values/issues/11#issuecomment-230399807, or mute the thread https://github.com/notifications/unsubscribe/ABBU8YNrZDgAiWwGs_xmpDHgQl84PSFXks5qSf6FgaJpZM4JEp4q .

[vemana]

you want a field of another type which is actually a pointer and for Chronicle-Values to understand that rather than return a long address.

Yes, underlying storage is pointer, but the Java interface is a Byteable object at that pointer.

I understand the proxy comment now. I think your suggestion is something like:

interface BPointerProxy {
  long getAddress();
  void setAddress(long address);

  B getB(); // How is this implemented ?
}

interface A {
  BPointerProxy getB();
  void setB(BPointerProxy proxy);

  int getX();
  void setX(int x);
}

With this model, BPointerProxy will be a Values interface and since it is code generated, BPointerProxy.getB() will still have to be a default method ?

Just for contrast, my current solution is as follows: Note that it suffers from allocation within getB() on every invocation; allocation that wouldn't be needed under native pointer support.

interface A { 
  long getBAddress();
  void setBAddress(long bAddress);

  long getX();
  void setX(long x);

  default B getB() {
    // object allocation :( uncacheable in the current object since this is a default method.
    B b = Values.newNativeReference(B.class);
    BytesStore bs = BytesStore.nativePointer();
    bs.set(getBAddress(), b.maxSize());
    b.bytesStore(bs, 0, b.maxSize());
    return b;
  }
} 

Personally, I don't see a no-allocation approach to "getB()" without "wrapping" (like ACompanion in my previous post). If you do, I will appreciate if you can share your approach. Thanks.

[peter-lawrey]

What we have done for arrays is have a nest object retained which implements the nested interface for use when you call get at index. That's how I would do here. On 5 Jul 2016 22:22, "vemana" notifications@github.com wrote:

you want a field of another type which is actually a pointer and for Chronicle-Values to understand that rather than return a long address.

Yes, underlying storage is pointer, but the Java interface is a Byteable object at that pointer.

I understand the proxy comment now. I think your suggestion is something like:

interface BPointerProxy { long getAddress(); void setAddress(long address);

B getB(); // How is this implemented ? }

interface A { BPointerProxy getB(); void setB(BPointerProxy proxy);

int getX(); void setX(int x); }

With this model, BPointerProxy will be a Values interface and since it is code generated, BPointerProxy.getB() will still have to be a default method ?

Just for contrast, my current solution is as follows: Note that it suffers from allocation within getB() on every invocation; allocation that wouldn't be needed under native pointer support.

interface A { long getBAddress(); void setBAddress(long bAddress);

long getX(); void setX(long x);

default B getB() { // object allocation :( uncacheable in the current object since this is a default method. B b = Values.newNativeReference(B.class); BytesStore bs = BytesStore.nativePointer(); bs.set(getBAddress(), b.maxSize()); b.bytesStore(bs, 0, b.maxSize()); return b; } }

Personally, I don't see a no-allocation approach to "getB()" without "wrapping" (like ACompanion in my previous post). If you do, I will appreciate if you can share your approach. Thanks.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/OpenHFT/Chronicle-Values/issues/11#issuecomment-230607028, or mute the thread https://github.com/notifications/unsubscribe/ABBU8TzuumhdTAUKeMK5ppZD-oXvH6BUks5qSsr8gaJpZM4JEp4q .

[leventov]

@vemana I think your idea makes sense. Thanks for thorough explaination and some examples of how it could be implemented. I'm going to try to implement this

[vemana]

@leventov Thanks for taking it up. Much appreciated.

In your design, please also consider pre-allocating memory so that all of the struct's data including pointed data is in one contiguous memory block.

For example,

interface A {
  int getA1();
  void setA1(int from);

  int getA2();
  void setA2(int from);
  ..........
  int getAn();
  void setAn(int from);

  @Pointer
  X getX();
  void setX(X from);

  @Pointer
  Y getY();
  void setY(Y from);
}

interface X {
  int getX1();
  void setX1(int from);

  int getX2();
  void setX2(int from);
  .....

  @Pointer // 2nd level pointer from A's point of view.
  Z getZ();
  void setZ(Z from);
}

Then

Values.newNativeReferenceForDeepCopy(A.class);

can recursively pre-allocate memory past the end of A instead of allocating at deep copy time,

aaaaaaaaaaaaaaaaaaaaaaaaa | xxxxxxxxxxxxZZZZZZ | yyyyyyyyyyyyyyyyyy
^ a1 a2 .... an go here ^ |  x1 x2 ... Z here  |  y1 y2 ... here

With this scheme, deep-copies translate to a single memcpy and improved data-locality could help processor cache performance..

Several details have been left unsaid, notably handling pointers to unbounded data and cyclic struct definitions. In practice such cases seem rather rare and Values has already established a practical precedent by requiring a max size on every String and array.

[leventov]

Hi @vemana, I've added a draft support in https://github.com/OpenHFT/Chronicle-Values/commit/abaf941725198fc99b62b9c6e704c495dc688bba

Usage:

interface A {
  B getB();
  void setB(@Pointer B b);
}

Principles:

• Pointer field is a "primitive long field" storing an address or 0 (even in heap objects!)
• When it stores some address, an B$$Native, internally cached per A$$Heap or A$$Native instance, is returned from getB() and similar methods, pointing to that address. When it stores 0, null is returned.
• To set it to 0, call setB(null), to set it to some address, call setB(b), where b is an instance of B$$Native pointing to the desired address.
• equals(), hashCode() rely on the address stored, not pointed contents
• toString() prints contents of the pointed structure (exactly as non-pointer field), or null

All the above is pretty consistent, but logic of readMarshallable() and writeMarshallable() is quite odd (I didn't found a better option):

• writeMarshallable() stores contents of the pointed structure, or null, if the pointer address is 0.
• a.readMarshallable(bytes) reads the serialized pointer field:
  • if it is something non-null, it checks if a.getB() already points to some non-zero address: if it is the case, it rewrites contents of the already pointed structure with the serialized contents; if it is 0, throws IllegalStateException.
  • if it is null, it simply sets a.setB(null).

Please share your thoughts.

This is not in the released version, to try it out you should build Chronicle Values from source, or use shapshots repository: https://oss.sonatype.org/content/repositories/snapshots/net/openhft/chronicle-values/1.5.4-SNAPSHOT/

[vemana]

leventov@: Just noticed this update from you. Many thanks. Will get back to you as soon as I get a chance to try it out.

[RobAustin]

closing as won't fix as this goes back to 2016