Scala Native bindings

[david-bouyssie]

I would like to work on Scala Native bindings of your C library. Could you please tell me which part of the API should be useful to be exposed for external use? Thanks.

[shuckc]

The library as it stands works with Q/KDB+ (from Kx), with a corresponding type 'K' used to hold objects to/from the user. This needs to be changed for a more generic macro CHRONICLE_TYPE or something, so that the individual language bindings can define it (python uses PyObject, for Java bindings, jni.h provides jobject etc.).

There are some extra wrapper functions that need pulling out as well, I hope to get around to this in a week or two as I now need v8 bindings for node.js for this.

[david-bouyssie]

Ok thank your for the clarification. In scala native I can use C types directly. So it won't be an issue. But having a more generic type would be better indeed. Thus I will wait for this new version before starting to work on the SN bindings. If you could keep me updated here it would be perfect. Thanks.

[david-bouyssie]

Hi Chris! Any update on this side? Do you have maybe an intermediate version I could use as a starting point? Thanks, David

[david-bouyssie]

Finally I decided to create a fork of your C code directly in Scala Native. I may need some advices because some things are not clear to me.

[david-bouyssie]

One question about this line of code in shmipc.c:

if (g->gl_pathc < 1) {
        return krr("no queue files - java run?");
    }

If I understand correctly libchronicle cannot create its own queue files? I have to execute a Java program first?

[shuckc]

Almost, that comment is historical - shmipc.c won't create/handle an empty queue directory, it hopes to find one queuefile existing to confirm the rollover settings/schedule. Once we are initialised, it will create new queue files as required, see https://github.com/TeaEngineering/libchronicle/blob/master/native/shmipc.c#L935 for the code that rolls the queuefile.

However there is a workaround in the kdb bindings that dummies up a sample queuefile and directory-listing.cq4t if it does not already exist, which is sufficient (although not pretty) to allow it to start up without help from Java. https://github.com/TeaEngineering/libchronicle/blob/master/bindings/kdb/shm.q#L16

Obviously that is pretty hacky just storing the bytes in code, but good enough for the trials/benchmarking I conducted. The plan is to rewrite that supporting code into c - this needs support in wire.h for writing the self-describing control structures, at the moment it only has read support.

[david-bouyssie]

Ok thank you for the update. I made a full rewrite of libchronicle in Scala Native. If you want I can try to make also a prototype for this missing initializer. Could you please give me some hints of specification or maybe give me a link to an existing implementation source file? Maybe you know already the Chronicle-Queue source files doing this job? Once this prototype is working I think it would be easy to backport to C.

[shuckc]

The wire format is a bit verbose upstream, it's done by the appender.writeDocument() Java code that invokes functions in chronicle-wire-1.7.5.jar. Writing support is probably easy enough from the read support I already have. There is some documentation upstream showing the decoded data.

The 'rolled' queuefiles created by queuefile_init are currently missing headers/index2index pages, there are TODO: markers showing where the wire data needs to be written. The configuration bits are already stored in the queue configuration, so the values are available to write.

[david-bouyssie]

I'm not sure to understand. Are you talking about the first directory initialization or the creation of new queue files here?

[david-bouyssie]

And do you think this implementation is a good reference or is too old? https://github.com/jh0x/cornelich

[shuckc]

That library is for a much (much) earlier chronicle-queue format, that pre-dates the self-describing wire format (and many other changes) so it's not that useful as a reference. It's also cpp and I needed c bindings.

[david-bouyssie]

Ok thanks. And regarding the Chronicle construction (https://github.com/OpenHFT/Chronicle-Queue#chronicle-construction) if I understood correctly you have a KDB example somewhere?

[shuckc]

Sure if you have a kdb environment the three demos are: send timestamp, print timestamps shm-sender.q and shm-recv.q

shm-recv-once.q - guaranteed/deliver once pattern

IPC client/server without using TCP shm-ipc-client.q and shm-ipc-server.q

You can also look at the command line c utility that gets built shmmain.c which can read and write without using and 3rd party dependancies (it builds with mock_k.h to stub out). This is the tool that the fuzzer and memory checker Makefile targets run against.

[david-bouyssie]

Thanks a lot!

Sure if you have a kdb environment I don't but any additional source code is useful :)

I managed to create a queue using CQ V4 and to do the shmipc_init in my Scala Native code.

If I understood libchronicle is not compatible with V5, isn't it?

[shuckc]

libchronicle I would say has good compatibility with OpenHFT v4, subject to some of the issues we have discussed.

The changes in v5 are not that major, although I'm not keen on the direction it's heading. I agree appenders should handle maintaining index pages, and tailers should be read only (I do exactly that when I setup the memory protection flags on the mmaps). The new central lockfile queuelock.cq4t seems unnecessary, and the various issues they claim it resolves could have been done nicely within the v4 scheme. It's a bit hard to tell through as I suspect quite a lot of the bug reporting is done behind closed doors so we might not see exactly why they added it.

Either way, I think they are reasonably backwards compatible with their own formats, so an OpenHFT Java v5 client should have no problem reading from a libchronicle queue appender.

[david-bouyssie]

Thank you again for your quick reply.

I had good progress today since the shmipc_init ported to Scala Native seems to work. I'm now debugging the call to shmipc_tailer.

If you are interested I can share the code somewhere.

BTW, SN folks will soon add interoperability with C. This means that in the near feature the code I'm working on now could also be bind to Python. Note that it's already possible to call Python from SN: https://github.com/lolgab/snipy

The only major issue of SN is the current lack of multi-threading support, but it seems it's not an issue here.

[david-bouyssie]

It seems I have an issue with the shmipc_peek code: shmipc: missing EOF for queuefile (cycle < highest_cycle-patch_cycles), bumping next_index from 2 to 4294967296

Maybe something is wrong in my code. Time for debugging...

[shuckc]

Hi there - in recent builds The external parts of the library are those exposed by libchronicle.h which is now quite a clean interface compared to early revisions. I will close this issue.