remark on replication/distribution

[RuedigerMoeller]

If you manage to define kind of in/out interfaces for replication messages, I could plug in a multicast based replication, which is favourable over TCP in many setups. (Talking of SharedHashmap)

[RobAustin]

We have been working on a similar solution over the last few weeks. We have some test cases for TCP Replication, see for example :

see net.openhft.collections.replication.TCPSocketReplicationTest3way

We have also started work on UDP replication, Although we have not yet completed testing this code : net.openhft.collections.replication.UDPSocketReplicationTest#testBufferOverflow

We have just check this code into:

<groupId>net.openhft</groupId>
<artifactId>collections</artifactId>
<version>3.1.1-SNAPSHOT</version>

So it should be available for download as a snapshot, in the next few hours.

Rob

On 27 May 2014, at 17:46, RuedigerMoeller notifications@github.com wrote:

If you manage to define kind of in/out interfaces for replication messages, I could plug in a multicast based replication, which is favourable over TCP in many setups.

— Reply to this email directly or view it on GitHub.

[Cotton-Ben]

Hi all,

I think the "distribution" part of Ruediger's title "remark on distribution/replication" deserves some content too!

Any comment wrt to OpenHFT ambitions to provide an explicit DistributedSharedHashMap interface? Where in fact the "Map<K,V>" set could be partitioned (instead of replicated) and synchronized across host separated compute nodes ... empowering PB+ capacity Maps?

Just now getting back into my "what's new at OpenHFT?" mode, I see a whole lot.

• SHM Listeners
• SHM Replicators
• TCP accommodations
• UDP accommodations
• IPC accommodations

Being blunt, if you then add DistributedSHM as operand view, OpenHFT is just a few details (on the operator side) from itself being a (highly preferred) JCACHE/JGRID provider.

[leventov]

@Cotton-Ben isn't distribution is Hazelcast's field?

[Cotton-Ben]

Hazelcast does provided a distributed view of my K,V data set ... but ... they use only {UDP,TCP} as a transport set ... and for our supercomputer grid (distributed to only locality=127.0.0.1, no NIC hops) the use of only {UDP,TCP} as transport is a delinquency (we ain't going to dance up and down the OSI stack if there is no NIC hop). OpenHFT solves this because we can use /dev/shm as an IPC transport. So, I'm still saying, given that y'all already solve our locality=localhost transport needs ... we'd like to audition you as a potential distributed operand provider ... empowering us to kill 2 birds with 1 OpenHFT stone.

[RuedigerMoeller]

@Rob Just my 2 pence: A reliable UDP low latency messaging is quite some effort (including retransmission on packet loss, message recording, total ordering for transparent failover, high avaiability ..). Additionally: large systems (and these are the ones requiring multicast-udp and off-heap storage) most of the time already have a udp-message-bus system. E.g. my distributed datagrid cluster @ Eurex runs on a commercial low latency udp messaging solution in production, fast-cast 1.0 (slow, but small footprint) in development/functional test, fast-cast 2.0 is considered for other, smaller exchange systems to avoid license fees. So imo best thing would be to have kind of a public transport interface defined. Ofc you still can provide your own implementations.

@Cotton-Ben I have built a distributed, continuous query datagrid running in production @ Eurex Exchange featuring evenly sharded data across all nodes. From my POV, OpenHFT shouldn't mud up layers. Providing IPC Off-Heap storage is one thing. Implementing replication another thing. Sharding data amongst distributed instances is then on top.

I currently evaluate to use SharedMemHashMap as a base storage foundation for offheap data (on heap data is still used for low latency market data distribution). So in my use case, messaging, sharding, streaming, querying, high avaiability is already in place.

In general an open source, high efficient distributed continuous datagrid ofc would be a great product as the performance of redis/mongo & co seems not-that good compared to the what is possible. Also they don't feature continuous queries, which is a must-have feature to scale large realtime systems. Funny enough I am currently also trending towards something like this with my os stuff :-) [will take years, I am currently stuck serializing java<=>JS<=>Dart its really ugly, but JSon does not cut it] The closed source stuff I did the last years turned out to be amazing (not because of super efficient implementation, but its very easy to extend and build stuff on top). Continuous distributed datagrids can act as a large-scale model to back GUI clients, backoffice systems and realtime computation. Its MVC at large scale.

[RuedigerMoeller]

@leventov only if you are willing to replace 'Million' by 1000.

[Cotton-Ben]

@RuedigerMoeller interesting ... nice job with EurexExchange delivery. I see your point it is a lot of work. For me, I face a race-condition ... from which whoever wins I will be happy.

Racer 1= some JCACHE/JGRID vendor adopting OpenHFT as their Off-Heap ZERO-COPY /dev/shm Java IPC transport provider

Racer 2=OpenHFT passing the 107/347 TCKs and itself becoming a JCACHE/JGRID provider

[RobAustin]

@RuedigerMoeller ( also posted here as we appear to have 2 threads going on )

you may find this interface useful

net.openhft.collections.ReplicatedSharedHashMap.EntryExternalizable its implemented by :

net.openhft.collections.VanillaSharedReplicatedHashMap

[RuedigerMoeller]

@Cotton-Ben seems i need some googling ;) @BoundedBuffer Thanks for the hint, I'll cose this one.