Java SDK for vineyard

[sighingnow]

Describe your problem

Overall design

• vineyard-core: core client implementation
  • client: don't binding to libvineyard_client.so, implements the client natively instead
    • benefits:
      • decoupled design
      • less build & distribute burden for dependent shared libraries.
• modules: vineyard-basic, vineyard-graph
• ds: arrow-based for off-heap data access, and, enables inter-operabiltity with generated FFI implementation.

Object resolver & builder

0. data structure

   • arrow based, to leverage off-heap data

1. Resolver: a global registery:

   • XML configration + dynamic injection: like spring or something like that
     • schema based
     • hierarchical
     • support register by
       • xml configuration (in multiple package, at multiple places, refer to spring, or log4j)
       • decorator
       • java code
       • something like https://docs.spring.io/spring-javaconfig/docs/1.0.0.m3/reference/html/creating-bean-definitions.html
         • https://www.jianshu.com/p/82043c64a8eb
         • https://www.jianshu.com/p/97d1c967e4b9
     • register entry:
       • typename, e.g., vineyard::Tensor
       • classname: e.g.,io.v6d.basic.Tensor`
       • (optional) package id: io.v6d.core:vineyard-core:[version], we can use URLClassLoader to download the jar at runtime.

2. Builder: like what we have in C++/Python, should be trivial.

3. Codegen

Package layout:

• [ ] Low level API for IPC/RPC client, blobs and metadata.

  • [ ] client/
    • [ ] Client: an interface class
    • [ ] IPCClient: with junixsocket to send/recv fd, and memory sharing
    • [ ] RPCClient
    • [ ] ds/
      • [ ] Object
      • [ ] ObjectMeta
      • [ ] resolvers/
      • a global registry
      • resolution: typename -> actual java class, may involve class loader as well
      • tools for reflect from arrow-java to actual ds
      • [ ] builders/
      • trivial
  • [ ] common/
    • [ ] protocol: implements the protocol.cc
    • [ ] objectid: ...
    • [ ] status, and error diagnostics

• [ ] High level API for common data structures (arrays, recordbatch, table, tensor, dataframe and graphs as well).

  • vector
  • arrow-based data structures
    • primitive array
    • recordbatch
    • table
  • tensor
  • dataframe

• [ ] connect to bigdata engines in apache

  • spark

Additional context

Part of #171.

[sighingnow]

See also #336.