search

eclipse-rdf4j / rdf4j

Eclipse RDF4J: scalable RDF for Java
https://rdf4j.org/
BSD 3-Clause "New" or "Revised" License
367 stars 164 forks source link

LMDB Support in RDF4J Server #3931

Closed Kvarnefalk closed 2 years ago

Kvarnefalk commented 2 years ago

Problem description

Before I start, thanks for such an awesome project.

I'm setting up an RDF4J server but struggle with the lwjgl-natives in the RDF4J server. I can patch it locally by adding the required native dependencies to tools/workbench.pom.xml and tools/server/pom.xml. After repackaging the server with that I'm able to create and query the lmdb database.

Preferred solution

Being able to access lmdb databases from RDF4J server and Workbench

Are you interested in contributing a solution yourself?

Perhaps?

Alternatives you've considered

This is the patch I ran applied locally. Probably have to extend it to include all natives. But maybe just introducing dependencies like this is not prefferred?

diff --git a/tools/server/pom.xml b/tools/server/pom.xml
index 0755b4cf9a..239bc2f17a 100644
--- a/tools/server/pom.xml
+++ b/tools/server/pom.xml
@@ -12,8 +12,39 @@
    <description>HTTP server implementing a REST-style protocol</description>
    <properties>
        <testserver.webapp.dir>${project.build.directory}/rdf4j-server</testserver.webapp.dir>
+       <lwjgl.version>3.3.1</lwjgl.version>
+       <lwjgl.natives>natives-linux</lwjgl.natives>
    </properties>
+   <dependencyManagement>
+       <dependencies>
+           <dependency>
+               <groupId>org.lwjgl</groupId>
+               <artifactId>lwjgl-bom</artifactId>
+               <version>${lwjgl.version}</version>
+               <scope>import</scope>
+               <type>pom</type>
+           </dependency>
+       </dependencies>
+   </dependencyManagement>
    <dependencies>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl</artifactId>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl-lmdb</artifactId>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl</artifactId>
+           <classifier>${lwjgl.natives}</classifier>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl-lmdb</artifactId>
+           <classifier>${lwjgl.natives}</classifier>
+       </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>rdf4j-http-server-spring</artifactId>
diff --git a/tools/workbench/pom.xml b/tools/workbench/pom.xml
index 64fbaf51c9..36e06926b8 100644
--- a/tools/workbench/pom.xml
+++ b/tools/workbench/pom.xml
@@ -10,7 +10,40 @@
    <packaging>war</packaging>
    <name>RDF4J: Workbench</name>
    <description>Workbench to interact with RDF4J servers.</description>
+   <properties>
+       <lwjgl.version>3.3.1</lwjgl.version>
+       <lwjgl.natives>natives-linux</lwjgl.natives>
+   </properties>
+   <dependencyManagement>
+       <dependencies>
+           <dependency>
+               <groupId>org.lwjgl</groupId>
+               <artifactId>lwjgl-bom</artifactId>
+               <version>${lwjgl.version}</version>
+               <scope>import</scope>
+               <type>pom</type>
+           </dependency>
+       </dependencies>
+   </dependencyManagement>
    <dependencies>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl</artifactId>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl-lmdb</artifactId>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl</artifactId>
+           <classifier>${lwjgl.natives}</classifier>
+       </dependency>
+       <dependency>
+           <groupId>org.lwjgl</groupId>
+           <artifactId>lwjgl-lmdb</artifactId>
+           <classifier>${lwjgl.natives}</classifier>
+       </dependency>
        <dependency>
            <groupId>${project.groupId}</groupId>
            <artifactId>rdf4j-client</artifactId>

Anything else?

No response

kenwenzel commented 2 years ago

@jeenbroekstra I think we've excluded the native binaries for LWJGL due to possible licensing issues according to Eclipse policies?

I also had to make similar changes to run the server for testing purposes.

abrokenjester commented 2 years ago

@kenwenzel yes that's correct. However doesn't mean we can't revisit that. Eclipse policy has been revised recently to be more flexible. We can at least put these up for review. I'll take another look.

Kvarnefalk commented 2 years ago

Alright that sounds great. I'll sit tight meanwhile. Thanks for the swift response.

abrokenjester commented 2 years ago

The LMDB code itself is covered under the "OpenLDAP Public License" (https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/LICENSE) - and this is also what the LWJGL native extensions are saying (see for example https://github.com/LWJGL/lwjgl3/blob/master/modules/lwjgl/lmdb/src/main/c/mdb.c ). Not a license I'd heard of before but it's OSI-approved at least: https://opensource.org/licenses/OLDAP-2.8 .

I'll check if this license is also accepted as-is by the EF. And if I understand the terms of the license correctly, we'll need to include a verbatim copy of the license somehow - probably at least by adding it into our about.md file.

kenwenzel commented 2 years ago

Another option could be to split the server into modules and deploy one additional WAR file with the LMDB libraries. We usually use OSGi and Karaf for modular applications. I don't know if it is even possible to have two WAR files with interdependencies.

abrokenjester commented 2 years ago

Another option could be to split the server into modules and deploy one additional WAR file with the LMDB libraries. We usually use OSGi and Karaf for modular applications. I don't know if it is even possible to have two WAR files with interdependencies.

I was actually thinking to just include the native extensions as regular runtime dependencies. That way they automatically get included in the WAR. I don't think they're particularly massive, are they?

abrokenjester commented 2 years ago

AFAICT we should be fine to include these. I'll put up a PR.

kenwenzel commented 2 years ago

Another option could be to split the server into modules and deploy one additional WAR file with the LMDB libraries. We usually use OSGi and Karaf for modular applications. I don't know if it is even possible to have two WAR files with interdependencies.

I was actually thinking to just include the native extensions as regular runtime dependencies. That way they automatically get included in the WAR. I don't think they're particularly massive, are they?

All native libraries (LWJGL+LMDB) are in sum about 600 kb. Maybe we could save this space at a later point if we replace MapDB with LMDB.