Open rdhabalia opened 8 years ago
Hi Raj,
I appreciate your interest in Circe.
The snapshot release you mention was never intended to be used by anyone. It was a test of the release process. The reason there is no official release is the difficulty you describe: there was not an easy way to include native libraries for all platforms in the release. While I think it can be manually done, I was looking for a better way when my schedule filled up with other projects. This Maven NAR thread describes where I ended up: https://groups.google.com/d/msg/maven-nar/55381nAYeLw/nVYq0SbnnCgJ
I can attempt to perform your first solution to get an initial release available. However, since it will take some time to re-familiarize myself with the project and configure build environments, it may be a few weeks before I can complete this, given my other obligations. Also, since any builds I create would be on recent OS versions, there would be no guarantee they would work on older versions. For instance, I often hear of people running into C library incompatibility when deploying code built with recent Ubuntu onto older RHEL/Centos instances. Of course, the simplicity of this native library should make that less likely.
I don't really understand the cause of your .nar vs .jar issue. I'd need to be more convinced that the approach advocated by the Nar plugin (i.e. loading .nar artifacts using NarSystem) is fundamentally flawed to consider replacing it. Also, your pull request has some minor issues, such as using tabs instead of spaces in new files and adding a redundant test case that will fail on any machine not reporting support for SSE 4.2 (either due to lack of processor, or more commonly, VM support). There is already a test case for SSE 4.2 CRC-32C that only runs if support is detected.
I'm still not sure what the real solution is going forward, as any build process that doesn't allow any developer to easily build for all platforms doesn't seem viable. The RocksDB build is an interesting data point, but it doesn't support Windows. Because the native library is so simple and free of platform dependencies, it's tempting to try cross-compilation using only Linux. Cross-compiling for OS X used to be problematic, but this project looks promising: https://github.com/tpoechtrager/osxcross. The remaining question would be how well Nar works with cross-compilation.
Hi Trevor,
Thank you for your prompt response.
- I don't really understand the cause of your .nar vs .jar issue...(i.e. loading .nar artifacts using NarSystem) is fundamentally flawed to consider replacing it...adding a redundant test case..
I have added test-case to explain issue: how NarSystem
creates issue while using circe-crc32c-sse42
by a different project and I also explained similar problem it in earlier thread as well.
circe-crc32c-sse42
it generates .nar
artifact which contains generated jni
library in it.Sse42Crc32C.java
, NarSystem.java
tries to load generated jni circe-crc32c-sse42-0.1-SNAPSHOT
using following code which you can see in NarSystem.javaSystem.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT");
ClassLoader
to load circe-crc32c-sse42-0.1-SNAPSHOT
library from java.library.path
or sun.boot.library.path
. And of course as this jni is a part of nar package and not present into either library.path, NarSystem class initialization will fail to load it and we can't use Sse42Crc32C.java
So, to resolve this issue:
nar-plugin
and need to pass absolute-path of the directory where nar plugin will unpack the library as a jvm argument: -Djava.library.path=/yh/git/crc/my-dir/nar/circe-crc32c-sse42-0.1-SNAPSHOT-x86_64-MacOSX-gpp-jni/lib/x86_64-MacOSX-gpp/jni
NarSystem
can load it correctly and one can use this utility.Therefore, I created a pull request: which creates a jar and it auto loads library and user don't have to pass -Djava.library.path
argument to access the utility. And test-case was essentially part of different artifact which shows how a separate artifact can use utility without performing any additional steps.
change in pull request
Request
jar
stuff on top of your utility to use it without performing any additional steps. So, if we can get jni
for various OS: uploaded in maven-repo then we can use it directly.Thanks for your work.
hi,
Thanks for providing java solution for crc32c-sse4.2 using native library. I can see you have uploaded SNAPSHOT version of this artifact at : Sonatype-Staging_maven-repo
However, we are not able to use this solution due to following issues:
Solutions: 1. Upload jni library to maven repo for all well-known OS: Right now, Maven repo only contains
jni
forwindows
so other system can't use this library. So, can you also please upload library forMacOSX
andLinux
as well.2. One Packaging of all OS specific JNI library using vagrant: Checkout RocksDB Build it uses
vagrant
to buildjni
for all OS environments and packages them into one artifact which you can deploy tomaven repo
jni
library under.nar
artifact. e.g:circe-crc32c-sse42.nar
NarSystem.java
which tries to load thisjni
library at runtime usingSystem.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT");
which fails obviously because it asksClassLoader
to load library fromjava.library.path
orsun.boot.library.path
. And as thisjni
library is packaged under.nar
so, definitely it is not present in eitherlibrary.path
and hence, initialization of the interface which provides API to compute checksum Sse42Crc32C.java fails.Solution
circe-crc32c-sse42.jar
insteadcirce-crc32c-sse42.nar
artifact and internally loads generatedjni
library without messing up withlibrary.path
and user can use this artifact without performing any additional magic. I have also added a test case which cleanly usesSse42Crc32C.java
to compute checksum by simply adding dependency of artifactcom.scurrilous:circe-crc32c-sse42:${project.version}
and without worrying about configuringlibrary.path
.