search

trevorr / circe

Circe: A High-Performance Hash Algorithm Framework & Library
Apache License 2.0
12 stars 1 forks source link

Packaging and Maven-Repo artifact deployment #2

Open rdhabalia opened 8 years ago

rdhabalia commented 8 years ago

hi,

Thanks for providing java solution for crc32c-sse4.2 using native library. I can see you have uploaded SNAPSHOT version of this artifact at : Sonatype-Staging_maven-repo

However, we are not able to use this solution due to following issues:

  1. Library not present into maven-repo: The available jni library in maven-repo is only available for windows OS and we can't use it for "Mac OS" or "Linux OS".

Solutions: 1. Upload jni library to maven repo for all well-known OS: Right now, Maven repo only contains jni for windows so other system can't use this library. So, can you also please upload library for MacOSX and Linux as well.

2. One Packaging of all OS specific JNI library using vagrant: Checkout RocksDB Build it uses vagrant to build jni for all OS environments and packages them into one artifact which you can deploy to maven repo

  1. Can't use artifact as it as: When we build circe-crc32c-sse42 using maven build:
    • It internally generates "jni" library based on OS where we build and it packages this generated jni library under .nar artifact. e.g: circe-crc32c-sse42.nar
    • Nar Plugin goal also generates NarSystem.java which tries to load this jni library at runtime using System.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT"); which fails obviously because it asks ClassLoader to load library from java.library.path or sun.boot.library.path. And as this jni library is packaged under .nar so, definitely it is not present in either library.path and hence, initialization of the interface which provides API to compute checksum Sse42Crc32C.java fails.

Solution

trevorr commented 8 years ago

Hi Raj,

I appreciate your interest in Circe.

The snapshot release you mention was never intended to be used by anyone. It was a test of the release process. The reason there is no official release is the difficulty you describe: there was not an easy way to include native libraries for all platforms in the release. While I think it can be manually done, I was looking for a better way when my schedule filled up with other projects. This Maven NAR thread describes where I ended up: https://groups.google.com/d/msg/maven-nar/55381nAYeLw/nVYq0SbnnCgJ

I can attempt to perform your first solution to get an initial release available. However, since it will take some time to re-familiarize myself with the project and configure build environments, it may be a few weeks before I can complete this, given my other obligations. Also, since any builds I create would be on recent OS versions, there would be no guarantee they would work on older versions. For instance, I often hear of people running into C library incompatibility when deploying code built with recent Ubuntu onto older RHEL/Centos instances. Of course, the simplicity of this native library should make that less likely.

I don't really understand the cause of your .nar vs .jar issue. I'd need to be more convinced that the approach advocated by the Nar plugin (i.e. loading .nar artifacts using NarSystem) is fundamentally flawed to consider replacing it. Also, your pull request has some minor issues, such as using tabs instead of spaces in new files and adding a redundant test case that will fail on any machine not reporting support for SSE 4.2 (either due to lack of processor, or more commonly, VM support). There is already a test case for SSE 4.2 CRC-32C that only runs if support is detected.

I'm still not sure what the real solution is going forward, as any build process that doesn't allow any developer to easily build for all platforms doesn't seem viable. The RocksDB build is an interesting data point, but it doesn't support Windows. Because the native library is so simple and free of platform dependencies, it's tempting to try cross-compilation using only Linux. Cross-compiling for OS X used to be problematic, but this project looks promising: https://github.com/tpoechtrager/osxcross. The remaining question would be how well Nar works with cross-compilation.

rdhabalia commented 8 years ago

Hi Trevor,

Thank you for your prompt response.

  1. I don't really understand the cause of your .nar vs .jar issue...(i.e. loading .nar artifacts using NarSystem) is fundamentally flawed to consider replacing it...adding a redundant test case..

I have added test-case to explain issue: how NarSystem creates issue while using circe-crc32c-sse42 by a different project and I also explained similar problem it in earlier thread as well.

System.loadLibrary("circe-crc32c-sse42-0.1-SNAPSHOT");

So, to resolve this issue:

Therefore, I created a pull request: which creates a jar and it auto loads library and user don't have to pass -Djava.library.path argument to access the utility. And test-case was essentially part of different artifact which shows how a separate artifact can use utility without performing any additional steps.

change in pull request

Request

Thanks for your work.