Consider putting Windows version in Maven Central

[ccleve]

If you put it there, I'll test it right away...

[krisskross]

No problem. I'll deploy it later today.

[krisskross]

Maybe not tonight. My old windows installation was 32 bit so I need to re-install the 64 bit version.

[ccleve]

Anything I can do to help? I'm on win64. I cloned the project locally and tried to do a build. Fixed some version errors, but don't have Visual Studio installed, so it didn't work.

[krisskross]

I'm installing Visual Studio right now. First time using it, but hopefully I'll manage. I'll let you know if I bump into problems. Thanks.

[krisskross]

Took hours to install Visual Studio on my vm :-( and vcbuild still cannot be found. Seems like vcbuild has been replaced in later versions of Visual Studio but it might be possible to get it back by install Windows SDK 7.

Here we go again...

[ccleve]

vcbuild is now msbuild. http://stackoverflow.com/questions/7761395/where-is-vcbuild-in-vs2010

[ccleve]

Can you configure TravisCI to do the build, and then just download the result? (Of course, that may make the Maven deploy-to-central difficult...) But at least you could test it.

On 1/9/2015 1:40 PM, Kristoffer Sjögren wrote:

Took hours to install Visual Studio on my vm :-( and vcbuild still cannot be found. Seems like vcbuild has been replaced in later versions of Visual Studio but it might be possible to get it back by install Windows SDK 7.

Here we go again...

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69387125.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

Im doing progress with vcbuild but the vm is super slow.

Interesting. I didn't know that TravisCI have support for osx and windows. I'll read up and try it now.

[krisskross]

Hmm. It's still in Beta. And windows is not available yet as I can tell. Do you have any examples? I need to configure the build with a different profile for each platform. Not sure how to do that.

[ccleve]

I'm not sure either, but this project had to solve the same problem: https://github.com/fusesource/leveldbjni

By "different profile", you mean Maven profile?

On 1/9/2015 4:11 PM, Kristoffer Sjögren wrote:

Hmm. It's still in Beta. And windows is not available yet as I can tell. Do you have any examples? I need to configure the build with a different profile for each platform. Not sure how to do that.

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69407920.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

Yes, a different Maven profile.

Anyway, i'm confident that i'll manage to release the windows artifact. It's just that my vm is so darn slow. Still waiting for Windows SDK 7 to install...

[krisskross]

Msbuild is working now. But there are some compile error that I need to fix.

[krisskross]

Can you please try this jar?

https://www.dropbox.com/s/wfglrwtjmg5q08g/lmdbjni-win64-0.2.6.jar?dl=0

[ccleve]

I included this in my pom:

      <dependency>
         <groupId>org.deephacks.lmdbjni</groupId>
         <artifactId>lmdbjni</artifactId>
         <version>0.2.4</version>
     </dependency>

and put the jar directly on the classpath. Ran some sample code and got:

Exception in thread "main" org.fusesource.lmdbjni.LMDBException: T at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:48) at org.fusesource.lmdbjni.Env.open(Env.java:59) at org.fusesource.lmdbjni.Env.open(Env.java:51) at org.fusesource.lmdbjni.Env.open(Env.java:47) at com.dieselpoint.benchmark.BenchmarkDb.makeLmdb(BenchmarkDb.java:378)

Here's the code:

     try (Env env = new Env()) {
         Env.pushMemoryPool(1024 * 512);
         env.open(lmdbDir);   // <<< line 378
         try (Database db = env.openDatabase()) {

             Benchmark.start("start lmdbinsert");

On 1/10/2015 12:21 PM, Kristoffer Sjögren wrote:

Can you please try this jar?

https://www.dropbox.com/s/wfglrwtjmg5q08g/lmdbjni-win64-0.2.6.jar?dl=0

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69465939.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

Hmm. Can you try debug the code a look what response code is actually returned on line 55 in Env?

[krisskross]

Make sure that lmdbDir exist also.

[ccleve]

lmdbDir exists and is empty. On line 55,

     int rc = mdb_env_open(pointer(), path, flags, mode);

pointer() returns self = 457126352 path = "./temp/lmdbdir" flags=0 mode=420 rc = 0

I will test an absolute path.

On 1/10/2015 12:58 PM, Kristoffer Sjögren wrote:

Make sure that lmdbDir exist also.

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69467373.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[ccleve]

An absolute path, "/temp/lmdbdir", returns the same result.

[ccleve]

Getting some weird results. When I run the same code repeatedly, I get different errors. It's a bit random. Sometimes it gets past line 55 and generates this error: Exception in thread "main" org.fusesource.lmdbjni.LMDBException: MDB_BAD_TXN: Transaction cannot recover - it must be aborted at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:48) at org.fusesource.lmdbjni.Transaction.commit(Transaction.java:42) at org.fusesource.lmdbjni.Database.put(Database.java:127) at org.fusesource.lmdbjni.Database.put(Database.java:118) at com.dieselpoint.benchmark.BenchmarkDb.makeLmdb(BenchmarkDb.java:385)

So the failure on line 55 is intermittent. I also tested an absolute Windows path: "c:\temp\lmdbdir" and got the same result.

[krisskross]

Strange. The build passed EnvTest. But the ZeroCopyTest failed.

Not sure if the commit I did was sound. I will look at this tomorrow.

[krisskross]

I will try to build liblmdb.a from mingw and use it from lmdbjni-win64 instead.

[krisskross]

Can you try these?

https://www.dropbox.com/s/2tsx05tjw5xdhkr/lmdbjni-win64-0.2.6.jar?dl=0 https://www.dropbox.com/s/6lmw3m29gpy5nmo/lmdb-win64-0.1.2.jar?dl=0

[ccleve]

I copied the first jar into my project, ran a very simple insert and get(), and it worked, no errors. I'm not clear on what the second jar is.

Also, it's hard for me to run the unit tests in the main project because there is a variety of pom errors. Parent poms don't resolve, missing artifacts. There's also an M2E/Eclipse error which I can fix. Can you check in the latest version of the project?

On 1/11/2015 9:20 AM, Kristoffer Sjögren wrote:

Can you try these?

https://www.dropbox.com/s/2tsx05tjw5xdhkr/lmdbjni-win64-0.2.6.jar?dl=0 https://www.dropbox.com/s/6lmw3m29gpy5nmo/lmdb-win64-0.1.2.jar?dl=0

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69497874.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

Sounds promising. I wasn't sure if lmdb-win64 was required. Ill remove it as a dependency from lmdbjni-win64.

This is the latest version of the project. Maybe the errors are because profiles? Can you tell me more exactly what the problem is because I can't see the errors you describe.

I'll publish the artifacts to maven central after we resolve these errors.

[krisskross]

I have released lmdbjni-win64-0.2.6 to Maven central. It should show up in about an hour or so.

[ccleve]

Some errors:

lmdb-win64/pom.xml refers to parent pom version

0.1.2-SNAPSHOT, but the parent is 0.1.3-SNAPSHOT. Same for lmdb-osx64. Module versions should be removed so they inherit from the parent. -SNAPSHOT should go away. M2E in Eclipse reports "Plugin configuration not covered by lifecycle execution" in lmdb-linux64. Here's the issue: https://www.eclipse.org/m2e/documentation/m2e-execution-not-covered.html I'm not sure whether the goal should be marked "ignore" or "execute". Is it possible that you did the deployment to Maven Central, but haven't yet checked into github the poms with the newer version numbers? (Haven't yet seen it on Maven Central. Will keep testing with the jar you sent.) On 1/11/2015 3:26 PM, Kristoffer Sjögren wrote: > I have released lmdbjni-win64-0.2.6 to Maven central. It should show > up in about an hour or so. > > — > Reply to this email directly or view it on GitHub > https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69512285. ## Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

Fixed.

[ccleve]

A few remaining items:

It's still not on Maven Central.

It's unclear to me what's the relationship between the lmdbjni project and the lmdb project, or how the versioning works. The 0.2.6 "release" still references -SNAPSHOT versions. See the pom in lmdbjni-all, for example. pom errors are preventing me from running the unit tests. lmdbjni-win64 has a dependency on lmdb-win64 version 1.2, which is 10 days old but not on maven central and I can't do my own build.

When I run my own benchmark/test on the jar you sent, it runs fine on a small number of records, but after I insert 1600 or so records I get:

Exception in thread "main" org.fusesource.lmdbjni.LMDBException: MDB_MAP_FULL: Environment mapsize limit reached at org.fusesource.lmdbjni.Util.checkErrorCode(Util.java:48) at org.fusesource.lmdbjni.Transaction.commit(Transaction.java:42) at org.fusesource.lmdbjni.Database.put(Database.java:127) at org.fusesource.lmdbjni.Database.put(Database.java:118) at com.dieselpoint.benchmark.BenchmarkDb.makeLmdb(BenchmarkDb.java:510)

I'm not using a transaction. Must be an internally-triggered commit.

The code is pretty simple:

     try (Env env = new Env()) {
         Env.pushMemoryPool(1024 * 512);
         env.open(lmdbDir);
         try (Database db = env.openDatabase()) {

             Benchmark.start("start lmdbinsert");
             int count = 0;
             for (SmallRec rec : smallRecs) {
                 count++;
                 System.out.println(count);
                 db.put(rec.getKey().getArray(), 

rec.getValue().getArray()); } Benchmark.end("end lmdbinsert", smallRecs.size()); } Env.popMemoryPool(); }

On 1/11/2015 4:25 PM, Kristoffer Sjögren wrote:

Fixed.

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69514883.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[krisskross]

MDB_MAP_FULL sounds like a proper error code. Try make the database bigger.

Strange about the release, I received a confirmation that the promote was successful. Ill have a look later tonight.

Details:

The following artifacts have been promoted to the "Releases" [id=releases] repository

/org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.pom (SHA1: 5f640b8763dc83dde120fc864e6609397a5f8d99) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.jar.asc (SHA1: 43acdd05dc384f929af2457426cfafa9394f815f) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.pom.asc (SHA1: 455ef4b48160656d9be38d82934db6e4a1cfd5c8) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.jar (SHA1: c8137f7a1282ac6f54639e8afd6d82d5df787543)

Action performed by Kristoffer Sjögren (stoffe@gmail.com)

[ccleve]

I added a setMapSize() call and still get the MDB_MAP_FULL error.

     FileUtil.deleteDir(lmdbDir);
     new File(lmdbDir).mkdirs();

     try (Env env = new Env()) {

         // make it a multiple of 8k
         long dbSize = (1024 * 1024 * 1024) * 8 * 100; // 800gb
         env.setMapSize(dbSize);

         Env.pushMemoryPool(1024 * 512);

         env.open(lmdbDir);
         try (Database db = env.openDatabase()) {

I deleted the db before this ran, so it should have been created with the big number. I also confirmed that it fails with a smaller db size.

I also tried to get the mdb statistics with:

Object stat = env.stat();

(The MDB_stat is not visible to java.) A call to .stat() generates this error:

#

A fatal error has been detected by the Java Runtime Environment:

#

EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x000007fef6811388,

pid=5640, tid=5768 #

JRE version: Java(TM) SE Runtime Environment (8.0_20-b26) (build

1.8.0_20-b26)

Java VM: Java HotSpot(TM) 64-Bit Server VM (25.20-b23 mixed mode

windows-amd64 compressed oops)

Problematic frame:

C [lmdbjni-64-0-2397937682373036762.2+0x11388]

#

Failed to write core dump. Minidumps are not enabled by default on

client versions of Windows #

An error report file with more information is saved as:

C:\dev\diesel.test\hs_err_pid5640.log

#

If you would like to submit a bug report, please visit:

http://bugreport.sun.com/bugreport/crash.jsp

The crash happened outside the Java Virtual Machine in native code.

See problematic frame for where to report the bug.

#

If you want, I'll try to generate the core dump.

Also -- if the promote failed, it's probably because you still reference SNAPSHOT versions.

On 1/12/2015 3:07 AM, Kristoffer Sjögren wrote:

MDB_MAP_FULL sounds like a proper error code. Try make the database bigger.

Strange about the release, I received a confirmation that the promote was successful. Ill have a look later tonight.

Details:

The following artifacts have been promoted to the "Releases" [id=releases] repository

/org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.pom (SHA1: 5f640b8763dc83dde120fc864e6609397a5f8d99) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.jar.asc (SHA1: 43acdd05dc384f929af2457426cfafa9394f815f) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.pom.asc (SHA1: 455ef4b48160656d9be38d82934db6e4a1cfd5c8) /org/deephacks/lmdb/lmdb-win64/0.1.2/lmdb-win64-0.1.2.jar (SHA1: c8137f7a1282ac6f54639e8afd6d82d5df787543)

Action performed by Kristoffer Sjögren (stoffe@gmail.com mailto:stoffe@gmail.com)

— Reply to this email directly or view it on GitHub https://github.com/deephacks/lmdbjni/issues/14#issuecomment-69543129.

Chris Cleveland Dieselpoint, Inc. +1 773.528.1700 x116 +1 312.339.2677 mobile http://dieselpoint.com ccleveland@dieselpoint.com

This email and any attachments contain information from Dieselpoint, Inc. and should be considered confidential. If this email is received in error, please delete it and notify the sender.

[ccleve]

I get the same MDB_MAP_FULL error on linux.

[krisskross]

I have checked maven central and the jars are there now. Maybe it was slow due to sonatype maintenance?

I can create 10 million records using the following gist on linux.

https://gist.github.com/krisskross/85d2d398f5b89976e8cf

Maybe try skip Env.pushMemoryPool(1024 * 512);

[krisskross]

I can call env.stat() on linux which prints the following.

{ms_branch_pages=0, ms_psize=4096, ms_depth=1, ms_leaf_pages=1, ms_overflow_pages=0, ms_entries=1}

[ccleve]

The MDB_MAP_FULL error was in my code, sorry. I successfully got the win64 version from the maven central repo and ran it. All looks good.

Now my only issue is that when I env.setMapSize( 8 gb ) on windows, it allocates the entire file in advance, rather than growing it slowly. This does not happen on linux.

[krisskross]

Glad to hear that!

This is how LMDB works. My windows knowledge is very limited but it looks like LMDB uses CreateFileMapping and CreateFile when the environment is opened.

You should ask these kind of questions on openldap-technical@openldap.org.

[krisskross]

I have created a google group where we can discuss these kind things further.

https://groups.google.com/forum/#!forum/lmdbjni

I will close this issue now.