Problem with binaries in the latest nuget

[mr-miles]

Hi

I have been using 0.9.7 successfully in a project for a few months. In my setup, there are two processes, basically one calculates and stores data into an lmdb environment, and the other process reads from it and does some read-only work.

This has all worked fine, but I just upgraded from 0.9.7 to 0.9.8 and I'm seeing strange issues:

• the two processes seem unable to open the environment at the same time;
• with 0.9.7 the files started small and grew as they were written, but with 0.9.8 they are fully allocated to the map size.
• opening the environment in one process seems to fail oddly - for example if there are hyphens in the path it fails, remove them and it works

In all cases, replacing with the binary in 0.9.7 makes it all work again

Just wondering if anyone has spotted any issues/changes with the new package that need attention. I'll try and make a more solid repro for the problems.

Thanks,

Miles

[mr-miles]

I'm able to reproduce the file growing issue consistently in my environment:

Code:

        var path = @"C:\temp\tester-098";
        using (var env = new LightningEnvironment(path, new EnvironmentConfiguration { MaxDatabases = 2, MapSize = 2L*1024*1024*1024}))
        {
            env.Open();
            using (var txn = env.BeginTransaction())
            using (var db = txn.OpenDatabase(configuration: new DatabaseConfiguration {Flags = DatabaseOpenFlags.Create}))
            {
                // write a key
                using (var cur = txn.CreateCursor(db))
                {
                    var k = new byte[] {32};
                    cur.Put(k, k, CursorPutOptions.None);
                }

                txn.Commit();
            }
        }

Against 0.9.8, my 2gb map is allocated on open; in 0.9.7 it auto-grows.

Are there any records from the build output for the binary still around?

(The other 2 issues appear to be down to the level of free space on the drive being below the initial map size intermittently, and that size being specified in a coupe of places. Apologies for the noise)

[jakoss]

After update i cannot open my database (i get exception `Failed to open environment at path..."). I wondered what can be the cause, and now i realized that my MapSize is 2TB. So if you're correct - it cannot open due to problem with free space on the drive.

EDIT: Manually setting MapSize to 200GB worked - file was resized to 200 GB and it's working properly. But still - unable to use file growing is serious regression

[CoreyKaylor]

Yes, in the last release I built from the latest released version (Dec 2016) of the upstream lmdb native lib. This did not incorporate the change yet and I didn't realize that. I'll released another version this weekend from a more recent build.

[JordanZaerr]

I rebuilt the older 0.9.17 lmdb source on my machine for a sanity check and it initializes the map size up front as well. Since that's how lmdb works on file systems that don't support sparse files I'm going to assume that behavior is controlled by a build flag somewhere. So I just need to turn that on. The only problem is figuring out which one. I don't really have time to dig through the 10K lines of source right now.

[CoreyKaylor]

In the makefile it's commented out in the more recent git history if you're curious. It's pretty easy to build these binaries on the windows 10 subsystem for linux FWIW.

https://github.com/LMDB/lmdb/blob/mdb.master/libraries/liblmdb/Makefile#L27

[mr-miles]

I don't know if this helps, but here's the original patch on the mailing list for the sparse file support:

http://www.openldap.org/lists/openldap-devel/201503/msg00012.html

Maybe it at least helps to find out where to look?

Thanks for your help

On Tue, May 16, 2017 at 4:40 PM, Jordan Zaerr notifications@github.com wrote:

I rebuilt the older 0.9.17 lmdb source on my machine for a sanity check and it initializes the map size up front as well. Since that's how lmdb works on file systems that don't support sparse files I'm going to assume that behavior is controlled by a build flag somewhere. So I just need to turn that on. The only problem is figuring out which one. I don't really have time to dig through the 10K lines of source right now.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/CoreyKaylor/Lightning.NET/issues/97#issuecomment-301823255, or mute the thread https://github.com/notifications/unsubscribe-auth/AAkB8NA_bdFayH2RZyflS-oMZXI3oF0-ks5r6cN6gaJpZM4NTS03 .

[JordanZaerr]

I have a PR(#99) up for the 64bit binary at the moment. The 32bit is being troublesome. Its built from the latest off the master branch here. You can take a look at that dll though and see if it solves your issues. Using the test code above it creates a 20KB file instead of 2GB.

I bumped the version of it to 'LMDB 0.9.20: (May 16, 2017)'

[JordanZaerr]

I have the 32 bit one in there now as well. The sample code above throws an exception when running the 32 bit dll. The map size is apparently too large. The binaries in the 0.9.7 nuget throw the same exception as well. Lowering the map size fixes the issue.

[jakoss]

@CoreyKaylor Is it possible to release new nuget version with these new binaries from PR?

[CoreyKaylor]

I've published 0.9.9, can reopen if you're still having issues with that version.