PowerShell wrapper for LiteDB

[v2kiran]

Check out the Initial release for PSLiteDB

[mbdavid]

Hi @v2kiran, that awsome! Nice job. I will update my README references to add your project. Thanks!

BTW, I saw you documentation and I was curious about performance speed: 30,000 documents in 2 minutes it's too much time. I don't know how do you implement.... using direct LiteDB I can insert 200,000 documents in less than 10 seconds (using SSD). Take a look on my lastest performance test here (v5 branch: https://github.com/mbdavid/LiteDB/tree/v5)

[v2kiran]

welcome Mauricio...thx for checking it!. Yeah I was surprised with the speed myself but I think thats mainly because I am converting the "powershell custom objects" to JSON in powershell using Convertto-Json, deserialize the JSON to BSON using Litedb and finally insert into collection.

I tried to do this in Litedb without using Converto-json but got errors about circular reference and something about fields being more than 20.

[LiteDB.JsonSerializer]::Deserialize( ( ConvertTo-Json -InputObject $i -Depth $Depth ) )

Note: $i here is a powershell object and depth is the serialization level.

I am pretty sure that the speed will be good if done all in c# instead of using powershell as the go-between.

[nightroman]

Hi @v2kiran , it's me again :) I was passing by, found this post, and ran similar performance tests out of curiosity.

Here is another kid on the block, Ldbc, the PowerShell module for LiteDB.

Performance test after the original:

Import-Module Ldbc
$dbPath = "C:\temp\Speedtest.LiteDB"

Use-LiteDatabase $dbPath {
    $P1 = Get-LiteCollection P1

    $sw = [System.Diagnostics.Stopwatch]::StartNew()

    # Insert 30000 records
    1..30000 | .{process{
        [PSCustomObject]@{
            FirstName  = "user-{0}" -f $_
            LastName   = "lastname-{0}" -f $_
            Age        = $_
            Occupation = "Singer{0}" -f $_
        }
    }} | Add-LiteData $P1

    "Count: $($P1.Count()) Time: $($sw.Elapsed)"
}

Output depending on options:

# original code:
Count: 30000 Time: 00:00:06.3219046

# + invoke in a transaction
Count: 30000 Time: 00:00:02.1306306

# + replace [PSCustomObject] with [ordered]
Count: 30000 Time: 00:00:01.0082221

The original performance test could be faster:

• PowerShell and machines are faster nowadays
• If notoriously slow ForEach-Object is replaced with fast .{process{...}}
• [PSCustomObject] is slow, using dictionaries for input is twice faster
• (and it's more likely that the intermediate JSON is the performance issue)

All is all, it is much slower than native LiteDB results but it's relatively fine for scripting. Scripting is not for high speed tasks but rather for data exploration and ad-hoc tweaks.

/cc @mbdavid

[v2kiran]

Hi @nightroman those are really good numbers :) Thanks for the tip with the process block, will try that out.