DomDatabaseWrapper is slow

[jorabin]

DomDatabaseWrapper accesses data using the DOM. While this ensures that things that KeepassJava2 doesn't understand are saved and loaded transparently it also means that it is not very quick. An alternative in-memory implementation would be quicker.

[ivanovpv]

Would it be that SAX based parsing could be much better? Memory based DOM based implementation would be for sure much faster but memory consumption concern is crucial

[jorabin]

I did an Illustration of SAX parsing, fwiw

[Kin-k]

Hello. What you plan to do with this problem? It's very important problem. I want use KeePassJava2 in a programs that have a session time only few minutes. And spend 1 minute for a small database loading is too long.

[jorabin]

Hi there. Yesterday's release (yes, yesterday!) contains two new implementations, both of which should be a lot faster. One is based on the Simple XML parser, the other is based on JAXB.

Note, however, that the DOM implementation is actually a little faster on load but a lot slower on traversing the database once loaded. If you run this test you'll see the difference (results from my system pasted below).

I'm keen to hear your experiences with this, if you'd be kind enough to post them back here.

  KdbxCreds creds = new KdbxCreds("123".getBytes());
  InputStream inputStream = getClass().getClassLoader().getResourceAsStream("test1.kdbx");

then

  Database database = SimpleDatabase.load(credentials, inputStream)

or

  Database database = JaxbDatabase.load(credentials, inputStream)

The maven coordinates are

<groupId>org.linguafranca.pwdb</groupId>
<artifactId>KeePassJava2</artifactId>
<version>2.1.0</version>

Note that the artifact id is now camel case.

Speed comparisons

Note that the DOM implementation is faster to load but much slower to traverse, as follows (output from this test:

Simple 5 loads 20 iterations 257 millis
Jaxb 5 loads 20 iterations 326 millis
Dom 5 loads 20 iterations 758 millis

Simple 10 loads 1 iterations 340 millis
Jaxb 10 loads 1 iterations 552 millis
Dom 10 loads 1 iterations 175 millis

Simple 1 loads 50 iterations 28 millis
Jaxb 1 loads 50 iterations 47 millis
Dom 1 loads 50 iterations 251 millis

[Kin-k]

Thank you!!!

Jaxb is very good.

Jaxb 5 loads 20 iterations 4053 millis Jaxb 10 loads 1 iterations 3832 millis Jaxb 1 loads 50 iterations 359 millis

Simple droped with error:

Exception in thread "main" org.simpleframework.xml.core.ValueRequiredException: Unable to satisfy @org.simpleframework.xml.ElementList(entry=, inline=false, data=false, name=CustomIcons, type=void, required=true, empty=true) on field 'customIcons' protected java.util.ArrayList org.linguafranca.pwdb.kdbx.simple.model.KeePassFile$Meta.customIcons for class org.linguafranca.pwdb.kdbx.simple.model.KeePassFile$Meta at line 2
    at org.simpleframework.xml.core.Composite.validate(Composite.java:644)
    at org.simpleframework.xml.core.Composite.readElements(Composite.java:449)
    at org.simpleframework.xml.core.Composite.access$400(Composite.java:59)
    at org.simpleframework.xml.core.Composite$Builder.read(Composite.java:1383)
    at org.simpleframework.xml.core.Composite.read(Composite.java:201)
    at org.simpleframework.xml.core.Composite.read(Composite.java:148)
    at org.simpleframework.xml.core.Composite.readVariable(Composite.java:623)
    at org.simpleframework.xml.core.Composite.readInstance(Composite.java:573)
    at org.simpleframework.xml.core.Composite.readUnion(Composite.java:549)
    at org.simpleframework.xml.core.Composite.readElement(Composite.java:532)
    at org.simpleframework.xml.core.Composite.readElements(Composite.java:445)
    at org.simpleframework.xml.core.Composite.access$400(Composite.java:59)
    at org.simpleframework.xml.core.Composite$Builder.read(Composite.java:1383)
    at org.simpleframework.xml.core.Composite.read(Composite.java:201)
    at org.simpleframework.xml.core.Composite.read(Composite.java:148)
    at org.simpleframework.xml.core.Traverser.read(Traverser.java:92)
    at org.simpleframework.xml.core.Persister.read(Persister.java:625)
    at org.simpleframework.xml.core.Persister.read(Persister.java:606)
    at org.simpleframework.xml.core.Persister.read(Persister.java:584)
    at org.simpleframework.xml.core.Persister.read(Persister.java:543)
    at org.simpleframework.xml.core.Persister.read(Persister.java:444)
    at org.linguafranca.pwdb.kdbx.simple.SimpleDatabase.load(SimpleDatabase.java:169)
    at org.linguafranca.pwdb.kdbx.OpenDbExample$SimpleDbLoader.load(OpenDbExample.java:33)
    at org.linguafranca.pwdb.kdbx.OpenDbExample.testDb(OpenDbExample.java:61)
    at org.linguafranca.pwdb.kdbx.OpenDbExample.main(OpenDbExample.java:81)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.intellij.rt.execution.application.AppMain.main(AppMain.java:147)

Dom is very very slow: Dom 1 loads 50 iterations 84087 millis And I can't wait end other Dom tests. They are so slowly.

[Kin-k]

I waited. :) Dom 5 loads 20 iterations 310040 millis Dom 10 loads 1 iterations 503688 millis

[jorabin]

Thanks very much for that!

The difference between your values and the ones I reported above is quite a lot? You are testing the same database?

I'm also curious about why the Simple test fails, that is very strange.

What kind of environment are you running this on?

(I am running on Mac OS X 10.11.6 with SSD and 2.6 GHz Intel Core i7)

[Kin-k]

I am testing on my database. It is above 8Mb but it including few attached photos. Environment: Win7 SSD i7-4600U 2.1/2.7GHz 16Gb RAM

[jorabin]

I guess the photos would explain why so long to load. The database I am testing is 2kb. I will take the problem with Simple not loading as a bug. How was your database created? It looks like some of the time fields were not initialised.

[Kin-k]

Database was created manualy in KeePass Password Safe 2.34

[jorabin]

I pushed a patch to the develop branch https://github.com/jorabin/KeePassJava2/tree/develop which may fix the Simple load problem. Would be interesting to know. Plus I think it's faster than JAXB.

[jorabin]

I think it time to close this issue, since the key point of performance seems to have been addressed, for now. If there remain problems with loading KDBX files using the Simple implementation please open a new issue.