search

jorabin / KeePassJava2

Java API for KeePass Password Databases - Read/Write 2.x (File versions 3 and 4), Read 1.x
Apache License 2.0
250 stars 71 forks source link

UnrecognizedPropertyException: Unrecognized field "Item" when using JacksonDatabase or SimpleDatabase #70

Open lordgreg opened 2 weeks ago

lordgreg commented 2 weeks ago

We have a kdbx encrypted with password, using KDBX db v2.x

ls foobar.kdbx
 foobar.kdbx
❯ file foobar.kdbx
foobar.kdbx: Keepass password database 2.x KDBX

Opening the file with SimpleDatabase:

Exception in thread "main" java.lang.IllegalStateException: org.simpleframework.xml.core.ElementException: Element 'Item' does not have a match in class org.linguafranca.pwdb.kdbx.simple.model.KeePassFile$CustomData at line 35
    at org.linguafranca.pwdb.kdbx.simple.SimpleSerializableDatabase.load(SimpleSerializableDatabase.java:80)
    at org.linguafranca.pwdb.kdbx.simple.SimpleSerializableDatabase.load(SimpleSerializableDatabase.java:41)
    at org.linguafranca.pwdb.kdbx.KdbxStreamFormat.load(KdbxStreamFormat.java:65)
    at org.linguafranca.pwdb.kdbx.simple.SimpleDatabase.load(SimpleDatabase.java:91)
    at FOO.keepass2helm.Example.main(Example.kt:22)
    at FOO.keepass2helm.ExampleKt.main(Example.kt:77)
    Suppressed: org.bouncycastle.crypto.io.InvalidCipherTextIOException: Error finalising cipher
        at org.bouncycastle.crypto.io.CipherInputStream.finaliseCipher(Unknown Source)
        at org.bouncycastle.crypto.io.CipherInputStream.close(Unknown Source)
        at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
        at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
        at org.linguafranca.pwdb.kdbx.KdbxStreamFormat.load(KdbxStreamFormat.java:76)
        ... 3 more
    Caused by: org.bouncycastle.crypto.InvalidCipherTextException: pad block corrupted
        at org.bouncycastle.crypto.paddings.PKCS7Padding.padCount(Unknown Source)
        at org.bouncycastle.crypto.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
        ... 8 more
Caused by: org.simpleframework.xml.core.ElementException: Element 'Item' does not have a match in class org.linguafranca.pwdb.kdbx.simple.model.KeePassFile$CustomData at line 35
    at org.simpleframework.xml.core.Composite.readElement(Composite.java:527)
    at org.simpleframework.xml.core.Composite.readElements(Composite.java:445)
    at org.simpleframework.xml.core.Composite.access$400(Composite.java:59)
    at org.simpleframework.xml.core.Composite$Builder.read(Composite.java:1383)
    at org.simpleframework.xml.core.Composite.read(Composite.java:201)
    at org.simpleframework.xml.core.Composite.read(Composite.java:148)
    at org.simpleframework.xml.core.Composite.readVariable(Composite.java:623)
    at org.simpleframework.xml.core.Composite.readInstance(Composite.java:573)
    at org.simpleframework.xml.core.Composite.readUnion(Composite.java:549)
    at org.simpleframework.xml.core.Composite.readElement(Composite.java:532)
    at org.simpleframework.xml.core.Composite.readElements(Composite.java:445)
    at org.simpleframework.xml.core.Composite.access$400(Composite.java:59)
    at org.simpleframework.xml.core.Composite$Builder.read(Composite.java:1383)
    at org.simpleframework.xml.core.Composite.read(Composite.java:201)
    at org.simpleframework.xml.core.Composite.read(Composite.java:148)
    at org.simpleframework.xml.core.Composite.readVariable(Composite.java:623)
    at org.simpleframework.xml.core.Composite.readInstance(Composite.java:573)
    at org.simpleframework.xml.core.Composite.readUnion(Composite.java:549)
    at org.simpleframework.xml.core.Composite.readElement(Composite.java:532)
    at org.simpleframework.xml.core.Composite.readElements(Composite.java:445)
    at org.simpleframework.xml.core.Composite.access$400(Composite.java:59)
    at org.simpleframework.xml.core.Composite$Builder.read(Composite.java:1383)
    at org.simpleframework.xml.core.Composite.read(Composite.java:201)
    at org.simpleframework.xml.core.Composite.read(Composite.java:148)
    at org.simpleframework.xml.core.Traverser.read(Traverser.java:92)
    at org.simpleframework.xml.core.Persister.read(Persister.java:625)
    at org.simpleframework.xml.core.Persister.read(Persister.java:606)
    at org.simpleframework.xml.core.Persister.read(Persister.java:584)
    at org.simpleframework.xml.core.Persister.read(Persister.java:543)
    at org.simpleframework.xml.core.Persister.read(Persister.java:444)
    at org.linguafranca.pwdb.kdbx.simple.SimpleSerializableDatabase.load(SimpleSerializableDatabase.java:75)
    ... 5 more

Opening with JacksonDatabase:

Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "Item" (class org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile$CustomData), not marked as ignorable (one known property: "any"])
 at [Source: (GZIPInputStream); line: 36, column: 10] (through reference chain: org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile["Meta"]->org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile$Meta["CustomData"]->org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile$CustomData["Item"])
    at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
    at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
    at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2241)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1793)
    at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1771)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
    at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
    at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
    at com.fasterxml.jackson.dataformat.xml.deser.XmlDeserializationContext.readRootValue(XmlDeserializationContext.java:104)
    at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905)
    at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3885)
    at org.linguafranca.pwdb.kdbx.jackson.JacksonSerializableDatabase.load(JacksonSerializableDatabase.java:72)
    at org.linguafranca.pwdb.kdbx.jackson.JacksonSerializableDatabase.load(JacksonSerializableDatabase.java:45)
    at org.linguafranca.pwdb.kdbx.KdbxStreamFormat.load(KdbxStreamFormat.java:65)
    at org.linguafranca.pwdb.kdbx.jackson.JacksonDatabase.load(JacksonDatabase.java:81)
    at FOO.keepass2helm.Example.main(Example.kt:21)
    at FOO.keepass2helm.ExampleKt.main(Example.kt:77)
    Suppressed: com.fasterxml.jackson.core.JsonParseException: pad block corrupted
 at [Source: (GZIPInputStream); line: 36, column: 10]
        at com.fasterxml.jackson.dataformat.xml.util.StaxUtil.throwAsParseException(StaxUtil.java:25)
        at com.fasterxml.jackson.dataformat.xml.deser.FromXmlParser.close(FromXmlParser.java:538)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4913)
        ... 7 more
    Caused by: org.bouncycastle.crypto.InvalidCipherTextException: pad block corrupted
        at org.bouncycastle.crypto.paddings.PKCS7Padding.padCount(Unknown Source)
        at org.bouncycastle.crypto.paddings.PaddedBufferedBlockCipher.doFinal(Unknown Source)
        at org.bouncycastle.crypto.io.CipherInputStream.finaliseCipher(Unknown Source)
        at org.bouncycastle.crypto.io.CipherInputStream.close(Unknown Source)
        at java.base/java.util.zip.InflaterInputStream.close(InflaterInputStream.java:231)
        at java.base/java.util.zip.GZIPInputStream.close(GZIPInputStream.java:136)
        at com.fasterxml.aalto.in.StreamScanner._closeSource(StreamScanner.java:117)
        at com.fasterxml.aalto.in.XmlScanner.close(XmlScanner.java:351)
        at com.fasterxml.aalto.stax.StreamReaderImpl._closeScanner(StreamReaderImpl.java:1854)
        at com.fasterxml.aalto.stax.StreamReaderImpl.closeCompletely(StreamReaderImpl.java:1553)
        at com.fasterxml.jackson.dataformat.xml.deser.XmlTokenStream.closeCompletely(XmlTokenStream.java:351)
        at com.fasterxml.jackson.dataformat.xml.deser.FromXmlParser.close(FromXmlParser.java:533)
        ... 8 more

Jax and Dom work without the issue.

If I export the file as YAML and check the Object, i see there are 3:

        <CustomData>
            <Item>
                <Key>KPXC_autosaveDelayMin</Key>
                <Value>0</Value>
            </Item>
            <Item>
                <Key>KPXC_RANDOM_SLUG</Key>
                <Value>XXXXXXXXXXXXXXX</Value>
            </Item>
            <Item>
                <Key>_LAST_MODIFIED</Key>
                <Value>Wed Nov X XX:XX:XX 2024 GMT</Value>
            </Item>
        </CustomData>

This is probably something that came in with manipulating the DB with KeePassXC from here and here.

haroon-sheikh commented 2 weeks ago

We're seeing the same issue on our side when using the JacksonDatabase.

jorabin-sense commented 2 weeks ago

Hey @lordgreg

Could you please confirm that you are using version 2.2.2?

I suspect the reason for this is that CustomData has been defined as being empty in both Simple and Jackson (which copied Simple to some degree) - whereas DOM and JAXB don't care about elements they don't know about. It's interesting that this has not come up before.

Could you please supply some kind of test file containing "safe" data so I can include it in a test for the fix.

A fix will be made in 2.2.3.

NB 2.2.3 provides in-memory protection of Strings for Jackson only and is from now on the very much the preferred implementation. The others will be dropped in V3. See https://github.com/jorabin/KeePassJava2/discussions/60 and https://github.com/jorabin/KeePassJava2/discussions/62.

Thanks Jo

jorabin-sense commented 2 weeks ago

Hi @haroon-sheikh

Likewise to the above - would you please supply a safe test file ...

Thanks Jo

lordgreg commented 2 weeks ago

Hi @jorabin-sense ,

Yes, I am using the version 2.2.2: image

test-KeePassJava2.kdbx.zip Pass: KeePassJava2

File doesnt have entries, but does have the Fields:

        <CustomData>
            <Item>
                <Key>KPXC_RANDOM_SLUG</Key>
                <Value>df59b2ef51126309416ff8b48ea69dfaa9f00c8699e4d0201da51aecf663d2edb697e5d733cf8fca2613c4d697f735181ca72e6ddd4f676e0e73cef9e655ac95abbcec5b9a7fc119e2932e56cc65c8b189af96861c2362f5abbec173dc8a11e0d02bdc7346361943ec6f88a5071ed197af851de0f52c7d9bf6ac1fc8952a85973a05f59a77765348db6c7ee97c1ecba2b22fb269c9d3bb978ef492994abfee6be48a0f963e82a9073d20392bbe2181da18bdb08e49fd8d194a77f96dbadfa185dc47f0084270a3995bc7c17112a02e58fa9493236eacf40f72e274e73f0189887c9707f26caeff9dccd83530a2f95e118b209cbd4c5b04cee48b329954</Value>
            </Item>
            <Item>
                <Key>_LAST_MODIFIED</Key>
                <Value>Wed Nov 6 13:53:33 2024 GMT</Value>
            </Item>
            <Item>
                <Key>KPXC_DECRYPTION_TIME_PREFERENCE</Key>
                <Value>1000</Value>
            </Item>
        </CustomData>

This is easily reproducible if you just create a new KDBX in KeePassXC :).

haroon-sheikh commented 2 weeks ago

Thanks for looking into this @jorabin-sense, I can confirm, we're also using 2.2.2.

test.kdbx.zip Pass: KeePassJava2

jorabin commented 5 days ago

Hi folks, that is now fixed, and 2.2.3-SNAPSHOT is pushed to the sonatype repo see #60 for the coordinates.

Please confirm this works for you.

Be aware that the effect of this change is to ignore custom data ... ie. if you load and save the database then the saved database won't contain the custom data.

haroon-sheikh commented 5 days ago

Thanks so much @jorabin - I can confirm the 2.2.3-SNAPSHOT works for us.

lordgreg commented 4 days ago

Hi.

Sadly, we are now getting another Exception:

Exception in thread "main" com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException: Unrecognized field "PreviousParentGroup" (class org.linguafranca.pwdb.kdbx.jackson.JacksonEntry), not marked as ignorable (13 known properties: "ForegroundColor", "Binary", "OverrideURL", "CustomIconUUID", "Times", "AutoType", "String", "IconID", "BackgroundColor", "Tags", "UUID", "History", "icon"])
 at [Source: (GZIPInputStream); line: 6886, column: 73] (through reference chain: org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile["Root"]->org.linguafranca.pwdb.kdbx.jackson.model.KeePassFile$Root["Group"]->org.linguafranca.pwdb.kdbx.jackson.JacksonGroup["Group"]->java.util.ArrayList[1]->org.linguafranca.pwdb.kdbx.jackson.JacksonGroup["Entry"]->java.util.ArrayList[31]->org.linguafranca.pwdb.kdbx.jackson.JacksonEntry["PreviousParentGroup"])
        at com.fasterxml.jackson.databind.exc.UnrecognizedPropertyException.from(UnrecognizedPropertyException.java:61)
        at com.fasterxml.jackson.databind.DeserializationContext.handleUnknownProperty(DeserializationContext.java:1153)
        at com.fasterxml.jackson.databind.deser.std.StdDeserializer.handleUnknownProperty(StdDeserializer.java:2241)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownProperty(BeanDeserializerBase.java:1793)
        at com.fasterxml.jackson.databind.deser.BeanDeserializerBase.handleUnknownVanilla(BeanDeserializerBase.java:1771)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:316)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:122)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:246)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:122)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer._deserializeFromArray(CollectionDeserializer.java:361)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:246)
        at com.fasterxml.jackson.databind.deser.std.CollectionDeserializer.deserialize(CollectionDeserializer.java:30)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at com.fasterxml.jackson.dataformat.xml.deser.WrapperHandlingDeserializer.deserialize(WrapperHandlingDeserializer.java:122)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at com.fasterxml.jackson.databind.deser.impl.FieldProperty.deserializeAndSet(FieldProperty.java:138)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.vanillaDeserialize(BeanDeserializer.java:310)
        at com.fasterxml.jackson.databind.deser.BeanDeserializer.deserialize(BeanDeserializer.java:177)
        at com.fasterxml.jackson.dataformat.xml.deser.XmlDeserializationContext.readRootValue(XmlDeserializationContext.java:104)
        at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:4905)
        at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:3885)
        at org.linguafranca.pwdb.kdbx.jackson.JacksonSerializableDatabase.load(JacksonSerializableDatabase.java:70)
        at org.linguafranca.pwdb.kdbx.jackson.JacksonSerializableDatabase.load(JacksonSerializableDatabase.java:41)
        at org.linguafranca.pwdb.kdbx.KdbxStreamFormat.load(KdbxStreamFormat.java:65)
        at org.linguafranca.pwdb.kdbx.jackson.JacksonDatabase.load(JacksonDatabase.java:81)
        at foo.bar.baz.keepass2helm.KeepassDb.<init>(KeepassDb.kt:16)
        at foo.bar.baz.keepass2helm.Keepass2Helm.run(Keepass2Helm.kt:61)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:198)
        at com.github.ajalt.clikt.parsers.Parser.parse(Parser.kt:18)
        at com.github.ajalt.clikt.core.CliktCommand.parse(CliktCommand.kt:395)
        at com.github.ajalt.clikt.core.CliktCommand.parse$default(CliktCommand.kt:392)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:410)
        at com.github.ajalt.clikt.core.CliktCommand.main(CliktCommand.kt:435)
        at foo.bar.baz.keepass2helm.Keepass2HelmKt.main(Keepass2Helm.kt:81)
        Suppressed: com.fasterxml.jackson.core.JsonParseException: pad block corrupted
jorabin commented 4 days ago

Oh dear. The example file you sent before works OK, so this must be a different one?

lordgreg commented 4 days ago

Oh dear. The example file you sent before works OK, so this must be a different one?

Correct. I will try to generate a new, reproducible KDBX.

Thank you for your great support!

jorabin commented 3 days ago

On reflection, I think the problem is that neither Simple nor Jackson support the KDBX 4.1 file format (JAXB and DOM do because they have no fixed idea of the exact format of the file beyond Groups/Entries etc.).

So, simply ignoring the content of CustomData doesn't go far enough. The changes listed in this note go further than that. I'm hoping it won't be too hard to make the adjustments, which would mean that the content of the file should remain intact on a read/write cycle ...

... as mentioned on previous occasions, I'm dropping support for Simple from release 3 on, it hasn't been maintained for a long time, so I will need to decide whether it's worth making this upgrade to the Simple implementation. Hoping that people will feel that using Jackson as a drop in replacement - if necessary - will suffice.

jorabin commented 3 days ago

I've made changes to fix this properly now, i.e. KDBX 4.1 file format now supported, hopefully. Should load and save the data intact.

@lordgreg will you please check that this version of 2.2.3-SNAPSHOT which I pushed to sonatype snapshot repo moments ago works for you.

Confirming that I didn't make any changes to Simple, other than what is mentioned above. i.e. that will remain not supporting 4.1 file format.