beans generate from opendj schema

[maxfieb]

It seems that the beans generator cannot parse schema from opendj ldap server.

The schema parser seems to get the schema successfully, but then apply regexes to the schema retrieved and fails to parse the schema. It looks like the parser assumes a particular order of regexes matching when parsing the schema.

Opendj seems to return things in a different, maybe unordered way, which causes the assumptions as to the order in which regexes will match, to fail. It seems like the ldap schema standard does not require a particular order, but maybe the ldap servers that are currently supported return a hashset, which is returned in a partially sorted way as a result.

Is this something that would be on the radar ?

[dfish3r]

Please attach the schema file that is producing errors and I'll take a look.

[maxfieb]

open source opendj binaries and source are at : OpenIdentityPlatform

used a standard install in port 1389, with sample data as part of the install

[maxfieb]

The schema I am currently running on my dev system is below. The running schema was exported into ldif, from the schema which my local copy was running.

If you run the opendj locally, there is a schema on the disk if you download the binary, but on the filesystem the schema text file is not used after the first successful boot, after that it's stored in the DIT under dn=schema in the form you see when you use ldapbrowser etc., the one on disk may differ from the one loaded into DIT id's use the running one (whitespace, order?).

opendj_schema.txt

[maxfieb]

Test was on ldaptive 2.0.1 release

@CommandLine.Command class GenerateBeans : Runnable {

var ldapConfig : LdapConfig? = null

fun generateBeans() {
    val schema = SchemaFactory.createSchema(
        ldapConfig!!.connectionFactory()
    )
    val generator = BeanGenerator(
        schema,
        "net.spacenow.kotlin.dto", arrayOf("inetOrgPerson","groupOfUniqueNames")
    )
    generator.generate()
    generator.write()
}

companion object {
    @JvmStatic
    fun main(args: Array<String>) {
        val ctx : ApplicationContext = Micronaut.build()
            .args(*args)
            .packages("net.spacenow.kotlin.config")
            .start()
        PicocliRunner.run(GenerateBeans::class.java, ctx, *args)
        ctx.close()
    }
}

override fun run() {
    generateBeans()
}

Exception was :

11:58:15.925 [ldaptive-SingleThread@1993366729-io-4-1] DEBUG io.netty.buffer.PoolThreadCache - Freed 3 thread-local buffer(s) from thread: ldaptive-SingleThread@1993366729-io-4-1
Exception in thread "main" picocli.CommandLine$ExecutionException: Error while running command (net.spacenow.kotlin.generate.GenerateBeans@aac3f4e): java.lang.IllegalArgumentException: Could not transcode attribute type
    at picocli.CommandLine.executeUserObject(CommandLine.java:1948)
    at picocli.CommandLine.access$1300(CommandLine.java:145)
    at picocli.CommandLine$RunLast.executeUserObjectOfLastSubcommandWithSameParent(CommandLine.java:2352)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2346)
    at picocli.CommandLine$RunLast.handle(CommandLine.java:2311)
    at picocli.CommandLine$AbstractParseResultHandler.handleParseResult(CommandLine.java:2172)
    at picocli.CommandLine.parseWithHandlers(CommandLine.java:2550)
    at picocli.CommandLine.run(CommandLine.java:3099)
    at picocli.CommandLine.run(CommandLine.java:3023)
    at io.micronaut.configuration.picocli.PicocliRunner.run(PicocliRunner.java:139)
    at net.spacenow.kotlin.generate.GenerateBeans$Companion.main(GenerateBeans.kt:39)
    at net.spacenow.kotlin.generate.GenerateBeans.main(GenerateBeans.kt)
Caused by: java.lang.IllegalArgumentException: Could not transcode attribute type
    at org.ldaptive.schema.transcode.AttributeTypeValueTranscoder.decodeStringValue(AttributeTypeValueTranscoder.java:22)
    at org.ldaptive.schema.transcode.AttributeTypeValueTranscoder.decodeStringValue(AttributeTypeValueTranscoder.java:12)
    at org.ldaptive.transcode.AbstractStringValueTranscoder.decodeBinaryValue(AbstractStringValueTranscoder.java:21)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.stream.ReferencePipeline$3$1.accept(ReferencePipeline.java:195)
    at java.base/java.util.stream.ReferencePipeline$2$1.accept(ReferencePipeline.java:177)
    at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
    at java.base/java.util.Spliterators$IteratorSpliterator.forEachRemaining(Spliterators.java:1801)
    at java.base/java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:484)
    at java.base/java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:474)
    at java.base/java.util.stream.ReduceOps$ReduceOp.evaluateSequential(ReduceOps.java:913)
    at java.base/java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234)
    at java.base/java.util.stream.ReferencePipeline.collect(ReferencePipeline.java:578)
    at org.ldaptive.LdapAttribute.getValues(LdapAttribute.java:283)
    at org.ldaptive.schema.SchemaFactory.createSchema(SchemaFactory.java:140)
    at org.ldaptive.schema.SchemaFactory.createSchema(SchemaFactory.java:102)
    at net.spacenow.kotlin.generate.GenerateBeans.generateBeans(GenerateBeans.kt:21)
    at net.spacenow.kotlin.generate.GenerateBeans.run(GenerateBeans.kt:45)
    at picocli.CommandLine.executeUserObject(CommandLine.java:1939)
    ... 11 more
Caused by: java.text.ParseException: Invalid attribute type definition: ( 2.5.18.12 NAME 'collectiveAttributeSubentries' EQUALITY distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 USAGE directoryOperation NO-USER-MODIFICATION X-ORIGIN 'RFC 3671' )
    at org.ldaptive.schema.AttributeType.parse(AttributeType.java:411)
    at org.ldaptive.schema.transcode.AttributeTypeValueTranscoder.decodeStringValue(AttributeTypeValueTranscoder.java:20)
    ... 29 more

[dfish3r]

The schema file you attached has a couple problems which make it invalid. Spurious whitespace and elements in the wrong order. I've attached a fixed version for your review. I think there is an argument to be made for relaxing the whitespace restrictions. But the rule element ordering would be harder to support. opendj_schema_fixed.txt

[maxfieb]

Does ldap schema mandate order of elements with a line, other than the beginning or trailing items ?

Other parsing implementations look for the start delimiter ( then the name of the first non-whitespace name, consulting a lookup table to see if it has no value, a quoted value, or a single non-space block, parse the optional value then look for the next name etc., ) then terminating on the end delimiter.

At the very least using one big regexp means that the match could appear in different indexes in the regex result. Ordering aside, there are likely several cases where presence or absence of a name might change the index in ways not already anticipated in the code. The ldap part of the middleware will work with many more products than the schema parser.

I could try to code something more generic / vendor agnostic, but not sure how it would work in with what is there.

[dfish3r]

Does ldap schema mandate order of elements with a line, other than the beginning or trailing items ?

The RFC strictly defines the order and whitespace. Implementations vary in their leniency.

At the very least using one big regexp means that the match could appear in different indexes in the regex result.

The regex is run against each attribute so I don't think your concern is valid. Although, ff you've found a bug, please share.

I could try to code something more generic / vendor agnostic, but not sure how it would work in with what is there.

I think I can make this component pluggable without too much API disruption and provide a CharBuffer implementation. Give me a day or two to hash something out.

[maxfieb]

tracking it back to the schema file that is used to initialize the instance, there is a double space to indent, where only a single space in an .ldif means line continuation. So the double space in the running schema comes from the install 00-core.ldif schema which is reading using ldif-specific line continuation rules

schema files were text based, this file used to initialize a new instance is in 00-core.ldif and conversion from .schema to .ldif has been pretty relaxed, moreso when it is read into a live cn=schema as text in a live database file. i'm guessing that the different ldap vendors went different ways on order and whitespaces, openldap went thier own way entirely here (predates opends).

attributeTypes: ( 2.5.18.12 NAME 'collectiveAttributeSubentries' EQUALITY distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 USAGE directoryOperation NO-USER-MODIFICATION X-ORIGIN 'RFC 3671' )

The rfc3671 has different whitespace, but same order.

( 2.5.18.12 NAME 'collectiveAttributeSubentries' EQUALITY distinguishedNameMatch SYNTAX 1.3.6.1.4.1.1466.115.121.1.12 USAGE directoryOperation NO-USER-MODIFICATION )

So i'm guessing that implementations read, but generally ignore, contiguous whitespace, and opendj (historically opends / sun one) might re-order the schema when it is read into a specific installed directory instance, vs the original pre-install schema.

00-core.ldif.txt

I could not find any standards on ldif specific schema formats, implementations vary widely as they span decades.

[dfish3r]

In terms of validity, I wasn't referring to LDIF in general but to RFC 4512 starting in section 4.1. Which strictly defines the format of schema definitions. And you're correct, implementations vary wildly in their leniency for most things LDAP. I hope to have a more lenient implementation that you can try this weekend.

[dfish3r]

Try out the latest snapshot, it should fix your issue. I've added a unit test that uses your schema.

[dfish3r]

@maxfieb any feedback on this solution?

[maxfieb]

I'll give it a try tonight :)

[maxfieb]

yep it looks good here, my test project connects to the opendj ldap, then generates some annotated beans, well done :)

[dfish3r]

Thanks!