Add .schema files to openldap

mathieudevos commented 5 years ago

Hey,

How can you rerun the base ldap.conf so that new .schema files can be added to the setup?

I have .schema files to emulate a windows AD, but I cannot reconfigure the whole thing since all the base files are deleted.

Any pointers on how to get this running would be great.

obourdon commented 5 years ago

@mathieudevos not sure I fully understand what you are asking here. Do you already have a running instance of osixia/openldap for which you want to change configuration and add .schema files ? Or do you simply want to add custom .schema files to a non-existing osixia/openldap instance ? Thanks for clarifying

mathieudevos commented 5 years ago

Hey,

Thanks for the quick response. I have a bunch of .schema files from here: https://github.com/dkoudela/active-directory-to-openldap/tree/master/schema

And they need to be in the following order for slaptest to compile:

microsoftattributetype
microsoftattributetypestd
core (modified core from that url)
... everything else
microsoftobjectclass

To do that I will need to most likely modify the boostrap script. However I'm at a loss here with doing that. Currently I rebuilt the container with a hard copy of that folder, manually edited the schema-to-ldif.sh file to do the following:

echo "include /container/service/slapd/schemas/microsoftattributetype.schema" >> convert.dat
echo "include /container/service/slapd/schemas/microsoftattributetypestd.schema" >> convert.dat
echo "include /container/service/slapd/schemas/core.schema" >> convert.dat
echo "include /container/service/slapd/schemas/cosine.schema" >> convert.dat
echo "include /container/service/slapd/schemas/inetorgperson.schema" >> convert.dat

if [ -e "/etc/ldap/schema/rfc2307bis.schema" ]; then
  echo "include /etc/ldap/schema/rfc2307bis.schema" >> convert.dat
else
  echo "include /etc/ldap/schema/nis.schema" >> convert.dat
fi

for schema in ${SCHEMAS} ; do
    echo "include ${schema}" >> convert.dat
done

echo "include /container/service/slapd/schemas/microsoftobjectclass.schema" >> convert.dat

Which passes the slaptest - however the server remains on "slapd starting" and never progresses beyond that. Going into the container and going to the /etc/ldap/slapd.d/cn=schema folder shows that there's just 5 files:

cn={0}core.ldif 
cn={1}cosine.ldif 
cn={2}nis.ldif 
cn={3}inetorgperson.ldif 
cn={4}ppolicy.ldif

Which has the microsoft ones missing from it.

Other ways around this? Or how would I just start with something like this? More than happy to contribute to this so that we can have a flag for windows openldap settings if we figure this out.

Fast edit: to answer your question, I will happily start from scratch on this, so a non-existing one would most likely be the valid format. I have a bunch of groups & users from a windows AD, and would like to duplicate them in an openldap container so I can test-run my program to connect to it :-).

dylanpiergies commented 5 years ago

I'm trying to do the same thing. The repository that @mathieudevos mentions above contains a customised schema that more closely matches that of Active Directory, along with a slapd.conf that contains, amongst one or two other things, some rewrite rules that influence the search behaviour. Any guidance on how we can apply this would be appreciated.

dylanpiergies commented 5 years ago

@obourdon To clarify, my intention here is to bootstrap a fresh container with the modified schema and configuration in a new image derived from osixia/openldap.

fbartels commented 5 years ago

And they need to be in the following order for slaptest to compile:

have you tried naming your files like 01microsoftattributetype.schema, 02microsoftattributetypestd.schema ... ?

quanah commented 4 years ago

There are two different things at play here.

".schema" files are traditionally only for slapd.conf ".ldif" files are for use with cn=config

Second, I would note that OpenLDAP master now has an AD schema (msuser.schema/msuser.ldif).

Generally one should not be in the business of converting schema, since OpenLDAP already ships with .ldif format schema files for use with cn=config.

guimaluf commented 4 years ago

I'm struggling here to do some similar thing. Add custom schemas to this openldap setup

Guarionex commented 4 years ago

I'm struggling as well. Trying to add AD-LDS schema to OpenLdap to no success.

monclaf commented 4 years ago

Hey, I would like to duplicate data from an existing Ldap server (OpenDJ) to an osixia/openLap container. I've exported data as an ldif file and also schema as another ldif file with LDap Admin tool. Default osixia schema is different from the one I've exported, so I would like to load at container creation the schema I would like. I've tried to do like the "Seed ldap database with ldif" paragraph without success. Does someone get succeed in defining its own schema ? Thanks for any help.

clayrisser commented 4 years ago

I'm wondering the same thing. How do get this to use active directory schema?

https://github.com/dkoudela/active-directory-to-openldap/tree/master/schema

@mathieudevos any luck?

clayrisser commented 4 years ago

There are two different things at play here.

".schema" files are traditionally only for slapd.conf ".ldif" files are for use with cn=config

Second, I would note that OpenLDAP master now has an AD schema (msuser.schema/msuser.ldif).

Generally one should not be in the business of converting schema, since OpenLDAP already ships with .ldif format schema files for use with cn=config.

How would I used msuser? I cannot find it in the docker image.

@quanah

lvillaca commented 4 years ago

I also had this issue, and tried editing schema-to-ldif.sh with no luck. I was further able to add it by producing a new Docker image, copying my new schema (via Dockerfile) to "/container/service/slapd/assets/config/bootstrap/schema/" - see custom schema

Kramins commented 4 years ago

I am also trying to figure out a solution to adding a new schema to an existing container. I am trying to add postfix-book.schema I managed to convert it to an ldif by creating a conf like so

include /etc/ldap/schema/core.schema
include /etc/ldap/schema/cosine.schema
include /etc/ldap/schema/inetorgperson.schema
include /ldif/postfix-book.schema

but I am unsure what to do with the generated files and directory, truthfully I was expecting it to generate one file.

can I just add the schema file to /container/service/slapd/assets/config/bootstrap/ldif/custom/

mckenziec commented 4 years ago

Has anyone made progress on this topic? I'm also in the midst of trying to incorporate an ActiveDirectory lite schema into osixia and having no end of troubles.

quanah commented 4 years ago

You might find this useful. It's a general guide on how to convert .schema files to .ldif files for use with cn=config.

mckenziec commented 4 years ago

Thanks. I was writing a reply about the dependency hell I couldn't resolve, but I actually managed to convert the schema files included in https://github.com/dkoudela/active-directory-to-openldap to ldif.

Here's the config I used. The order of the schema files is key:

include microsoftattributetypestd.schema  
include microsoftattributetype.schema     
include core.schema  
include cosine.schema    
include inetorgperson.schema
include nis.schema
include microsoftobjectclass.schema      
include collective.schema
include dyngroup.schema
include misc.schema
include corba.schema       
include java.schema                       
include openldap.schema
include pmi.schema
include ppolicy.schema
include duaconf.schema

I'm at the same point as Karmis. What do I do with the resulting directory ldif files? Do I just mount the direct as a volume in osixia? Thanks.

mckenziec commented 4 years ago

Also as quanah suggested, I grabbed the msusers.ldif and tried messing around it.

Simply adding it as a custom ldif didn't work. There are complaints about a duplicate attribute. BTW, the encoded characters are different everytime I start the osixia container. I make sure there's no persistent storage of the ldap data.

Add custom bootstrap ldif...
Processing file /container/run/service/slapd/assets/config/bootstrap/ldif/custom/msuser.ldif
5ef26d9a conn=1024 fd=12 ACCEPT from PATH=/var/run/slapd/ldapi (PATH=/var/run/slapd/ldapi)
5ef26d9a conn=1024 op=0 BIND dn="" method=163
5ef26d9a conn=1024 op=0 BIND authcid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" authzid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"
5ef26d9a conn=1024 op=0 BIND dn="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" mech=EXTERNAL sasl_ssf=0 ssf=71
5ef26d9a conn=1024 op=0 RESULT tag=97 err=0 text=
5ef26d9a conn=1024 op=1 ADD dn="cn=msuser,cn=schema,cn=config"
5ef26d9a conn=1024 op=1 RESULT tag=105 err=80 text=olcAttributeTypes: Duplicate attributeType: "�~4v"
5ef26d9a conn=1024 op=2 UNBIND
5ef26d9a conn=1024 fd=12 closed
5ef26d9a conn=1025 fd=12 ACCEPT from IP=127.0.0.1:58360 (IP=0.0.0.0:389)
5ef26d9a conn=1025 op=0 BIND dn="cn=admin,dc=csdpac,dc=demo" method=128
5ef26d9a conn=1025 op=0 BIND dn="cn=admin,dc=csdpac,dc=demo" mech=SIMPLE ssf=0
5ef26d9a conn=1025 op=0 RESULT tag=97 err=0 text=
5ef26d9a conn=1025 op=1 ADD dn="cn=msuser,cn=schema,cn=config"
5ef26d9a conn=1025 op=1 RESULT tag=105 err=50 text=
5ef26d9a conn=1025 op=2 UNBIND
5ef26d9a conn=1025 fd=12 closed
ldap_add: Other (e.g., implementation specific) error (80)
    additional info: olcAttributeTypes: Duplicate attributeType: "�~4v"
adding new entry "cn=msuser,cn=schema,cn=config"

ldap_add: Insufficient access (50)
adding new entry "cn=msuser,cn=schema,cn=config"
*** /container/run/startup/slapd failed with status 50

mckenziec commented 4 years ago

The admin account doesn't have access to cn=config. I can't view the schema of any running osixia container. If I embed msuser.lidf as /container/service/slapd/assets/config/bootstrap/ldif/06-msuser.ldif in a Docker image that's FROM the osixia image, I still get:

adding new entry "cn=msuser,cn=schema,cn=config"
ldap_add: Insufficient access (50)

Maybe I'm going crazy, but I thought seeding the osixia container would allow for schema level seeding. What am I missing? I'm on the verge of throwing the whole thing out.

quanah commented 4 years ago

There seems to be something seriously wrong with the msuser LDIF file you're adding, given that it prints out those characters. I would note that OpenLDAP already has this file converted for use so you shouldn't need to convert it yourself.

mckenziec commented 4 years ago

Hi quanah.

That's the ldif I'm using.

curl  https://git.openldap.org/openldap/openldap/-/raw/master/servers/slapd/schema/msuser.ldif -o ./ldif/msuser.ldif

Here's my Dockerfile:

FROM osixia/openldap
COPY ${PWD}/ldif/msuser.ldif /container/service/slapd/assets/config/bootstrap/ldif/06-msuser.ldif

I build it tagged as "test/test-ldap", and run with compose:

  ldap:
    image: test/test-ldap:latest
    container_name: ldap
    domainname: "test.local"
    hostname: "ldap"
    command: --copy-service --loglevel debug
    ports:
      - "389:389"
    env_file:
      - ${PWD}/ldap.env

My ldap.env just has this in it:

LDAP_LOG_LEVEL=256
LDAP_ORGANISATION=Test Inc.
LDAP_DOMAIN=test.local
LDAP_BASE_DN=dc=test,dc=local
LDAP_ADMIN_PASSWORD=admin
LDAP_TLS=false

Docker logs for the container indicates the same problem, no matter how I approach it:

...
5ef29060 conn=1024 fd=12 ACCEPT from PATH=/var/run/slapd/ldapi (PATH=/var/run/slapd/ldapi)
5ef29060 conn=1024 op=0 BIND dn="" method=163
5ef29060 conn=1024 op=0 BIND authcid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" authzid="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth"
5ef29060 conn=1024 op=0 BIND dn="gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth" mech=EXTERNAL sasl_ssf=0 ssf=71
5ef29060 conn=1024 op=0 RESULT tag=97 err=0 text=
5ef29060 conn=1024 op=1 ADD dn="cn=msuser,cn=schema,cn=config"
5ef29060 conn=1024 op=1 RESULT tag=105 err=80 text=olcAttributeTypes: Duplicate attributeType: "�~4�"
5ef29060 conn=1024 op=2 UNBIND
5ef29060 conn=1024 fd=12 closed
5ef29060 conn=1025 fd=12 ACCEPT from IP=127.0.0.1:34038 (IP=0.0.0.0:389)
5ef29060 conn=1025 op=0 BIND dn="cn=admin,dc=test,dc=local" method=128
5ef29060 conn=1025 op=0 BIND dn="cn=admin,dc=test,dc=local" mech=SIMPLE ssf=0
5ef29060 conn=1025 op=0 RESULT tag=97 err=0 text=
5ef29060 conn=1025 op=1 ADD dn="cn=msuser,cn=schema,cn=config"
5ef29060 conn=1025 op=1 RESULT tag=105 err=50 text=
5ef29060 conn=1025 op=2 UNBIND
5ef29060 conn=1025 fd=12 closed
ldap_add: Other (e.g., implementation specific) error (80)
    additional info: olcAttributeTypes: Duplicate attributeType: "�~4�"
adding new entry "cn=msuser,cn=schema,cn=config"

ldap_add: Insufficient access (50)
adding new entry "cn=msuser,cn=schema,cn=config"
*** /container/run/startup/slapd failed with status 50

*** Run commands before finish...
*** Killing all processes...
5ef29060 daemon: shutdown requested and initiated.
5ef29060 slapd shutdown: waiting for 0 operations/tasks to finish
5ef29060 slapd stopped.

I've tried making is a custom volume mount, embedding it in a different location and setting LDAP_SEED_INTERNAL_LDIF_PATH, and as you can see above, I embedded it in my own Docker image in the same ldif directory that the osixia container will run last.

The default ldif files are run, including setting the admin password. It's when the msuser.ldif is processed that things fall apart. It looks like cn=admin doesn't have access to cn=config.

I don't doubt there could be something wrong with the ldif, or my apporach, but out-of-the-box I can access/view the installed default schema. The admin account doesn't seem to have the ACLs needed to update things.

I have no clue where those attributeType characters are coming from.

mckenziec commented 4 years ago

I'm not familiar enough with the ldif ACL syntax to say if the included one would impact the msuser.ldif:

dn: olcDatabase={1}{{ LDAP_BACKEND }},cn=config
changetype: modify
delete: olcAccess
-
add: olcAccess
olcAccess: to * by dn.exact=gidNumber=0+uidNumber=0,cn=peercred,cn=external,cn=auth manage by * break
olcAccess: to attrs=userPassword,shadowLastChange by self write by dn="cn=admin,{{ LDAP_BASE_DN }}" write by anonymous auth by * none
olcAccess: to * by self read by dn="cn=admin,{{ LDAP_BASE_DN }}" write by * none

https://github.com/osixia/docker-openldap/blob/stable/image/service/slapd/assets/config/bootstrap/ldif/02-security.ldif

There was a commit to it in April: https://github.com/osixia/docker-openldap/issues/402

quanah commented 4 years ago

I'm not familiar enough with the ldif ACL syntax to say if the included one would impact the msuser.ldif:
dn: olcDatabase={1}{{ LDAP_BACKEND }},cn=config

The Osixia container already provides full write access to the cn=config db for the root user over ldapi. There is nothing you need to change in regards to the ACLs. The above change also is completely invalid since it's modifying the olcDatabase={1}mdb,cn=config database access rules and not the access rules for the cn=config DB.

This particular error is bizarre, as it seems to indicate that somehow you have gotten non-ascii text in your LDIF file:

Duplicate attributeType: "�~�4�"

I would note there is a typo in your curl command, you have msuser.lidf instead of msuser.ldif as the destination object. Is it possible your install is picking up something other than the actual msuser.ldif from the OpenLDAP repo?

mckenziec commented 4 years ago

Thanks quanah. I won't pretend to understand openldap enough to diagnose this. That is a typo in my post, not in how I downloaded the file. Sorry about that.

mckenziec commented 4 years ago

I don't know where the (maybe utf-8) characters are coming from. It's literally the 1st non-comment entry in the msuser.ldif. I grepped through all the olcAttributeTypes and they aren't complicated. I downloaded the file directly to my CentOS8 dev env, then copy it into the image from my Dockerfile.

I also blew away all my osixia images and rebuilt. Then I inspected the msuser.ldif file inside the container and it looks the same.

752319951 commented 3 years ago

Hello, what's the correct way to add a custom .schema file based on the above discussion？

Kumassy commented 3 years ago

/service/slapd/assets/schema-to-ldif.sh converts .schema files into .ldif, so you don't need to convert .schema files manually:

https://github.com/osixia/docker-openldap/blob/a3f7fb31ed882180bff6cd9fdd3182f2c81ea612/image/service/slapd/assets/schema-to-ldif.sh

Kumassy commented 3 years ago

@752319951 Only you have to do is

Mount .schema files to /container/service/slapd/assets/config/bootstrap/schema/
Start container with --copy-service (See: https://github.com/osixia/docker-openldap/issues/301#issuecomment-724669148, https://github.com/osixia/docker-openldap#seed-ldap-database-with-ldif )
Persist /etc/ldap/slapd.d, /var/lib/ldap

here is my docker-compose.yml

version: '3'
services:
  ldap-service:
    image: osixia/openldap:1.1.8
    container_name: ldap-service
    environment:
      LDAP_ORGANISATION: example
      LDAP_DOMAIN: example.com
      LDAP_ADMIN_PASSWORD: admin
    ports:
      - "389:389"
      - "636:636"
    command: --copy-service
    volumes:
      - ldapdb:/var/lib/ldap
      - ldapconf:/etc/ldap/slapd.d
      - ./foobar.schema:/container/service/slapd/assets/config/bootstrap/schema/foobar.schema

752319951 commented 3 years ago

@Kumassy Thank you very much, this method works!

by the way... I export whole schema file from old ldap and try to use above method but failed.. so i delete irrelevant content (maybe contents in core.schema etc.) and keep newly added content only. This process is complex for me. is there any way to use whole old schema file?

Kumassy commented 3 years ago

startup.sh converts .schema -> .ldif and ldapadd them, only when is_new_schema returns 1.

Writing a shell script based on these lines may work.

https://github.com/osixia/docker-openldap/blob/7ce084bab1f9016830548558241fdefb60862482/image/service/slapd/startup.sh#L138-L145

https://github.com/osixia/docker-openldap/blob/7ce084bab1f9016830548558241fdefb60862482/image/service/slapd/startup.sh#L329-L339

kevdogg commented 3 years ago

Sorry to labor the point, but I have existing ldap container with database -- not starting from scratch. Is it possible to add a schema in this instance? The bootstrap method seems to be for empty containers. I'd like to add ldapns.schema schema.

Kumassy commented 3 years ago

I don't have a good workaround for this. But I use docker-phpLDAPadmin to export/import existing LDAP records.

Here is what I try:

export .idif using docker-phpLDAPadmin
clear volumes related to container to add a new schema
import .idif via docker-phpLDAPadmin

Vincz commented 3 years ago

Hey guys, did any of you made any progress about this? I'm stuck at the same point, unable to load my active directory schema.

christf commented 2 years ago

@752319951 Only you have to do is

* Mount .schema files to `/container/service/slapd/assets/config/bootstrap/schema/`

* Start container with `--copy-service` (See: [Error volume mounting custom schema  #301 (comment)](https://github.com/osixia/docker-openldap/issues/301#issuecomment-724669148), https://github.com/osixia/docker-openldap#seed-ldap-database-with-ldif )

* Persist `/etc/ldap/slapd.d`, `/var/lib/ldap`

here is my docker-compose.yml

version: '3'
services:
  ldap-service:
    image: osixia/openldap:1.1.8
    container_name: ldap-service
    environment:
      LDAP_ORGANISATION: example
      LDAP_DOMAIN: example.com
      LDAP_ADMIN_PASSWORD: admin
    ports:
      - "389:389"
      - "636:636"
    command: --copy-service
    volumes:
      - ldapdb:/var/lib/ldap
      - ldapconf:/etc/ldap/slapd.d
      - ./foobar.schema:/container/service/slapd/assets/config/bootstrap/schema/foobar.schema

this has not worked for me. I have worked with both ldap (mounted to /container/service/slapd/assets/config/6 bootstrap/ldif/custom/foo.ldif) and schema (mounted to container/service/slapd/assets/config/bootstrap/schema/foo.schema) files.

When using the ldif mount approach the changed schema is visible during the first startup of the container only but not on consecutive starts. I am using --copy-service and these mounts:

/srv/docker/openldap/data/ldap:/var/lib/ldap
/srv/docker/openldap/data/slapd:/etc/ldap/slapd.d
/srv/docker/openldap/data/certs:/container/service/slapd/assets/certs/ there are no error messages in the container logs and I have debug turned on.

What can I do to investigate?

daufinsyd commented 2 years ago

Hi :)

Did anyone managed to add new .schema after the db is created ? @kevdogg did you found a workaround may be ?

kevdogg commented 2 years ago

@daufinsyd Yes it's definitely possible to add schema after db is created since I added at least 2-3 new schemas. Only problem is -- I don't remember exactly how I did it since I've been working with the db for about 1-2 years now since I've done it. The bootstrap method as @christf spoke about above isn't going to work since the db is already created and the bootstrap is only valid when first starting the container.

I believe the following will work - however I'd create a backup of the database and config before trying, since I don't exactly remember how completed the process. Looking back at some of my notes (which unfortunately weren't as thorough as they should have been), here's what I came up with and I believe it should give you a start on the process:

In order to add any schema, you either need to convert the .schema file to its corresponding .ldif equivalent, or the schema might be distributed in a ldif equivalent form. If you have the ldif equivalent (for example here is the sudo schema ldif as an example - https://github.com/Lullabot/openldap-schema/blob/master/sudo.ldif) you would add the module like (I'm running this command within the actual container itself) - ldapadd -Y EXTERNAL -H ldapi:/// -f <name of .ldif file>

If you need to convert the schema file to an ldif file, the process is a little convoluted but doable -- process is documented here: https://www.zytrax.com/books/ldap/ch6/slapd-config.html#use-schemas. Another link I found useful was: https://www.cyrill-gremaud.ch/how-to-add-new-schema-to-openldap-2-4/. And yet another video -- which kind of runs through one of the scenarios explained in the documentation (if you like to learn via videos): https://www.youtube.com/watch?v=qAedVMMunk8.

Please note I'm doing a bind mount for my volumes:

    volumes:
      ...
      - /data/ldap/config:/etc/ldap/slapd.d

When messing around with config I really find ApacheDirectoryStudio (https://directory.apache.org/studio/) helpful since it gives you a graphical representation of what is going on. You don't specifically need this tool, however its in some cases a lot quicker than querying the database directory.
Other useful commands (While in the container): List all installed schemas:

ldapsearch -Q -Y EXTERNAL -H ldapi:/// -b cn=schema,cn=config cn

Verify config as valid (While in the container):

slaptest -F /etc/ldap/slapd.d -v

gjvoosten commented 1 year ago

I don't know where the (maybe utf-8) characters are coming from. It's literally the 1st non-comment entry in the msuser.ldif. I grepped through all the olcAttributeTypes and they aren't complicated. I downloaded the file directly to my CentOS8 dev env, then copy it into the image from my Dockerfile.

After some experimenting, it appears the file is simply too large… :scream: I downloaded https://git.openldap.org/openldap/openldap/-/raw/master/servers/slapd/schema/msuser.ldif and split it into 5 chunks named 00-msuser.ldif through 04-msuser.ldif:

$ wc -l *
   996 00-msuser.ldif
  1003 01-msuser.ldif
   997 02-msuser.ldif
  1004 03-msuser.ldif
   299 04-msuser.ldif
  4299 total

Then I ran:

$ docker run --volume `pwd`:/container/service/slapd/assets/config/bootstrap/schema/custom osixia/openldap:latest --copy-service -l debug

and all was well. So the bug is that apparently the container can't read files above a certain size.

Matt-CyberGuy commented 1 month ago

Hey guys, I'm trying to figure out how to make this work as well. I have a firewall that has a directory connector for pulling down users and groups. I'd like to use this for authenticating vpn users... but when I place the msuser.ldif file in /container/service/slapd/assets/config/bootstrap/schema/custom nothing seems to change, our connector fails to be able to connect unless I use the default format for openldap... but even when I do that, users and groups do not enumerate.

osixia / docker-openldap

Add .schema files to openldap #354