Profile attributes from LDAP not populating

[joshuaboniface]

I'm using ma1sd backed by an OpenLDAP directory for two main purposes: (1) authentication, and (2) auto-populating of user attributes in their profile.

I have a config as follows (real domains sanitized):

ldap:
  enabled: true
  connection:
    host: "ldap.domain.tld"
    port: 389 
    bindDn: "cn=admin,dc=domain,dc=tld"
    bindPassword: "SuperSecretPassword"
    baseDNs:
      - "o=domains,dc=domain,dc=tld"
  filter: "(&(objectClass=mailUser)(accountStatus=active))"
  attribute:
    uid:
      type: "uid"
      value: "uid"
    name: "cn"
    threepid:
      email:
        - "mail"
      name:
        - "cn"

The user has all the configured attributes like so:

cn: Testy McGee
givenName: Testy
sn: McGee
accountStatus: active
userPassword: {SSHA}abcdefg
mail: test2@maildomain.me
uid: testymcgee        

From my understanding, this should be auto-populating my user attributes based on the values in LDAP, however this is not happening. Specifically, I expect the "cn" "Testy McGee" to be the user's "Display Name" in Element rather than their "uid", and their email to be visible under "Email addresses".

During login, I see the following messages which seem to indicate that the threepid values are found (real domains sanitized):

Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Checking possible match, DN: mail=test2@emaildomain.me,ou=Users,domainName=emaildomain.me,o=domains,dc=domain,dc=tld
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Attempting authentication on LDAP for mail=test2@emaildomain.me,ou=Users,domainName=emaildomain.me,o=domains,dc=domain,dc=tld
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Authentication successful for mail=test2@emaildomain.me,ou=Users,domainName=emaildomain.me,o=domains,dc=domain,dc=tld
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - DN mail=test2@emaildomain.me,ou=Users,domainName=emaildomain.me,o=domains,dc=domain,dc=tld
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Processing 3PIDs for profile
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Processing 3PID type email
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider -         Attribute mail has 1 value(s)
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Processing 3PID type name
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider -         Attribute cn has 1 value(s)
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.backend.ldap.LdapAuthProvider - Found 2 3PIDs
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.auth.AuthManager - @testymcgee:matrixdomain.ca was authenticated by LdapAuthProvider, publishing 3PID mappings, if any
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.auth.AuthManager - Processing io.kamax.matrix.ThreePid@7127fc85 for @testymcgee:matrixdomain.ca
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.invitation.InvitationManager - Looking up possible pending invites for email:test2@emaildomain.me
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.auth.AuthManager - Processing io.kamax.matrix.ThreePid@c938518d for @testymcgee:matrixdomain.ca
Sep 01 00:47:59 synapse0.i.hostdomain.net ma1sd[277816]: [XNIO-1 task-1] INFO io.kamax.mxisd.invitation.InvitationManager - Looking up possible pending invites for name:testy mcgee

But when I log in, neither the Display Name nor Email addresses are populated as expected:

Am I just missing something really obvious here, or could this be a bug?

This has happened on both 2.3 and 2.5.

[janonym1]

Have you configured ma1sd to sync your ldap data with your matrix account when you login? I am not seeing a login policy set, It should look something like:

    config:
      policy:   #for config/policy/threepid update and login rules, see also https://github.com/ma1uta/matrix-synapse-rest-password-provider#profile-auto-fill
        all:
          threepid:
            update: true
            replace: true #overwrite existing 3PIDs with LDAP data
        registration:   #what to do, when FIRST login happens and matrix account gets created
          profile:
            name: true   #set displayname
            update: true   #update displayname
            replace: true   #overwrite (if existing) with LDAP source
        login:   #what to do, each time the user logs in
          profile:
            name: true
            update: true
            replace: true

I found this config at the old mxisd and used it, since ma1sd is a fork of it. However, we had mixed results with the (sometimes) undocumented features like this. For example, to correctly show the email-addresses within the matrix profile in element, we need to restart ma1sd often because of some timeout error (which never got fixed)

[joshuaboniface]

@janonym1 That sounds likely precisely the missing bit. I have nothing like that. Am I correct to presume it should look like this ins my homeserver.yaml:

password_providers:
  - module: "rest_auth_provider.RestAuthProvider"
    config:
      endpoint: "http://<internal IP to ma1sd>:8090"
      policy:
        all:
          threepid:
            update: true
            replace: true
        registration:
          profile:
            name: true
            update: true
            replace: true
        login:
          profile:
            name: true
            update: true
            replace: true

Or does it go somewhere else?

[janonym1]

like described here https://github.com/ma1uta/matrix-synapse-rest-password-provider#profile-auto-fill the relevant config policy settings are within ma1sd ldap part, so you just add the config policy part to you ma1sd config, e.g.:

[...ma1sd_config...]
ldap:
  enabled: true
  connection:
    host: "ldap.domain.tld"
    port: 389 
    bindDn: "cn=admin,dc=domain,dc=tld"
    bindPassword: "SuperSecretPassword"
    baseDNs:
      - "o=domains,dc=domain,dc=tld"
  filter: "(&(objectClass=mailUser)(accountStatus=active))"
  attribute:
    uid:
      type: "uid"
      value: "uid"
    name: "cn"
    threepid:
      email:
        - "mail"
      name:
        - "cn"
  config:
    policy:
      all:
        threepid:
          update: true
          replace: true #overwrite existing 3PIDs with LDAP data
      registration:   #what to do, when FIRST login happens and matrix account gets created
        profile:
          name: true   #set displayname
          update: true   #update displayname
          replace: true   #overwrite (if existing) with LDAP source
      login:   #what to do, each time the user logs in
        profile:
          name: true
          update: true
          replace: true
[...ma1sd_config...]

the config part is a part of the ldap part of the ma1sd config, so take care of your intendation there

most of this is still v1 though, so I am not sure how well it works. ma1sd seems to be not maintained for a long time now

[joshuaboniface]

Well the plot thickens - that seems to have helped in that it's now trying, but I get the following:

Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,576 - rest_auth_provider - 78 - INFO - POST-33 - User @testeymcgee:matrixdomain.ca already exists, registration skipped
Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,576 - rest_auth_provider - 81 - INFO - POST-33 - Handling profile data
Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,576 - rest_auth_provider - 95 - INFO - POST-33 - Display name was not set because it was not given or policy restricted it
Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,577 - rest_auth_provider - 99 - INFO - POST-33 - Handling 3PIDs
Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,577 - rest_auth_provider - 106 - INFO - POST-33 - Looking for 3PID email:test2@emaildomain.me in user profile
Sep 01 10:21:32 synapse0.i.hostdomain.net matrix-synapse[836]: 2022-09-01 10:21:32,577 - rest_auth_provider - 119 - INFO - POST-33 - 3PID is present, skipping

And still nothing set in the profile.

Blast these undocumented features :sweat_smile:

[janonym1]

it says "Display name was not set because it was not given or policy restricted it", are you sure you set it correctly? you specifically need to set the following (because your user already exists, it seems):

      login:   #what to do, each time the user logs in
        profile:
          name: true
          update: true
          replace: true

also, it seems most of this is experimental, so your mileage may vary: https://github.com/ma1uta/ma1sd/blob/master/docs/features/experimental/profile.md

[joshuaboniface]

Yep, those are definitely set. Though it seems like it's the REST provider expecting them, not ma1sd itself, so having that config in the ma1sd.yaml seems incorrect. I'm going to keep experimenting with the code and see if I can get to the bottom of it.

[janonym1]

I am using the ansible playbook made by slavi where the REST stuff is easily configured. of course your https://github.com/ma1uta/matrix-synapse-rest-password-provider#synapse-rest-password-provider must be configured

its possible I misunderstood the config part though, I need to recheck that

edit: according to the (old) docs, I think it should be set inside ma1sd: https://github.com/ma1uta/ma1sd/blob/master/docs/features/registration.md#example

[joshuaboniface]

Unfortunately it seems a bit more of an in-depth problem than just configs.

The call in the REST provider to yield store.set_profile_displayname(localpart, display_name) doesn't seem to be doing anything - it never updates the DB. But, given that this is no longer an ma1sd issue but a REST provider issue, I'm going to close this out. Thanks so much for your help @janonym1!

[joshuaboniface]

And now that I know what to look for: https://github.com/ma1uta/matrix-synapse-rest-password-provider/issues/6

[joshuaboniface]

And for the partial final resolution, to keep it all in one place for the Mighty Google to find:

I was using a tagged version of the REST password provider, 0.1.4, but it seems the actual fix for this (https://github.com/ma1uta/matrix-synapse-rest-password-provider/pull/8) is only in their master right now. Updating to the master version seems to have solved it, at least for the Display Name part. The email 3PID still isn't showing up in Element, but it is present in the Database, so there's something else wonky there, but I'll keep digging to see if I can find out what.

[janonym1]

have you tried restarting ma1sd? it losed the DB connection after a few hours and then it doesnt show the email in the profile

[joshuaboniface]

Yea multiple times while testing, and it doesn't make a difference.

[joshuaboniface]

Email ended up being due to me not having outbound emailing configured in the Homeserver. It's still not showing up, but slowly progressing through the layers.

[joshuaboniface]

And the last bit seemed to be enabling the v2 Matrix API and setting up hashing. Now the emails show, though using them to find other users doesn't work properly but I don't care too much about that.