ldapsync crashes on objectGUID error

VeselaHouba commented 3 years ago

After upgrading to docker.seadrive.org/seafileltd/seafile-pro-mc:8.0.2 following issue appears when trying to sync from LDAP

./seafile-server-latest/pro/pro.py ldapsync

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 57, in run
    migrate_dn_pairs(settings=self.settings)
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 46, in migrate_dn_pairs
    uuid = results[0][1][config.group_uuid_attr][0]
KeyError: 'objectGUID'

Fiddled a bit with /opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py. My Groups don't have objectGUID at all.

            print(results[0])
            if (not results or not results[0][1]):
                continue
            else:
                uuid = results[0][1][config.group_uuid_attr][0]
                add_group_uuid_pair(grp_dn_pair.group_id, uuid)
....
('cn=SomeGroupName,cn=groups,dc=domain,dc=com', {})

But it just started crashing somewhere else in ldap_group_sync.py.

Workaround

config.group_uuid_attr is not documented at all. But blind shot worked and I set GROUP_UUID_ATTR = gidNumber. In ccnet.conf. Then I was able to run the sync without crashing. I guess this is not correct as it fills gidNumber into GroupIdLDAPUuidPair table, but whatever.

The funny part is, that objectGUID is NOT Group UuID, but Globally Unique IDentifier and AFAIK it's used only in AD, not openLDAP or similar.

128-bit value that is unique not only in the enterprise but also across the world.

https://docs.microsoft.com/en-us/previous-versions/windows/it-pro/windows-2000-server/cc961625(v=technet.10)

feiniks commented 3 years ago

Hello, you can download the latest version 8.0.4-pro, which should solve this issue.

VeselaHouba commented 3 years ago

Not available on docker repo yet, latest one is 8.0.2. @feiniks any idea what's the release cycle there?

feiniks commented 3 years ago

Hello,it seems that the new version has not been released yet, we will release it later.Sorry, I’m not sure about the release cycle.

VeselaHouba commented 3 years ago

Ok, I'll wait until new image appears and will test & confirm if the issue is resolved.

VeselaHouba commented 3 years ago

Nope, problem still persists on 8.0.4

root@fb34909188c1:/opt/seafile# ./seafile-server-latest//pro/pro.py ldapsync
Exception in thread Thread-1:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 56, in run
    migrate_dn_pairs(settings=self.settings)
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 45, in migrate_dn_pairs
    uuid = results[0][1][config.group_uuid_attr][0]
KeyError: 'objectGUID'

Exception in thread Thread-2:
Traceback (most recent call last):
  File "/usr/lib/python3.6/threading.py", line 916, in _bootstrap_inner
    self.run()
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 56, in run
    migrate_dn_pairs(settings=self.settings)
  File "/opt/seafile/seafile-server-latest/pro/python/seafevents/ldap_syncer/ldap_sync.py", line 45, in migrate_dn_pairs
    uuid = results[0][1][config.group_uuid_attr][0]
KeyError: 'objectGUID'

killing commented 3 years ago

Sorry this is a documentation issue. We forgot to mention the new option in the documentation. Now it's updated.

GROUP_UUID_ATTR: Since Seafile pro 8.0, UUID is used to identify groups in LDAP/AD servers. Before that, group DN is used. So when a group changes name, the old group will be deleted and a new group will be created. This is not a desirable behavior. With the new mechanism, the rename can be detected and the groups in Seafile remains intact. The default attribute is "ObjectGUID", which is available in AD. For other LDAP servers, please refer to https://ldapwiki.com/wiki/Universally%20Unique%20Identifier .

VeselaHouba commented 3 years ago

Since I don't have any better unique identifier for groups than gidNumber, I guess setting GROUP_UUID_ATTR= gidNumber for non-AD LDAPs should work. Not sure what happens during group renaming, if the gidNumber stays the same, but I don't have environment to test it.

killing commented 3 years ago

What LDAP server do you use? Is EntryUUID attribute available, as it's RFC standard? But since you've setup with gidNumber, it's not advisable to change to another. The migration only works for the first time.

VeselaHouba commented 3 years ago

openldap/slapd managed by Univention UCS. You are correct, entryUUID is available in extended attributes. I'll give it a try.

VeselaHouba commented 3 years ago

Hmm there's still something broken

When I clean all groups from seafile and run the script for the first time everything seems to work

...
... More debug output
...
[06/24/2021 12:24:24] [DEBUG] create group 2289, and add uuid pair cn=MEMBER_CN<->2289 success.
[06/24/2021 12:24:24] [DEBUG] add member MEMBER_UID to group 2289 success.
[06/24/2021 12:24:24] [INFO] LDAP group sync result: add [5]group, update [0]group, delete [0]group

But when I run the script second time, it inserts the same groups again, but this time with no members

root@fb34909188c1:/opt/seafile# ./seafile-server-latest//pro/pro.py ldapsync
[06/24/2021 12:42:50] [INFO] LDAP user sync result: add [0]user, update [0]user, deactive [0]user, add [0]role, update [0]role
[06/24/2021 12:42:50] [INFO] LDAP profile sync result: add [0]profile, update [0]profile, delete [0]profile
[06/24/2021 12:42:50] [INFO] LDAP dept sync result: add [0]dept, update [0]dept, delete [0]dept
[06/24/2021 12:42:53] [WARNING] add group uuid pair 2290<->GROUP1_CN failed.
[06/24/2021 12:42:53] [WARNING] add group uuid pair 2291<->GROUP2_CN failed.
[06/24/2021 12:42:53] [WARNING] add group uuid pair 2292<->GROUP3_CN failed.
[06/24/2021 12:42:53] [WARNING] add group uuid pair 2293<->GROUP4_CN failed.
[06/24/2021 12:42:53] [WARNING] add group uuid pair 2294<->GROUP5_CN failed.
[06/24/2021 12:42:53] [INFO] LDAP group sync result: add [0]group, update [0]group, delete [0]group

That's also why the group auto increment index is so high, we have only like 5 testing groups, but they are inserted repeatedly with cron.

GroupIdLDAPUuidPair seems to be filled somehow strangely

MariaDB [seahub_db]> select * from GroupIdLDAPUuidPair;
+------+----------+--------------------------------------+
| id   | group_id | group_uuid                           |
+------+----------+--------------------------------------+
| 2253 |       61 | 454d463a-df3c-103a-8a7c-69daecd9b3bf |       <---- matches entryUUID of GROUP_1
| 2255 |       62 | 881228be-df3c-103a-8a8b-69daecd9b3bf |
| 2256 |       63 | a1ad749a-df3c-103a-8a9a-69daecd9b3bf |
| 2257 |       64 | 4b73adee-82d8-103a-95a8-0d011e9cf1a6 |
| 2258 |     2285 | GROUP1_CN, but truncated to 36 characters |
| 2259 |     2286 | GROUP2_CN, but truncated to 36 characters |
| 2260 |     2287 | GROUP3_CN, but truncated to 36 characters |
| 2261 |     2288 | GROUP4_CN, but truncated to 36 characters |
| 2262 |     2289 | GROUP5_CN, but truncated to 36 characters |

killing commented 3 years ago

Have you cleaned up GroupDNPair table in ccnet_db? Make sure all tables related to group in ccnet_db are cleaned up.

VeselaHouba commented 3 years ago

I've now completely cleaned all Group related tables

ccnet_db.Group
ccnet_db.GroupDNPair
ccnet_db.GroupUser
seahub_db.GroupIdLDAPUuidPair

And then imported groups with

./seafile-server-latest/pro/pro.py ldapsync

First run seemed ok, but following entries were created in seahub_db.GroupIdLDAPUuidPair

MariaDB [seahub_db]> select * from GroupIdLDAPUuidPair;
+------+----------+--------------------------------------+
| id   | group_id | group_uuid                           |
+------+----------+--------------------------------------+
| 4568 |     4595 | cn=group_1,cn=groups,dc=OMMIT |
| 4569 |     4596 | cn=group_2,cn=groups,dc=OMMIT |
| 4570 |     4597 | cn=group_3,cn=groups,dc=OMMIT |
| 4571 |     4598 | cn=group_4,cn=groups,dc=OMMIT |
| 4572 |     4599 | cn=group_5,cn=groups,dc=OMMIT |
+------+----------+--------------------------------------+

And GROUP_UUID_ATTR is silently ignored for some reason, truncated group DN is used instead

ENABLE_GROUP_SYNC = true
GROUP_OBJECT_CLASS = posixGroup
GROUP_MEMBER_ATTR = memberUid
GROUP_UUID_ATTR = entryUUID
GROUP_FILTER = <OMMIT>
IMPORT_GROUP_STRUCTURE = true
CREATE_GROUP_REPO = true

And on second run, duplicate groups are created again.

@killing Can we move this to private conversation over support channel, and then post only results here? I would like to avoid some unintentional data leak.

haiwen / seafile

ldapsync crashes on objectGUID error #2450

Workaround