Add syncrepl (rfc-4533) consumer (persistent search)

t2y commented 1 year ago

After I discussed it with @cpuschma, I changed the design for maintainability.

Design

I added the syncrepl feature as one of the asynchronous search functions. The syncrepl is available on slapd in OpenLDAP project and it provides a persistent search. The syncrepl is proposed by rfc-4533.

(the below description is the original when I created this PR)

(Old) Design

At first, I tried to implement syncrepl feature into the same source. For example, the searchResponse struct has an asynchronous search process, DecodeControl() function has to decode various controls. However, I am concerned that they are challenging to handle syncrepl communications with other generic LDAP communications.

It has a complicated specification because syncrepl is proposed by rfc-4533. I think dividing existent control.go/response.go and related syncrepl code is stable and maintainable (Of course both belong to the same ldap package). Additionally, they won't put a bug on each other.

control_syncrepl.go
response_syncrepl.go

Data flow

syncrepl has 4 Controls:

Sync Request Control is used for client requests
Sync State Control is used for server responses
- request with mode=RefreshAndPersist, then get a response with ApplicationSearchResultEntry (4)
Sync Done Control is used for server responses
- request with mode=RefreshOnly, then get a response with ApplicationSearchResultDone (5)
Sync Info Control is used for server responses
- request with mode=RefreshAndPersist, then get a response with ApplicationIntermediateResponse (25)

Client                                                  Server
  |                                                      |
  |------ Sync Request Control ------------------------> |
  |                                                      |
  | <-------- Sync Info Message (optional) ------------- |
  |                                                      |
  |      (One or more times for each entry)              |
  | <----- Sync State Control (with Entry) ------------- |
  |                                                      |
  | <-------- Sync Info Message (optional) ------------- |
  |                                                      |
  | <----------- Sync Done Control --------------------- |
  |                                                      |

Controls

Sync Request Control as the following.

Control Type: Sync Request ("1.3.6.1.4.1.4203.1.9.1.1") Criticality: true Mode: 3 Cookie: rid=000,csn=20230719002941.016654Z#000000#000#000000 ReloadHint: false

Sync Request with mode=RefreshAndPersist

Got entries and Sync Info Control as the following.

Control Type: Sync Info ("1.3.6.1.4.1.4203.1.9.1.4") Criticality: false Value: 1 RefreshDelete[Cookie: rid=000,csn=20230719074010.612296Z#000000#000#000000 RefreshDone: true]

Control Type: Sync Info ("1.3.6.1.4.1.4203.1.9.1.4") Criticality: false Value: 3 SyncIdSet[Cookie: rid=000,csn=20230719072347.153542Z#000000#000#000000 RefreshDeletes: true SyncUUIDs: [f1688779-d7f3-4c20-9580-24dcb21149a9 1ee6cb16-86c5-41a7-a52a-9168f1f24703 3d11877e-a030-4007-aafe-c94bb0284fdb]]

When a user is added

Got the entry and a Sync State Control such as the following.

Control Type: Sync State ("1.3.6.1.4.1.4203.1.9.1.2") Criticality: false State: 1 EntryUUID: c033cfc4-89cb-4571-9088-6e62be597266 Cookie: rid=000,csn=20230719062607.286081Z#000000#000#000000

When a user is modified

Got the entry and a Sync State Control such as the following.

Control Type: Sync State ("1.3.6.1.4.1.4203.1.9.1.2") Criticality: false State: 2 EntryUUID: 33a88143-1334-4f6b-853c-27239af30c54 Cookie: rid=000,csn=20230719062553.176851Z#000000#000#000000

When a user is deleted

Got the entry and a Sync State Control such as the following.

Control Type: Sync State ("1.3.6.1.4.1.4203.1.9.1.2") Criticality: false State: 3 EntryUUID: 1ee6cb16-86c5-41a7-a52a-9168f1f24703 Cookie: rid=000,csn=20230719071122.501306Z#000000#000#000000

Sync Request with mode=RefreshOnly

Got entries and a Sync Done Control such as the following.

Control Type: Sync Done ("1.3.6.1.4.1.4203.1.9.1.3") Criticality: false Cookie: rid=000,csn=20230719002941.016654Z#000000#000#000000 RefreshDeletes: true

How to test

To enable syncrepl on the server side, add the below configuration into slapd.conf.

overlay syncprov

Then, call Syncrepl() method described in examples_test.go. I confirmed syncrepl feature works with OpenLDAP server 2.4/2.5 in my environment.

I need more test patterns or more considerations since I'm new to LDAP protocol. Could you review it?

References

t2y commented 1 year ago

https://github.com/go-ldap/ldap/pull/446 will fix DATA RACE with GetLastError(). It is not an error due to this PR.

t2y commented 1 year ago

I rebased to include #446 for CI (github actions).

t2y commented 1 year ago

@cpuschma @johnweldon I've been waiting for this feature since March 2023 (#422). I have added the syncrepl feature so that the existing process has no side effects. What do you think?

johnweldon commented 1 year ago

I agree with @cpuschma; thank you for your work on this feature.

My involvement with this project is very limited now, so I'm okay with deferring to the other @go-ldap/committers as far as requested changes.

My overarching values are:

Keeping the public surface area of this library as limited as possible (without sacrificing new, useful features)
Striving for comprehensive code test coverage

With this in mind, my vote is to approve this MR

t2y commented 1 year ago

@johnweldon Thank you for reviewing! I agree with your idea.

And then, I will try to implement this PR without duplicating the code this weekend. I think it's possible, but it might be a little complex. I will carefully integrate my new code to prevent the side effect.

go-ldap / ldap