Can I combine DirSync Cookie with paging? Or some save-point functionality?

arora-kushal commented 2 years ago

I am using DirSync Control Cookie(https://docs.microsoft.com/en-us/windows/win32/ad/polling-for-changes-using-the-dirsync-control) to get the latest changes using the below technique. Is it possible to get that result with pagination ?

Example: If 500 updates have happened, can i get updates from 1-50, or 51-100 (paging with skip) ?

According to this very old article (https://docs.microsoft.com/en-us/archive/blogs/isrpfeplat/using-the-dirsync-control), it seems that it is not possible. If so, Do we have any workaround?

Alternatively, Can I have some save point like functionality? Like If I get 800 records, and lets say I have processed 200 records and my code crashes. Then, can I get next 600 records only (not 800) using the same previous cookie?

dotnet-issue-labeler[bot] commented 2 years ago

I couldn't figure out the best area label to add to this issue. If you have write-permissions please help me learn by adding exactly one area label.

arora-kushal commented 2 years ago

Tagging the area owners @buyaa-n @joperezr @steveharter @BRDPM @grubioe @jay98014

ghost commented 2 years ago

Tagging subscribers to this area: @dotnet/area-system-directoryservices, @jay98014 See info in area-owners.md if you want to be subscribed.

Issue Details

I am using DirSync Control Cookie(https://docs.microsoft.com/en-us/windows/win32/ad/polling-for-changes-using-the-dirsync-control) to get the latest changes using the below technique. Is it possible to get that result with pagination ? Example: If 500 updates have happened, can i get updates from 1-50, or 51-100 (paging with skip) ? According to this very old article (https://docs.microsoft.com/en-us/archive/blogs/isrpfeplat/using-the-dirsync-control), it seems that it is not possible. If so, Do we have any workaround? Alternatively, Can I have some save point like functionality? Like If I get 800 records, and lets say I have processed 200 records and my code crashes. Then, can I get next 600 records only (not 800) using the same previous cookie?

Author:	arora-kushal
Assignees:	-
Labels:	`area-System.DirectoryServices`, `untriaged`
Milestone:	-

vdailly commented 2 years ago

Using the DirSyncRequestControl control it is unknown how many results would be returned by the search until. You have to loop indefinitely until the server return a DirSyncResponseControl control with DirSyncResponseControl.MoreData == false. The server can return 0 entries in a loop but the search is not yet done. The DirSyncResponseControl control returned by the server on each loop contains a DirSyncResponseControl.Cookie which is an opaque structure. Nevertheless the content of the cookie is very close of the replUpToDateVector attribute stored on the NC. It does contains replication information with most of the replication partner for this NC in the domain. DirSyncRequestControl and PageResultRequestControl works very differently. Under the hood DirSyncRequestControl calls the domain controller replication APIs.

I wouldn't recommend to yield results or process them unless the complete search loop is done. The purpose of a search with a DirSyncRequestControl is to obtain only latest changed records. Instead run the query frequently on a frequently updated domain controller to get minimal records to process in a row.

So to answer your questions:

If 500 updates have happened, can i get updates from 1-50, or 51-100 (paging with skip) ? Not really, unless you store the latest processed item. But you cannot guarantee that objects may be ordered the same on next query, especially if the connection connects to another domain controller. In a full meshed Active Directory environnement the DirSyncResponseControl.Cookie value store replication information about all the domain controllers, so that you get only latest updated records whatever the domain controller you connect to. In a non-full meshed Active Directory environnement, it is possible that the search start from scratch even while providing a Cookie value and so your saved index will be wrong.
Alternatively, Can I have some save point like functionality? Like If I get 800 records, and lets say I have processed 200 records and my code crashes. Then, can I get next 600 records only (not 800) using the same previous cookie? If you get 800 records in a loop (among 2000 by example), and you process the 800, then you may use the intermediate DirSyncResponseControl.Cookie value returned by the server to get the remaining 1200. This should work. If you fail to process the 800 records in a loop, then you must restart with the previous cookie to ensure data integrity.

Here a very minimal example to search on a domain controller with a DirSyncRequestControl.

var con = new LdapConnection("server");
con.Bind();

var req = new SearchRequest();
var dirsync = new DirSyncRequestControl();
req.Controls.Add(dirsync);

bool moredata = true;
while(moredata) {
    var res = (SearchResponse)con.Send(req);
    dirsyncres = (DirSyncResponseControl)res.Controls[0]
    dirsync.Cookie = dirsyncres.Cookie
    moredata = dirsyncres.MoreData
}

joperezr commented 2 years ago

@jay98014 @kumarravi do you mind taking a look at this one?

kumarravi commented 2 years ago

Hi Jose

I am super stomped with work right now and am also working from India temporarily. I am sorry I will not be able to look into it for at least a month. I can try after that.

Thanks and Regards Ravi Kumar

On Wed, Mar 23, 2022 at 10:46 AM Jose Perez Rodriguez < @.***> wrote:

@jay98014 https://github.com/jay98014 @kumarravi https://github.com/kumarravi do you mind taking a look at this one?

— Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/63273#issuecomment-1076632660, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQNE64HGBMQSRB2SJKJ6ITVBNKIZANCNFSM5LEP2CQQ . You are receiving this because you were mentioned.Message ID: @.***>

BRDPM commented 2 years ago

Looping @Travis @.***> since this is ADFS space.

Travis - not sure who is on point this month (maybe we should edit our invite to specify) but we should probably create an RFC for tracking this.

From: Ravi Kumar @.> Sent: Thursday, March 24, 2022 5:19 AM To: dotnet/runtime @.> Cc: Cliff Fisher @.>; Mention @.> Subject: Re: [dotnet/runtime] Can I combine DirSync Cookie with paging? Or some save-point functionality? (Issue #63273)

Hi Jose

I am super stomped with work right now and am also working from India temporarily. I am sorry I will not be able to look into it for at least a month. I can try after that.

Thanks and Regards Ravi Kumar

On Wed, Mar 23, 2022 at 10:46 AM Jose Perez Rodriguez < @.<mailto:@.>> wrote:

@jay98014 https://github.com/jay98014 @kumarravi https://github.com/kumarravi do you mind taking a look at this one?

- Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/63273#issuecomment-1076632660, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQNE64HGBMQSRB2SJKJ6ITVBNKIZANCNFSM5LEP2CQQ . You are receiving this because you were mentioned.Message ID: @.<mailto:@.>>

- Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fruntime%2Fissues%2F63273%23issuecomment-1077567805&data=04%7C01%7CCliff.Fisher%40microsoft.com%7C7a76e3ca36a547cbe41008da0d908a34%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837211696442150%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=Apw184v0PSXAltFK8NTeCE41NmEU4nkoRtNvJRy0Xcs%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVUHGVZNKMYDANKAJJ3XXX3VBRMU5ANCNFSM5LEP2CQQ&data=04%7C01%7CCliff.Fisher%40microsoft.com%7C7a76e3ca36a547cbe41008da0d908a34%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837211696442150%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=iASX%2B0NnTvnV7FdTvurk3lsxP1o3J1kAlwBAQY6lLAM%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

BRDPM commented 2 years ago

It looks like vdaillyhttps://github.com/vdailly provided a good response regarding available options on 1/13. Would it be possible to get elaboration on what add'l information is required?

Thanks, Travis

From: Cliff Fisher @.> Sent: Thursday, March 24, 2022 12:29 PM To: dotnet/runtime @.>; dotnet/runtime @.>; Travis Collavo @.> Cc: Mention @.***> Subject: RE: [dotnet/runtime] Can I combine DirSync Cookie with paging? Or some save-point functionality? (Issue #63273)

Looping @Travis @.***> since this is ADFS space.

Travis - not sure who is on point this month (maybe we should edit our invite to specify) but we should probably create an RFC for tracking this.

From: Ravi Kumar @.**@.>> Sent: Thursday, March 24, 2022 5:19 AM To: dotnet/runtime @.**@.>> Cc: Cliff Fisher @.**@.>>; Mention @.**@.>> Subject: Re: [dotnet/runtime] Can I combine DirSync Cookie with paging? Or some save-point functionality? (Issue #63273)

Hi Jose

I am super stomped with work right now and am also working from India temporarily. I am sorry I will not be able to look into it for at least a month. I can try after that.

Thanks and Regards Ravi Kumar

On Wed, Mar 23, 2022 at 10:46 AM Jose Perez Rodriguez < @.<mailto:@.>> wrote:

@jay98014 https://github.com/jay98014<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fjay98014&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275552470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9uNAWboz6fffz4J%2BpYlRIs7CGRoTrrBBwrlkMnUAL%2B4%3D&reserved=0> @kumarravi https://github.com/kumarravi<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fkumarravi&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275552470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=2lTpCGk%2F1xSk1ZLx%2FAw%2FfaHvuQW0x4VdJXp5DdH6Y0s%3D&reserved=0> do you mind taking a look at this one?

- Reply to this email directly, view it on GitHub https://github.com/dotnet/runtime/issues/63273#issuecomment-1076632660<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fruntime%2Fissues%2F63273%23issuecomment-1076632660&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275552470%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=%2F2K2NM1PZ9HqtJuioLSwo%2B5ERrW7NL3OqVmvLQcfTmQ%3D&reserved=0>, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQNE64HGBMQSRB2SJKJ6ITVBNKIZANCNFSM5LEP2CQQ<https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FABQNE64HGBMQSRB2SJKJ6ITVBNKIZANCNFSM5LEP2CQQ&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275602481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=9O74XtffGDtvlzA4PXdD8CZo%2Bb%2BoLEDSrUHm3b7Y3MU%3D&reserved=0> . You are receiving this because you were mentioned.Message ID: @.<mailto:@.>>

- Reply to this email directly, view it on GitHubhttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fdotnet%2Fruntime%2Fissues%2F63273%23issuecomment-1077567805&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275602481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=rQ4Sjx753ZkZfkGTUGT3E03bBusVH5dQkk0M3%2BIDsJY%3D&reserved=0, or unsubscribehttps://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAVUHGVZNKMYDANKAJJ3XXX3VBRMU5ANCNFSM5LEP2CQQ&data=04%7C01%7Ctcolla%40microsoft.com%7C66a1311186b046802e1b08da0dcc83d9%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C637837469275602481%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=FkTGCnZlzS1BxBFfI47TnOILQ4zLcHhnmbPaAI9UcO8%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.**@.>>

arora-kushal commented 2 years ago

Hi @BRDPM , @vdailly indeed provided very good explaination on whatever is available.

I am here requesting to provide additional functionality which is not supported yet.

Paging with skip functionality. As per the comment it looks like in order to achieve this items should always be returned in exact same order. Always.
Currently, DirSync Cookie is a opaque structure. Can we somehow build our own Dirsync cookie with information like (old DirSync cookie (or some static info) + new timestamp) ?

vdailly commented 2 years ago

The cookie is your timestamp. It is a point in time about the last time you performed DirSync search against a specific domain controller (and its associated highest USN at the time of your request). Additionnaly, the cookie contains information about the current domain controller replication partners. Each time you perform the request, you need to backup the cookie in a file or anywhere you want. You can serialize/deserialize as base64 string if you want.

byte[] cookie = LoadFromFileOrAnywhere();
var dirsync = new DirSyncRequestControl(cookie);

DirSync searches are bound to a Naming Context, you cannot start your search anywhere in the tree (unlike with PageResultRequestControl), and generally speaking requires the Replicate Directory Changes right granted on the NC (unless you use the option DirectorySynchronizationOptions.ObjectSecurity on the control).

Imagine 2 domains controllers DC A and DC B, with DC A having the highestCommittedUSN: 12458 (this is an attribute of the RootDSE).

Expanding base 'DC=domain,DC=local'...
Getting 1 entries:
Dn: DC=domain,DC=local
...
replUpToDateVector: dwVersion: 2, dwReserved1: 0, V2.cNumCursors: 1, V2.dwReserved2: 0, rgCursors: 
    {uuidDsa: 12ebc578-aa4c-4d2c-a986-c5606b85db9, usnHighPropUpdate: 589654, timeLastSyncSuccess: 04/15/2020 13:42:06}, 
...

cNumCursors represent the number of replication partners (available, shutdown or even not existing anymore), and rgCursors the list of each replication partner. When the first DirSync request is performed on DC A, then the last Cookie returned by the server will contain:

DC A TimeStamp: 13/04/2022 14:05:20 highestCommittedUSN: 12458
the replUpToDateVector of DC A containing the most up to date value known for DC B commited on DC A.

You can refer to Microsoft documentation on the DirSync pseudo-code to get more detailed information. Even with that some details are not well documented.

The next time you perform the search you must provide the saved Cookie to the DirSyncRequestControl, everything depends on which domain controller you connect:

if the next search you perform is against DC A, then DC A will return all changed entries since the last highestCommittedUSN: 12458. If highestCommittedUSN is now 12558, then there is 100 attribute updates on DC A.
if the next search you perform is against DC B, then DC B will use the replUpToDateVector of DC A to get the usnHighPropUpdate value of itself DC B, so that DC B (which has now a highestCommittedUSN of 589664) will return only 10 attributes updates. These updates may be some updates that you already received initially from DC A that DC B is replicating and commiting.

The mechanism ensure that you don't miss any update. But this doesn't mean you may not see some updates multiple times, especially when connecting to alternate domain controllers very frequently.

You can code something equivalent using highestCommittedUSN and uSNCreated / uSNChanged attributes on objects and using the basic PageResultRequestControl. Anyway you may need to include the ShowDeletedControl if you want to detect deleted objects. PageResultRequestControl returns only "living" objects while DirSyncRequestControl include also most recent deleted objects. And you may parse yourself replUpToDateVector if you want failover with replication partners. But using this strategy you don't know which attributes have changed exactly on an object. You get only the complete object (all requested attributes) unless you dig into attributes metadata replPropertyMetaData. But replPropertyMetaData is a constructed attribute you can request only with a Base search scope, meaning for each object returned by a PageResultRequestControl search you would need to search again using the base scope (with no guarantee the object hasn't been moved or deleted in between). That's where the DirSync control is much more powerful but it may not be appropriate for each use case.

For some cases, the not well known DirSyncEx control available with from Windows Server 2012 instructs the search to include always some desired attributes. By default, only the DistinguishedName and the objectguid attribute are returned. When requesting some attributes, you can add the suffix ;dirSyncAlwaysReturn. By example objectclass;dirSyncAlwaysReturn would always return the objectclass attribute, which may be desired. In a classical DirSync search, the objectclass is returned only the first time (first request or newly created object on subsequent searches) because it does never change (unless adding some auxiliary class to the object by example ...).

Forward-Linked attributes (like member attribute on groups) have their own subtilities, depending the usage of the option DirectorySynchronizationOptions.IncrementalValues. The forwrd link attribute is not returned as the requested member but with a suffix member;range=0-0 and member;range=1-1 or something like that in the attribute name. Few information is available but match the LinkStamp function. But attribute;range=0-0 are the deleted values and attribute;range=1-1 the added values. This is especially useful if you handle very large groups that are updated frequently.

In any case, while powerful I would reserve the DirSync control to very large directories or advanced use cases.

dotnet / runtime

Can I combine DirSync Cookie with paging? Or some save-point functionality? #63273