DataChangeDetectionPolicy loses column name in serialisation

[rossb-mmt]

• @azure/search-documents:
• 11.2.1:
• Windows:
• [ ] nodejs
  • 16.3.0:
• [ ] typescript
  • 4.0.0:
• Is the bug related to documentation in
  • [ ] README.md
  • [ ] source code documentation
  • [ ] SDK API docs on https://docs.microsoft.com

Describe the bug When creating or updating a DataSourceConnection to a cosmosdb data source, when you pass e.g. a HighWaterMarkChangeDetectionPolicy it gets serialised as a DataChangeDetectionPolicy, thereby losing the highWaterMarkColumnName field and causing an error in the api:

A valid high water mark column name must be specified for this change detection policy.

To Reproduce Steps to reproduce the behavior:

1. With an existing Azure Cognitive Search service, create a new SearchIndexerClient Then run:

   await searchIndexerClient.createOrUpdateDataSourceConnection(
               {
                   name: "my-datasource-name",
                   type: "cosmosdb",
                   connectionString: "...",
                   container: { 
                       name: "my_container" 
                   } ,
                   dataChangeDetectionPolicy: {
                       odatatype: "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
                       highWaterMarkColumnName: "_ts"
                    }
               }
           )

   The actual body of the html message sent the REST API is:

   {"name":"my-datasource-name","type":"cosmosdb","credentials":{"connectionString":"..."},"container":{"name":"my_container"},"dataChangeDetectionPolicy":{"@odata.type":"#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy"}}

Expected behavior The column name should be serialised

Additional context The behaviour is also incorrect with DataDeletionDetectionPolicy where it also serialises without the optional fields softDeleteColumnName and SoftDeleteMarkerValue, even when they are present.

[azure-sdk]

Label prediction was below confidence level 0.6 for Model:ServiceLabels: 'Cosmos:0.24762703,Search:0.1694629,Storage:0.14159'

[sarangan12]

@rossb-mmt, Thanks for raising the issue. I have validated the scenarios with the latest version of search-documents SDK (11.3.0-beta.7) with the following code:

// Copyright (c) Microsoft Corporation.
// Licensed under the MIT license.

/**
 * @summary Demonstrates the DataSource Connection Operations.
 */

import {
  SearchIndexerClient,
  AzureKeyCredential,
  SearchIndexerDataSourceConnection,
  HighWaterMarkChangeDetectionPolicy,
} from "@azure/search-documents";

import * as dotenv from "dotenv";
dotenv.config();

const endpoint = process.env.ENDPOINT || "";
const apiKey = process.env.SEARCH_API_ADMIN_KEY || "";
const connectionString = process.env.CONNECTION_STRING || "";
const dataSourceConnectionName = "example-ds-connection-sample-1";

async function createDataSourceConnection(
  dataSourceConnectionName: string,
  client: SearchIndexerClient
) {
  console.log(`Creating DS Connection Operation`);
  const dataSourceConnection: SearchIndexerDataSourceConnection = {
    name: dataSourceConnectionName,
    description: "My Data Source 1",
    type: "cosmosdb",
    container: {
      name: "my-container-1",
    },
    connectionString,
  };
  await client.createDataSourceConnection(dataSourceConnection);
}

async function getAndUpdateDataSourceConnection(
  dataSourceConnectionName: string,
  client: SearchIndexerClient
) {
  console.log(`Get And Update DS Connection Operation`);
  const ds: SearchIndexerDataSourceConnection = await client.getDataSourceConnection(
    dataSourceConnectionName
  );
  ds.container.name = "Listings_5K_KingCounty_WA";
  ds.dataChangeDetectionPolicy = {
    odatatype: "#Microsoft.Azure.Search.HighWaterMarkChangeDetectionPolicy",
    highWaterMarkColumnName: "_ts",
  };
  console.log(`Updating Container Name of Datasource Connection ${dataSourceConnectionName}`);
  await client.createOrUpdateDataSourceConnection(ds);
}

async function listDataSourceConnections(client: SearchIndexerClient) {
  console.log(`List DS Connection Operation`);
  const listOfDataSourceConnections: Array<SearchIndexerDataSourceConnection> =
    await client.listDataSourceConnections();

  console.log(`List of Data Source Connections`);
  console.log(`*******************************`);
  for (let ds of listOfDataSourceConnections) {
    console.log(`Name: ${ds.name}`);
    console.log(`Description: ${ds.description}`);
    console.log(`Connection String: ${ds.connectionString}`);
    console.log(`Data Change Detection Policy: ${ds.dataChangeDetectionPolicy}`);
    if (ds.dataChangeDetectionPolicy) {
      const highWaterMarkChangeDetectionPolicy =
        ds.dataChangeDetectionPolicy as HighWaterMarkChangeDetectionPolicy;
      console.log(
        `Data Change Detection Policy - HighWaterMarkColumn Name: ${highWaterMarkChangeDetectionPolicy.highWaterMarkColumnName}`
      );
    }
    console.log(`Data Deletion Detection Policy: ${ds.dataDeletionDetectionPolicy}`);
    console.log(`Etag: ${ds.etag}`);
    console.log(`DataContainer`);
    console.log(`\tName: ${ds.container.name}`);
    console.log(`\tQuery: ${ds.container.query}`);
    console.log();
  }
}

async function deleteDataSourceConnection(
  dataSourceConnectionName: string,
  client: SearchIndexerClient
) {
  console.log(`Deleting DS Connection Operation`);
  await client.deleteDataSourceConnection(dataSourceConnectionName);
}

async function main() {
  console.log(`Running DS Connection Operations Sample....`);
  if (!endpoint || !apiKey || !connectionString) {
    console.log("Make sure to set valid values for endpoint and apiKey with proper authorization.");
    return;
  }
  const client = new SearchIndexerClient(endpoint, new AzureKeyCredential(apiKey));
  try {
    await createDataSourceConnection(dataSourceConnectionName, client);
    await getAndUpdateDataSourceConnection(dataSourceConnectionName, client);
    await listDataSourceConnections(client);
  } finally {
    await deleteDataSourceConnection(dataSourceConnectionName, client);
  }
}

main().catch((err) => {
  console.error("The sample encountered an error:", err);
});

I am able to verify that the code changes are working fine and the properties are serialized correctly. Could you please validate with the above code and version? Let me know if you are still blocked.

[rossb-mmt]

Thank you, I can confirm 11.3.0-beta.7 does indeed work correctly.