Issue with indexer creation - datasource validation

[ihorleleka]

Azure.RequestFailedException: Data source name must only contain lowercase letters, digits or dashes, cannot start or end with dashes and is limited to 128 characters. Status: 400 (Bad Request)

at Azure.Search.Documents.IndexersRestClient.CreateOrUpdate(String indexerName, SearchIndexer indexer, String ifMatch, String ifNoneMatch, CancellationToken cancellationToken) at Azure.Search.Documents.Indexes.SearchIndexerClient.CreateOrUpdateIndexer(SearchIndexer indexer, Boolean onlyIfUnchanged, CancellationToken cancellationToken) at ExamineX.AzureSearch.AzureSearchIndex.A(Boolean , SearchIndex , SearchIndexer , SearchIndex& , SearchIndexer& ) at ExamineX.AzureSearch.AzureSearchIndex.D() at ExamineX.AzureSearch.AzureSearchIndex.a(Boolean ) at ExamineX.AzureSearch.AzureSearchIndex.CreateIndex() at ExamineX.AzureSearch.Umbraco.UmbracoAzureSearchIndex.CreateIndex()

[JamieTownsend84]

Hi @ihorleleka - we've had this issue on a site a few days ago, currently have a Microsoft support ticket open. We found moving away from UK South to UK West worked. Not ideal but a solution until a solution :)

[Shazwazza]

I've been able to replicate this issue. New versions with a fix will be released today. It's to do with new Azure validation which seems to be rolling out region to region (i.e. USWest is not affected yet).

The problem is:

• ExamineX uses an indexer to store field mappings and by default uses a non-existent data source named "NULL"
• "NULL" violates the new explicit validation that's been rolled out

The fix is to change this to a valid name, so "none" has been chosen.

The APIs allow creating an indexer with a non-existent data source, but now the name needs to adhere to the validation rules. If you are using the BlogMedia package and have everything configured correctly, then this issue doesn't exist and the indexer data source name will be mapped to: your-site-name-indexname-blob-media

[Shazwazza]

New versions will be released today with a fix:

• 2.0.10
• 3.0.7
• 4.1.1

[Shazwazza]

These versions have now been published.

[daniel-eriksson]

Can't find the 2.1.0 version in Nuget, is it the v 2.0.10 that has this fix? Couldn't find any information about this release, https://github.com/SDKits/ExamineX/releases?page=1

[Shazwazza]

@daniel-eriksson sorry that was a typo, it's 2.0.10 released on the same date as 3.0.7 and 4.1.1

[BarryFogarty]

Hi @Shazwazza 👋 - I'm still experiencing this issue after updating to v2.0.12, specifically where a field is being added to the index via the TransformingIndexValues event handler. This is using Umbraco v8.18.7. Is it possible that something to do with this method may have missed out on the previous update?

Re our setup: When a fresh index is created, and rebuilt via Umbraco Examine dashboard, the custom field gets indexed properly, but the above Azure.RequestFailedException: Data source name.. error still occurs when saving a document that triggers a TransformingIndexValues event handler. In this case, indexing still actually works fine and does not have any tangible effect on functionality. When the code is introduced to an existing index, the error means the custom field is not added to the index. NB When running on standard filesystem Examine, the issue doe not arise.

We are using a standard IComponent with standard Examine API methods to add the field to the ValueSet like so:

public void Initialize()
{
    // index type and name checking removed

    _indexProvider = _examineManager.IndexProviderFor(Constants.UmbracoIndexes.ExternalIndexName);

    _indexProvider.TransformingIndexValues += ConfigureSearchableLocationField;
}

public void Terminate()
{
    _indexProvider.TransformingIndexValues -= ConfigureSearchableLocationField;
}

private void ConfigureSearchableLocationField(object sender, IndexingItemEventArgs e)
{
    // logic removed
        e.ValueSet.Add("locationPath", string.Join(" ", locations));
}

Let me know if I can provide more info, thanks!

[Shazwazza]

Hey @BarryFogarty! Is this the exact same issue here where the indexer cannot be created because its specifying an invalid (wrong casing) Data Source? By default (unless using the Blob media package addition for ExamineX) the data source name will be "none". This may cause some warnings in your logs because the data source "null" does not actually exist, but this is expected. ExamineX uses the indexer as it's mapping source for field names defined in your Examine code vs what the field names actually are in Azure Search. This warning can be ignored and won't cause any issues. But if the indexer cannot be created, then ExamineX will entirely fail, so this doesn't sound like it's the same issue.

To me, your issue sounds like you are trying to dynamically add a field to a ValueSet and that is not being propogated to Azure Search?

[BarryFogarty]

Thanks - you are correct in your assumptions @Shazwazza - the problem is not a failure to create the indexer, sorry if that was not clear. I am indeed trying to dynamically add a field to a ValueSet when a document is saved in Umbraco. With a fresh index this works and the logged error occurred processing the index batch exception (with the exact same stack trace as above) does not seem to have any adverse impact on the indexing process. To be clear though, it is an actual exception (not just a warning) in the log - but if you're happy this can be ignored in this scenario, that's good enough for me :-)

Just FYI, what I did find was that if the code to add the custom field was introduced to an existing populated index, the new field would never get added. I don't know how you could repro this however, as it could be due to a some other index corruption.

Thanks again for taking a look 👍

[Shazwazza]

Hi @BarryFogarty, can you confirm via the Azure Portal what the JSON structure of your Indexer is? The data source should be set to "none". I re-checked the code and it shouldn't throw/log an exception in this case so if that is happening, this may be another bug if Azure Search is now restricting creating indexers with a data source that doesn't exist. Can you post your log/stack trace just to be sure?

[BarryFogarty]

Thanks @Shazwazza - here is our indexer JSON structure - interestingly we have some with a NULL datasource name and some with 'none' (I believe the 'none' ones have been rebuilt using ExamineX v2.0.12):

{
  "@odata.context": "https://srch-XXX-prod-westeurope.search.windows.net/$metadata#indexers/$entity",
  "@odata.etag": "\"0x8DAD9FB3F1B3556\"",
  "name": "www-redacted-com-external-rebuild",
  "description": null,
  "dataSourceName": "NULL",
  "skillsetName": null,
  "targetIndexName": "www-redacted-com-external-rebuild",
  "disabled": true,
  "schedule": null,
  "parameters": {
    "batchSize": null,
    "maxFailedItems": null,
    "maxFailedItemsPerBatch": null
  },
  "fieldMappings": [
    {
      "sourceFieldName": "__NodeId",
      "targetFieldName": "x__NodeId",
      "mappingFunction": null
    },
    ...
    {
      removed for brevity
    }
  ],
  "outputFieldMappings": [],
  "cache": null,
  "encryptionKey": null
}
}

Interestingly, in both NULL and none cases, the following error exists: The data source is invalid or not found. Please reset the indexer, then update the data source in the JSON tab.

Testing so far (with 'none' indexers) show no indexing errors in the log, and indexing works as expected. I will be able to test with the NULL indexes next week to see if I can repro the original errors again.

[Shazwazza]

Hi @BarryFogarty, yes that warning will always show up in the Azure Portal but indexing will still work. That said, this error shouldn't really get into your logs since that would mean something has failed. Yes, the "NULL" data source name is invalid though and may cause you problems. Rebuilding that index with the latest ExamineX will set this to "none" which is a valid name and shouldn't give you issues. You can also manually change "NULL" to "none" by editing the JSON directly in the azure portal.

My own example:

[BarryFogarty]

Cheers @Shazwazza - that indexer definition setup is the same as ours in the Azure Portal with ref to the datasource name and the warning message. I cannot repro my original issue now on any rebuilt index (running ExamineX v2.0.12), so I'm putting the previous occurrence of the indexer exceptions in the log down to the use of indexes that had not been rebuilt since the version update. I'll keep an eye but hopefully I won't have anything else to add. Thanks again for looking into it so thoroughly 🙌

[Shazwazza]

Thanks for reporting @BarryFogarty , glad its all working now :)