search

SDKits / ExamineX

Issue tracker for ExamineX
https://examinex.online
5 stars 0 forks source link

No Index Events (Transforming Index, Create Update) Are fired #103

Closed r-modica-cti closed 3 months ago

r-modica-cti commented 3 months ago

Describe the bug Not sure this is necessarily a bug or something has changed but the documentation doesn't really give any specifics on how this should be done specifically with ExamineX and newer versions of Umbraco which send us down a notification handler path for populating a custom index (which speaking with the ExamineX team directly is not required) but this issue we are having extends to the CreatingOrUpdatingIndex event which also doesn't seem to be fired.

The baseline issue is when using a Composer on Umbraco 13, no registered events for a custom index are fired even though the Initialize method is hit and the delegated events are being setup.

The code below is a spike so some fields will not make sense, but we are trying to get a baseline implementation ready to act as a guide for our team to implement.

License ID 617c0799-7cef-477d-bc1c-e1e121768e39

To Reproduce

Registering a custom index:

{
    /// <summary>
    /// Set the custom search index
    /// </summary>
    public static void RegisterSearchIndexes(this IServiceCollection services)
    {
        services
            .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>(SearchSettings.Indexes.Editorial)
            .ConfigureOptions<EditorialIndexOptions>();
    }
}

with the following options:

    public class EditorialIndexOptions : IConfigureNamedOptions<AzureSearchIndexOptions>
    {
        private readonly IOptions<IndexCreatorSettings> _settings;

        public EditorialIndexOptions(IOptions<IndexCreatorSettings> settings)
            => _settings = settings;

        public void Configure(string? name, AzureSearchIndexOptions options)
        {
            if (name?.Equals(SearchSettings.Indexes.Editorial) is false)
            {
                return;
            }

            options.FieldDefinitions = new (
                new(EditorialIndexSettings.RootSiteId, AzureSearchFieldDefinitionTypes.Integer),
                new (EditorialIndexSettings.Id, AzureSearchFieldDefinitionTypes.Integer),
                new (EditorialIndexSettings.Name, AzureSearchFieldDefinitionTypes.FullText),
                new (EditorialIndexSettings.Name, AzureSearchFieldDefinitionTypes.FullText)
            );
        }

        // not used
        public void Configure(AzureSearchIndexOptions options) => throw new NotImplementedException();
    }

With the following composer:

The Composer is hit, the initalize method is run and you can see it hitting the event handler delegation, but none of those events are ever hit, the index is not populated and even re-indexing does not hit those events. the documentation for all this is a little fragmented between Umbraco / Examine and Examine X and it is really difficult to figure this out. 

public class EditorialIndexerComponent : IComponent
{
    private readonly IExamineManager _examineManager;
    private readonly IUmbracoContextFactory _umbracoContextFactory;

    public EditorialIndexerComponent(IExamineManager examineManager,
        IUmbracoContextFactory umbracoContextFactory)
    {
        _examineManager = examineManager ?? throw new ArgumentNullException(nameof(examineManager));
        _umbracoContextFactory =
            umbracoContextFactory ?? throw new ArgumentNullException(nameof(umbracoContextFactory));
    }

    public void Initialize()
    {
         if (_examineManager.TryGetIndex(SearchSettings.Indexes.Editorial, out var editorialIndex)
                && editorialIndex is UmbracoContentAzureSearchIndex azureIndex)
         {
            azureIndex.TransformingIndexValues += IndexerComponent_TransformingIndexValues;
            azureIndex.CreatingOrUpdatingIndex += IndexerComponent_CreatingOrUpdatingIndex;
        }
    }

    private void IndexerComponent_CreatingOrUpdatingIndex(object sender, CreatingOrUpdatingIndexEventArgs e)
    {
        //var brand = e.AzureSearchIndexDefinition.Fields.FirstOrDefault(x => x.Name == "brand");
        //brand.IsFacetable = true;
    }

    private void IndexerComponent_TransformingIndexValues(object sender, IndexingItemEventArgs e)
    {
        if (int.TryParse(e.ValueSet.Id, out var nodeId))
        {
            var validTypes = new List<string>()
            {
                "newsPost",
                "publicationPost",
                "webinar"
            };

            var isValidType = validTypes.Any(x => x.Equals(e.ValueSet.ItemType, StringComparison.OrdinalIgnoreCase));

            if (!isValidType)
            {
                return;
            }

            using (var umbracoContext = _umbracoContextFactory.EnsureUmbracoContext())
            {
                var contentNode = umbracoContext.UmbracoContext.Content?.GetById(nodeId);

                if(contentNode == null)
                {
                    return;
                }

                var updatedValues = e.ValueSet.Values.ToDictionary(x => x.Key, x => x.Value.ToList());
                var rootSiteId = contentNode.Root().Id;

                updatedValues.TryAdd(EditorialIndexSettings.RootSiteId, new List<object> { rootSiteId });
                updatedValues.TryAdd(EditorialIndexSettings.Id, new List<object> { contentNode.Id });
                updatedValues.TryAdd(EditorialIndexSettings.Name, new List<object> { contentNode.Name });
                updatedValues.TryAdd(EditorialIndexSettings.DocumentType, new List<object> { contentNode.ContentType.Alias });

                e.SetValues(updatedValues.ToDictionary(x => x.Key, x => (IEnumerable<object>)x.Value));
            }
        }
    }

    public void Terminate()
    {
    }
}

Expected behavior We would expend the events to be triggered so we can step into the code and correctly setup the indexes.

Versions

Additional context To tap into some of the additional features provided by Azure Search we need custom indexes for specific scenarios such as making sure the field definitions are correct or for things like suggesters which need to be created on index creation.

Shazwazza commented 3 months ago

@r-modica-cti

My first suggestion is to start with the basics:

  1. Get your custom index working with normal Examine/Lucene.
  2. Then get this working with ExamineX/AzureSearch.

... You need to get step 1 working first.

I have this working but there is a bug in Umbraco itself (see comments below). And as you've noted, the Umbraco docs for a custom index aren't very clear - their docs for creating a custom index is based on custom data (i.e. your own database), not just using Umbraco data.

Please be sure to read all code comments below.

Creating a new Umbraco content based index

/// <summary>
/// Set the custom search index
/// </summary>
public static void RegisterSearchIndexes(this IServiceCollection services)
{
    services
        // Add a new UmbracoContentIndex to be based on Umbraco content/media
        .AddExamineLuceneIndex<UmbracoContentIndex, ConfigurationEnabledDirectoryFactory>("EditorialIndex")

        // Enroll this index with ExamineX
        .AddExamineXAzureSearchIndexFromUmbraco<UmbracoAzureSearchContentIndex>("EditorialIndex")

        // Configure the Examine/Lucene settings for working locally
        .ConfigureOptions<LuceneEditorialIndexOptions>()

        // Configure the ExamineX/AzureSearch specific settings (if required)
        .ConfigureOptions<AzureSearchEditorialIndexOptions>();
}

Configure the new index with normal Examine settings

You would do this so that you can work locally with normal Examine/Lucene

public class LuceneEditorialIndexOptions : IConfigureNamedOptions<LuceneDirectoryIndexOptions>
{
    private readonly IUmbracoIndexConfig _umbracoIndexConfig;

    public LuceneEditorialIndexOptions(IUmbracoIndexConfig umbracoIndexConfig)
    {
        _umbracoIndexConfig = umbracoIndexConfig;
    }

    public void Configure(string? name, LuceneDirectoryIndexOptions options)
    {
        if (name?.Equals("EditorialIndex") is false)
        {
            return;
        }

        // IMPORTANT: This is the bug in Umbraco. If this is not set, you will get a NullReferenceException when
        // rebuilding your index - which you will see in your logs.
        options.Validator = _umbracoIndexConfig.GetContentValueSetValidator();

        // Configure custom fields.
        // NOTE: These will AUTOMATICALLY be converted/used when this index is enrolled with ExamineX/AzureSearch
        options.FieldDefinitions.AddOrUpdate(new("editorialRootSiteId", FieldDefinitionTypes.Integer));
        options.FieldDefinitions.AddOrUpdate(new("editorialId", FieldDefinitionTypes.Integer));
        options.FieldDefinitions.AddOrUpdate(new("editorialName", FieldDefinitionTypes.FullText));
    }

    // not used
    public void Configure(LuceneDirectoryIndexOptions options) => throw new NotImplementedException();
}

Configure ExamineX specific settings

This is optional, all of the above settings will be automatically converted to ExamineX settings. However, in some cases you might need to have specific settings applied for ExamineX/Azure Search.

/// <summary>
/// Specific configuration for the ExamineX index
/// </summary>
public class AzureSearchEditorialIndexOptions : IConfigureNamedOptions<AzureSearchIndexOptions>
{
    public void Configure(string? name, AzureSearchIndexOptions options)
    {
        if (name?.Equals("EditorialIndex") is false)
        {
            return;
        }

        // Configure custom fields.
        // NOTE: This is NOT necessary because these will have already been converted automatically by ExamineX
        // when the index is enrolled with the AddExamineXAzureSearchIndexFromUmbraco method.
        // For example, if you breakpoint here, you will see that there are already 3 field definitions with the correct types.
        options.FieldDefinitions.AddOrUpdate(new("editorialRootSiteId", AzureSearchFieldDefinitionTypes.Integer));
        options.FieldDefinitions.AddOrUpdate(new("editorialId", AzureSearchFieldDefinitionTypes.Integer));
        options.FieldDefinitions.AddOrUpdate(new("editorialName", AzureSearchFieldDefinitionTypes.FullText));
    }

    // not used
    public void Configure(AzureSearchIndexOptions options) => throw new NotImplementedException();
}

Add event handlers

If you want to make more granular changes to the ExamineX/AzureSearch index. For example, to make a field facetable.

if (examineMgr.TryGetIndex("EditorialIndex", out var index))
{
    // bind to event (regardless of whether Examine/ExamineX is used)
    index.TransformingIndexValues += Index_TransformingIndexValues;

    if (index is AzureSearchIndex examineXIndex)
    {
        examineXIndex.CreatingOrUpdatingIndex += ExamineXIndex_CreatingOrUpdatingIndex;
    }
}
private void Index_TransformingIndexValues(object sender, IndexingItemEventArgs e)
{
    // TODO: transform the values
}
private void ExamineXIndex_CreatingOrUpdatingIndex(object sender, CreatingOrUpdatingIndexEventArgs e)
{
    var brand = e.AzureSearchIndexDefinition.Fields.FirstOrDefault(x => x.Name == "brand");
    if (brand is not null)
    {
        brand.IsFacetable = true;
    }
}
r-modica-cti commented 3 months ago

Hi @Shazwazza,

The events side of things, where is that setup, is that in a composer?

Shazwazza commented 3 months ago

Wherever you want. It can be done in a composer, or in Startup.Configure. It makes no difference.