[Azure Search] Consider shingling

[loic-sharma]

Target queries:

• "mysql" - The desired second result should be "MySqlConnector"
• "fxcop" - The desired top result should be "Microsoft.CodeAnalysis.FxCopAnalyzers"
• "entityframework" - The desired second result should be "Microsoft.EntityFrameworkCore"
• "youtrack" - The desired first result should be "YouTrackSharp"
• "systemtesting" - The desired first result should be "SystemTestingTools"
• "clrheap" - The desired first result should be "ClrHeapAllocationAnalyzer"
• "streamdeck" - The desired first result should be "StreamDeckSharp"

This MAY be helped:

• ClosedXml - ClosedXML.* packages should be second

[loic-sharma]

Changes here: https://github.com/NuGet/NuGet.Services.Metadata/compare/dev...loshar-shingles?expand=1

Results: this improved "mysql", "fxcop", and "entityframework" queries, however this regressed the "aspnet" query. The queries that were improved are more common, so this seems like a good tradeoff. We think the "aspnet" query may be improved by the "edge dependencies" work, so we want to try that first before continuing this change.

[loic-sharma]

As part of this work, consider folding IdentifierCustomTokenFilter into PackageIdCustomTokenizer.

[loic-sharma]

@chgill-MSFT suggested that we prioritize fields that aren't shingled over shingled fields. This may help the case where searching for aspnet return AspNetCore results instead of AspNet results.