search

microsoft / winget-cli

WinGet is the Windows Package Manager. This project includes a CLI (Command Line Interface), PowerShell modules, and a COM (Component Object Model) API (Application Programming Interface).
https://learn.microsoft.com/windows/package-manager/
MIT License
23.03k stars 1.43k forks source link

Index V2 consumption #4462

Closed JohnMcPMS closed 4 months ago

JohnMcPMS commented 4 months ago

Change

This change adds the ability to actually use the V2 index database. This primarily required fixing the ignored search functionality from the previous PR and updating the ISource implementation to be aware of V2 and how to retrieve the package version manifest files. Additionally, a few places in the code were updated to avoid requesting all of the package versions when we could instead use only the latest version. This prevents the need to retrieve the package version file at all, which is the goal for all read-only operations (search, list, update [aka list but filtered to things with updates]). In reality, we do need to get the package version files when we need to do ARP version mapping, but that constitutes only 5% of the total packages in winget-pkgs currently.

While the V2 usage code is always enabled, an experimental feature (indexV2) is required to attempt to retrieve the V2 index from the CDN. Disabling this feature should naturally transition one back to V1 on the next update attempt.

File caching, based on the known SHA256 hash of the file, is added as a general utility. This is then used for V1 manifest files and also for V2 package version data and manifest files. While the files are generally small, it is not uncommon for them to be used a few times over a short period.

Thanks to @msftrubengu for pointing out that I forgot the include the exports in the .def file in the previous PR. 🥇

Validation

Added many new unit tests and updated many existing unit tests. Updated E2E tests to generate a V2 index in addition to the V1, and added a test pass that runs all of the existing E2E tests using the V2 index.

Microsoft Reviewers: Open in CodeFlow
github-actions[bot] commented 4 months ago

@check-spelling-bot Report

:red_circle: Please review

See the :open_file_folder: files view or the :scroll:action log for details.

Unrecognized words (1)

msz

Previously acknowledged words that are now absent ata bitspace EPester epth hrow issuetitle mapview Mta oop PFM STARTUPINFOW testdata visualstudiocode :arrow_right:
To accept :heavy_check_mark: these unrecognized words as correct and remove the previously acknowledged and now absent words, run the following commands ... in a clone of the [git@github.com:JohnMcPMS/winget-cli.git](https://github.com/JohnMcPMS/winget-cli.git) repository on the `index-v2-use` branch ([:information_source: how do I use this?]( https://github.com/check-spelling/check-spelling/wiki/Accepting-Suggestions)): ``` sh curl -s -S -L 'https://raw.githubusercontent.com/check-spelling/check-spelling/v0.0.21/apply.pl' | perl - 'https://github.com/microsoft/winget-cli/actions/runs/9021448768/attempts/1' ```
Available :books: dictionaries could cover words not in the :blue_book: dictionary This includes both **expected items** (547) from .github/actions/spelling/expect.txt and **unrecognized words** (1) Dictionary | Entries | Covers -|-|- [cspell:win32/src/win32.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/win32/src/win32.txt)|53509|20| [cspell:python/src/python/python-lib.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/python/src/python/python-lib.txt)|3873|3| [cspell:python/src/python/python.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/python/src/python/python.txt)|453|2| [cspell:python/src/common/extra.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/python/src/common/extra.txt)|741|2| [cspell:php/php.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/php/php.txt)|2597|2| [cspell:npm/npm.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/npm/npm.txt)|288|2| [cspell:django/django.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/django/django.txt)|859|2| [cspell:csharp/csharp.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/csharp/csharp.txt)|19|2| [cspell:sql/src/tsql.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/sql/src/tsql.txt)|455|1| [cspell:scala/scala.txt](https://raw.githubusercontent.com/check-spelling/cspell-dicts/v20220816/dictionaries/scala/scala.txt)|833|1| Consider adding them using (in `.github/workflows/spelling3.yml`): ``` yml with: extra_dictionaries: cspell:win32/src/win32.txt cspell:python/src/python/python-lib.txt cspell:python/src/python/python.txt cspell:python/src/common/extra.txt cspell:php/php.txt cspell:npm/npm.txt cspell:django/django.txt cspell:csharp/csharp.txt cspell:sql/src/tsql.txt cspell:scala/scala.txt ``` To stop checking additional dictionaries, add: ``` yml with: check_extra_dictionaries: '' ```
Warnings (1) See the [:open_file_folder: files](4462/files/) view or the [:scroll:action log](https://github.com/microsoft/winget-cli/actions/runs/9021448768/job/24788811225#step:4:1) for details. [:information_source: Warnings](https://github.com/check-spelling/check-spelling/wiki/Event-descriptions) | Count -|- [:information_source: unexpected-line-ending](https://github.com/check-spelling/check-spelling/wiki/Event-descriptions#unexpected-line-ending) | 1 See [:information_source: Event descriptions](https://github.com/check-spelling/check-spelling/wiki/Event-descriptions) for more information.
If the flagged items are :exploding_head: false positives If items relate to a ... * binary file (or some other file you wouldn't want to check at all). Please add a file path to the `excludes.txt` file matching the containing file. File paths are Perl 5 Regular Expressions - you can [test]( https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your files. `^` refers to the file's path from the root of the repository, so `^README\.md$` would exclude [README.md]( ../tree/HEAD/README.md) (on whichever branch you're using). * well-formed pattern. If you can write a [pattern](https://github.com/check-spelling/check-spelling/wiki/Configuration-Examples:-patterns) that would match it, try adding it to the `patterns.txt` file. Patterns are Perl 5 Regular Expressions - you can [test]( https://www.regexplanet.com/advanced/perl/) yours before committing to verify it will match your lines. Note that patterns can't match multiline strings.