With S3 as Sleet source, choco cannot list more than 30 packages

emgarten / Sleet

A static nuget feed generator for Azure Storage, AWS S3, and more.

MIT License

361 stars 43 forks source link

With S3 as Sleet source, choco cannot list more than 30 packages #200

Closed prsshini closed 1 month ago

prsshini commented 2 months ago

Our S3 bucket source contains 1612 packages with 194 folders (with different versions of nupkgs in each folder). Choco source is set as Sleet source The threshold of 1,000 packages per source has been met. Please refine your search, or specify a page to find any more results.

PS C:\Users\vradhs2> choco source Chocolatey v2.2.2 test - https://s3.amazonaws.com/bucket-name/index.json | Priority 0|Bypass Proxy - False|Self-Service - False|Admin Only - False.

Choco search command can only list 30 packges . Rest of the packages in the S3 bucket is not getting listed. Chocola 30 packages found. The threshold of 1,000 packages per source has been met. Please refine your search, or specify a page to find any more results.

We tried choco search --page=100 and choco search --page-size 100 All of them have the same behavior to list only 30 packages.

Could you let me know if any sleet settings that is preventing to list all the packages?

emgarten commented 2 months ago

Sleet doesn't limit the search results, you can verify this by looking at the /search/query file in your feed to ensure it has the expected packages.

The 30 limit is likely coming from the NuGet protocol library chocolatey is using which limits it to 30 entries per page. Sleet being static only has a single page of search results.

You can workaround this by using an external search provider that supports paging: https://github.com/emgarten/Sleet/blob/main/doc/external-search.md

prsshini commented 1 month ago

I hosted the sleet.search in the awsapigateway and setup the feedsettings as mentioned here: https://github.com/emgarten/Sleet/blob/main/doc/external-search.md

But it recommends to recreate the package after the feedsettings is set to externalsearch. Is this recreation mandatory? recreate downloads all the packages and our packages are pretty big and a lot to download and upload everytime the feedsetting is set. Can we re-index the packages using init after we set the feedsetting?

emgarten commented 1 month ago

It may work without recreating, are you seeing accurate search results?

Search itself would not be impacted by the recreate, it only reads the feed.

prsshini commented 1 month ago

it does work without recreating. But after setting up the feed-settings, it recommends to recreate. Just wanted to know if its mandatory or recommended for any specific purpose.

emgarten commented 1 month ago

Normally after changing the settings the feed would need to be recreated for the settings to take effect, I think you're seeing the generic message from that.

In this case search is unique, the setting change only switches the url so the new search endpoint is used.

prsshini commented 1 month ago

After setting up the sleet feed settings, the choco client forwards the request to the external search endpoint as expected without having to run the recreate. so it seems like a generic message and safe to ignore the recreate command after feedsetting setup.