search

bonsai-rx / bonsai

The compiler, IDE, and standard library for the Bonsai visual programming language for reactive systems
https://bonsai-rx.org
MIT License
136 stars 28 forks source link

Add support for filtered package feeds #1603

Open glopesdev opened 10 months ago

glopesdev commented 10 months ago

Motivation

To ensure a smooth transition from existing community and official MyGet feeds into the NuGet Gallery, we need to introduce built-in support for the nuget.org feed to the package manager, and provide package filter mechanisms to approximate the level of curation currently done by the MyGet feeds.

Detailed Design

The NuGet v3 API provides several package filter mechanisms allowing constraining existing package source results by using either author, tags or package type filters which can be leveraged by the package manager.

Proposal 1: Package Filter Checkbox

This option is a straightforward check to enable/disable filtering of packages from any feed, similar to the "Include prerelease" checkbox. As a first pass, this could be implemented by appending package filter tags to the query, e.g. Bonsai Rx.

Proposal 2: Feed-specific Filters

As an alternative to, or in addition to, the above filter proposal, we could introduce feed-specific filters, for example to be able to easily curate packages published by the bonsai organization for the official "Bonsai Packages" feed.

Proposal 3: Merge MyGet feeds with NuGet community feed

Migration from the MyGet feed to nuget.org is expected to take several months, if not years. However as space is running out on MyGet we need to start the transition as soon as possible. A useful first step would be to coalesce both the nuget.org feed and the Community Packages feed into a single curated package source. This way it would be easier to start redirecting developers to publish their packages on nuget.org, and mark the MyGet community feeds as read-only.

Proposal 4: Mark example packages with a specific package type

Example packages do not by themselves contain any reusable bits, but are used as self-contained applications which are copied on open to a new folder. Their installation and usage is very different from normal packages. At first glance they seem like a good candidate for setting NuGet package types to improve both client behavior and server package search.

Bear in mind that filtering for package types is currently not surfaced (or incorrectly surfaced) via the NuGet client API: https://github.com/NuGet/Home/issues/8915

Package Types

Operator Package

Gallery Package

PathogenDavid commented 1 month ago

Proposal 1: Package Filter Checkbox

Is there a practical reason for being able to install arbitrary NuGet packages in Bonsai? (IE: Is there even a reason to provide the checkbox? Would someone ever want to just install Newtonsoft.Json directly?)

Obviously downstream dependencies would always appear in the updates tab.

package filter tags to the query

Does designating a package type block a package from being consumed as a normal NuGet package?

I wonder if this transition might also be a good excuse to encourage explicit tagging of packages intended as something a Bonsai user would install. A specific package type feels more robust than a tag by convention.

Proposal 2: Feed-specific Filters

Another thought similar to this: We could introduce the concept of "Recommended Packages". Have packages commonly used float to the top of the package browser with a little star or something (or maybe a filter for recommended packages.)

We could even allow packages to provide their own recommendations. So if you install Contoso.Bonsai it might automatically add Contoso.Bonsai.Graphics to your recommendations.

Could maybe even offer to install recommendations when you install a top-level package kinda like apt.

(This might be deviating too far from NuGet purity though.)

Proposal 3: Merge MyGet feeds with NuGet community feed

Is this something that actually takes any effort on our part? The All package source is the default, isn't it?

if not years

Hopefully not! I like to hope the .NET modernization will create pressure to encourage maintainers to update their packages and upload them to the appropriate place. (Or if there's OSS packages that are unmaintained but still valuable we do that modernization and reupload ourselves for the sake of the community.)

glopesdev commented 1 month ago

Is there a practical reason for being able to install arbitrary NuGet packages in Bonsai? (IE: Is there even a reason to provide the checkbox? Would someone ever want to just install Newtonsoft.Json directly?)

Yes, to be able to use package dependencies when scripting local project extensions in C#.

For example, it is possible to install MathNet.Numerics on your Extensions.csproj and use it in the code of your local script. The script itself will be compiled into the Extensions assembly and loaded at boot-time, but the MathNet.Numerics assembly still has to be found and loaded by Bonsai for the code to function. Currently the way this is done is by requiring any script dependencies to also be installed and declared in Bonsai.config, which requires the ability to install arbitrary packages from the package manager.

A different way to do this would be to read all script dependencies declared in Extensions.csproj at boot-time and load them directly from the NuGet package cache itself. This is discussed briefly in #1477 and is part of a bigger discussion on project files, dependencies, and environments, which we do want to have but most likely would be actionable only for 3.0.

Does designating a package type block a package from being consumed as a normal NuGet package?

No, and in fact the documentation for custom package types starts by explicitly discussing the scenario of custom extensions we are interested in, so package types definitely feel like the way to go.

The only reason we did not adopt them earlier is they are a relatively new feature, and unfortunately the MyGet server never got to the point of supporting them. I think starting from 3.0 we should definitely make it mandatory to mark all packages with a BonsaiPackage or BonsaiExtension package type, and we can also start packaging all the official packages like that, but we will need an interim solution for community and legacy 2.x packages.

Another thought similar to this: We could introduce the concept of "Recommended Packages".

This is definitely an interesting idea and might solve some of the problems we have currently with people forgetting to install the Design package corresponding to a package they have just installed and then struggling to find editors and visualizers. This should probably be turned into a separate issue, and I would love to discuss some options on how to implement this.

Is this something that actually takes any effort on our part? The All package source is the default, isn't it?

The current default is the curated official feed of Bonsai packages, but I agree maybe you are right and we just drop nuget.org in the package source list and instruct people to go to the All feed for third-party packages. If we incorporate some of the other proposed solutions for filtering packages I agree this one could indeed be dropped.

Or if there's OSS packages that are unmaintained but still valuable we do that modernization and reupload ourselves for the sake of the community

This is something we could indeed do. We tried making a call for this in the discussions following a MyGet outage last year and some people did reply, but there are several packages which remain "orphaned". We could potentially create a temporary NuGet account bonsai-community and use that to upload all community packages and then offer to transfer ownership of packages. Alternatively we can drop the MyGet feed on 3.0 and let people reach out explicitly.

PathogenDavid commented 1 month ago

Yes, to be able to use package dependencies when scripting local project extensions in C#.

Ah yes, of course. I knew I was forgetting something!

Inferring it from the Extensions.csproj is definitely an attractive proposition, but I agree that's a more complicated discussion.

This should probably be turned into a separate issue, and I would love to discuss some options on how to implement this.

Done! https://github.com/bonsai-rx/bonsai/issues/1954

The current default is the curated official feed of Bonsai packages

Ah, it turns out that the "All Packages" list is the default only when you don't have the MyGet feed configured. I don't think we should remove NuGet.org from the package source dropdown, but making the all feed the default might work.

As an aside, the Bonsai packages-only feed could just be a meta feed similar to the all-feed rather than a checkbox.

Alternatively we can drop the MyGet feed on 3.0 and let people reach out explicitly.

I love me a good scream test 😁 Worst case people just manually re-add the legacy feed.

That being said I think making the bonsai-community NuGet.org organization would be a reasonable idea. (We don't need it to be a user, you can make a package which is only owned by an org.)