NuGet.Server: Adding support for SQL Server (and other database backends)

[NickCraver]

One of the things we really want at Stack Overflow is a simple internet NuGet feed without external dependencies. This has a few legs to it, we want to:

• Use our existing SQL Server high availability (not add a new thing we need to failover)
• Have HA on the NuGet feed inside a single data center (not skew based on when timers hit like NuGet.Server today)
• Read-through caching so a build server never goes to the internet (this will be a separate issue - just including here)
  • Example: on a miss it goes to nuget.org, grabs the package, caching it locally on the way back. The next fetch is local.
• Publish our own internal-only packages to the feed

NuGet.Server

NuGet.Server is the simplicity and minimal setup and dependencies we want, but is still ultimately file based and therefore has inherent limitations:

• Skew between servers (or DFS shares)
• Overall speed, as reading full packages from disk is expensive to rebuild indexes and such
• Scale, because of the above

NuGetGallery

The obvious question: why not just use Gallery?

• The gallery has the same file store dependencies as server, but for Lucene searching.
• The gallery is updated a lot, for internal deployment this ins't a win because all we need is a very simple v2 feed. Of all the changes to NuGetGallery, how many of those actually impact the v3 feed (or any feed at all)? A relatively small portion. Since the gallery has lots of UI and other concerns, the amount of churn, updates, code to certify, etc. is a huge ratio to relevant changes just to host a feed.
• Setup is quite a bit more complex, even locally - this is a one time cost but it illustrates the weight of Gallery vs. what's actually needed.
• I want to be clear: Gallery is great, it's just not a great fit.

So: SQL Server. I played with what's needed to make this happen and it's not all that much. I have a working version of this on a branch here: NickCraver/NuGet.Server (craver/sql-server branch) with most of the relevant pieces in this commit: https://github.com/NickCraver/NuGet.Server/commit/96953922c5a645f24548345388797dfe44b0056a

Here's a view on the current state of things:

_{Note: the Packages table has PackageData directly in it (again - just a first pass), this returns too much data for most things. It's quite easy to make this another table and I'll do that next.}

It's a first pass with a few pitfalls, but all of them are straightforward to remedy. This is just the rest of a few nights getting a feed up. It's already fully functional, but needs a few global changes to be workable:

• .NET 4.6 -> .NET 4.6.1 min version, to support netstandard2.0 EF Core is built on.
• DI needs request scope (the current DI is very simple, but a data context really should be request scope - it's currently long lived because of this limitation). This is bad because in a multi-server scenario not checking the DB per request (looking at the materialized context instead) would by desynced and error-prone. Pretty much any DI swap would support this.
• Some generally useful helpers should moved around.

Overall note: While this issue and current code supports SQL Server, if we can figure out the dependency issue EF Core can very easily be swapped out for MySQL, SQLite, Postgres, etc. - anything that EF Core supports as the usage is very basic. Perhaps making the provider itself a plugin somehow would make things like NuGet.Server.MySQL an option. I'm not sure how that would work...so I'm starting the discussion here with SQL Server.

cc @joelverhagen

[lahma]

I love everything here and would like to replace our NuGet server with less disk-based feed - SQL Server is already natural backup target. The transparent mirror is great too.

As a current user I'd like to see easy migration path. We have a folder full of package names with subfolders for versions...

[NickCraver]

@lahma That reminds me! Currently the file system watcher where you can drop files into the \Packages folder resides in the ServerPackageRepository (and therefore is only active when the backend file store is in use), but IMO, is a distinct piece of functionaity.

There's no reason to tie the folder drop functionality to the ultimate backend store. If we moved this (and had some event handlers on the thing perhaps), it'd just pickup files and drop them into whatever backend store is in use. The leads me to migration: while not a one-click solution "just copy your packages here and we'll suck them in" is still pretty appealing as an easy-to-do option serving multiple use cases.

@joelverhagen thoughts here? I could tackle moving it into a shared place as part of this proposal if welcomed.

[joelverhagen]

thoughts here? I could tackle moving it into a shared place as part of this proposal if welcomed.

Love the idea. This would also facilitate improved test coverage of that component... It's a bit fiddly right now. I tried this in the past but ran into pain. Definitely worth another stab.

Thus far it seems like upstream and decouple file system watcher belong in the main packages: NuGet.Server or one of I two dependencies.

And the EF integration belongs in NuGet.Server.Database or some such name.

[ghost]

In case others are looking for a hosted option that already does a lot of this, you can check out VSTS NuGet which gives you geo-redundancy private feeds with the read-through caching you mention.

Funny enough, our project started out with doing exactly what you are proposing - I remember hitting the same DI scoping issue you're talking about. However, we ended up going in different direction. Unfortunately, we were doing it on an internal SQL framework that makes the code we wrote not particularly helpful to your pursuit.