Unable to Load Package by Id

[ElanHasson]

Thank you for putting this together. I've deployed it to Kubernetes and have storage backed by AzureFile.

I was able to publish a package, but I can't seem to get it out.

I get an error 500 when I attempt to FindPackagesById(). Perhaps I got the syntax wrong? http://xxx:9011/api/v2/FindPackagesById()/$count?includePrerelease=false&$top=160&id=%27PACKAGEID%27&includeAllVersions=false

Thanks

[tomzo]

Hi,

The v2 search is implemented only as far as was needed for the nuget cli, dotnet cli and paket clients, which are tested continuously on each commit. I did not test it with visual studio. Yes, the syntax is probably "wrong", but there is no official documentation on what is the right one. All work is reverse-engineering based on clients queries and with some help of the obsolete code in nuget.server v2. There was someone asking about visual studio earlier on the chat. I have no idea why visual studio makes so different queries than other clients. But my bet is that each version of VS will make those differently. I am not up for implementing all that. Probably the right way to handle VS would be to implement V3 API, which is now documented.

[ElanHasson]

No worries.

Last night I implemented push and download of packages via the nuget client APIs from the official packages. To say it was trivial would be an understatement 😭

I did the push against the V2 API and downloads against the V2/V3 LiGet endpoints and they worked great!

I'm hosting this in Azure and need to be able to scale the solution, so backing the package store on an Azure File share makes the most sense as it allows multiple readers and writers. I think this works for the local packages, but I am not familiar with DBreeze and it's story around concurrent writers to the database.

As far as running as a cache, I am curious as to why you did not choose to just add the cached packages to the local server and expire them from there versus using DBreeze.

A thought I've had is to store the cached packages as local package and expire out of the package store may make more sense as that may sidestep any concurrent writer issues DBreeze may have.

[tomzo]

Last night I implemented push and download of packages via the nuget client APIs from the official packages. To say it was trivial would be an understatement :sob:

I think that's main point to stay away from V2. I initially tried implementing some of V2 search, even tried to approach the ODdata thing, but the supporting libraries (xml-related) do not work on dotnet core.

I'm hosting this in Azure and need to be able to scale the solution, so backing the package store on an Azure File share makes the most sense as it allows multiple readers and writers. I think this works for the local packages,

I did that part by copying implementation from nuget.server. But that depended on obsolete nuget.core, which crashes on dotnet core. So I removed some of references to it, and replaced by new official nuget pakages. There is only little of nuget.core commited in liget's source, but that eventually should be removed.

As far as running as a cache, I am curious as to why you did not choose to just add the cached packages to the local server and expire them from there versus using DBreeze. A thought I've had is to store the cached packages as local package and expire out of the package store may make more sense as that may sidestep any concurrent writer issues DBreeze may have.

I wanted to use a ready KV store with implemented transactions, rather than bare filesystem. DBreeze wasn't the best choice because it doesn't allow to close transaction from different thread than opened it. So basically, now there are 2 transactions opened when a query is made, which is suboptimal.

It is possible to add other backends, either for private repository, or for caching. In the long term, I will probably add openstack swift backend. Adding a filesystem-based store for cache shouldn't be hard. All services have interfaces, which if implemented, you get new backend. If you are up for it, then go ahead.

[ElanHasson]

I will take a look at the code-- presently I'm getting an issue when I attempt to scale the k8s deployment to more than one instance-- I have to check the logs, my hunch is DBreeze is writing some sort of lock file to the cache preventing multiple writers-- but I'll let you know what I find.

IMO offloading the atomic operations to the file (or other storage) system seems like the easiest approach. I'm assuming the cache is caching the actual package, not just the response?

[tomzo]

I have to check the logs, my hunch is DBreeze is writing some sort of lock file to the cache preventing multiple writers-- but I'll let you know what I find.

Yes, I think there is such lock.

I'm assuming the cache is caching the actual package, not just the response?

It is caching both. The .nupkg stream (unchanged from original) and a processed response to some queries (cache responses replace occurences of api.nuget.org with your liget server url).