search

loic-sharma / BaGet

A lightweight NuGet and symbol server
https://loic-sharma.github.io/BaGet/
MIT License
2.62k stars 680 forks source link

Allow PDBs to be stored in a compressed form #154

Open loic-sharma opened 5 years ago

loic-sharma commented 5 years ago

Allow customers that generate large amounts of PDBs to store these in compressed snupkg form. These symbols should be extracted when requested, on the fly. See: https://twitter.com/dotMorten/status/1070402374714281984

Follow up to: https://github.com/loic-sharma/BaGet/issues/130

LordMike commented 5 years ago

You could gzip compress pdbs and store them. If users come in with Accept-Encoding: gz, serve the file raw, else de-/recompress and serve it.

dotMorten commented 5 years ago

@LordMike Oh now you're just trying to be clever :-)

loic-sharma commented 5 years ago

We could do something for nupkgs too, but note that not all NuGet clients support gzip. Gzip was introduced in v3.4.0 of the registration resource. Edit: nuget.org gzips JSON responses, not the nupkgs themselves.

LordMike commented 5 years ago

Additonal option, add caching to the unpacked files. The few files actually requested are stored in ready-to-consume form, those that aren't are permanently placed in archives.

Does a client request files from CDN's directly, or are they redirected to it from the Nuget services?

loic-sharma commented 5 years ago

Right now, I'm leaning towards gzipping over storing snupkgs. Storing snupks introduces complexity since we need to store metadata in the database.

Does a client request files from CDN's directly, or are they redirected to it from the Nuget services?

You can do both. nuget.org services content directly from CDN, Azure DevOps uses redirection.

dotMorten commented 5 years ago

note that not all NuGet clients support gzip

It isn't the NuGet client who's requesting the symbols but Visual Studio doing it.

loic-sharma commented 5 years ago

@dotMorten Sorry, my comment wasn't clear. I meant that we could gzip nupkgs as well.

LordMike commented 5 years ago

Wouldn't gain you much. Assuming the zip file is already compressed with Deflate, which Gzip also uses :)

You could recompress the nupkg on ingestion, ensuring that it actually is compressed and that it's compressed for storage (high compression).

Doing that could also remove extranous artifacts in the nupkg - imagine if somebody embedded Eicar as a random data file in their nupkg :P. I'm unsure how Nupkg signing works, so this may or may not work.

dotMorten commented 5 years ago

A agree that would just be double-compression. Not much point. And please don't modify the nuget packages - then I wouldn't know what package I'm testing, of it the repacking caused some issues.

loic-sharma commented 5 years ago

You guys are right. I'm being silly, don't mind me 😋

LordMike commented 5 years ago

@dotMorten do you need the package to be identical to the one you uploaded?

I can get the argument that if the rewriting missed a crucial file, it would be bad, but then all files in a nupkg should be specced out, so no files should exist beyond those defined. Right?