nuget install command not using packages already in Cache folder

[LtKlaus]

I ran the following command to get MyPackage from our NuGet server

nuget.exe Install MyPackage -o Packages -ExcludeVersion

This downloaded the package %LocalAppData%\NuGet\Cache and then extracted it to my Packages folder in my project folder.

I then run the same command again and it starts downloading the package from our server again. In my Cache folder I can see MyPackage.1.2.3.nupkg that it downloaded the first time and MyPackage.1.2.3.nupkg.tmp as it's downloading the same package again.

Why wouldn't it use the one that's already in the Cache folder?

When installing package in the NuGet UI in Visual Studio it seems to grab from the Cache folder if the correct version of the package is already there. Why doesn't this happen when calling from the command line?

I was able to get around this by adding %LocalAppData%\NuGet\Cache to my package sources. I can leave it like this if needed but figured that using the Cache would be automatic.

The reason this is an issue for me is that I have to download packages through a VPN connection to one of our offices in another country and the connection is quite slow. If our server has a newer package I'm stuck waiting for the download but don't want to have to wait if the version I need is already in my local Cache.

Thanks in advance for any help.

[yishaigalatzer]

@LtKlaus can you share the version of nuget.exe you are using?

Specifically when you install a package without a version, you are forcing NuGet to go to the server to find the latest version.

However when I tried this with 3.4, I don't see the package download happening in fiddler only access to metadata:

First request (clean cache)

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
2   200 HTTPS   api.nuget.org   /v3/index.json  903 max-age=120, must-revalidate    application/ld+json nuget:12712         
3   200 HTTPS   api.nuget.org   /v3/registration1/microsoft.aspnet.webapi.client/index.json 75,222  no-store    application/json    nuget:12712         
4   200 HTTPS   api.nuget.org   /v3/registration1/newtonsoft.json/index.json    70,466  no-store    application/json    nuget:12712         
5   200 HTTPS   api.nuget.org   /packages/newtonsoft.json.6.0.4.nupkg   1,513,616       binary/octet-stream nuget:12712         
6   200 HTTPS   api.nuget.org   /packages/microsoft.aspnet.webapi.client.5.2.3.nupkg    188,921     binary/octet-stream nuget:12712         

Second request (nupkg in cache)

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
2   200 HTTPS   api.nuget.org   /v3/registration1/microsoft.aspnet.webapi.client/index.json 75,222  no-store    application/json    nuget:13712         
3   200 HTTPS   api.nuget.org   /v3/registration1/newtonsoft.json/index.json    70,466  no-store    application/json    nuget:13712         

[LtKlaus]

I'm using NuGet.exe v3.3.0.212. I can confirm that I'm still able to reproduce this in my environment, but specifically when one of my sources is our internal NuGet server which is running Klondike v2.0.0.

I just created a new C# console application, added the Moq package to it, then used the following command to install the package again

nuget install Moq -o Packages -ExcludeVersion

I see the same issue where Moq.4.2.1510.2205.nupkg is already in my Cache folder but when I run the above command I can see Moq.4.2.1510.2205.tmp in the Cache while it's downloading a new copy.

I understand that not providing a version number will force NuGet to check the server for the latest version. My expectation would be that, NuGet would poll all sources to find what the latest version, then check it's Cache to see if it already has that version before it starts to download it.

If I remove our Klondike server from my sources, I don't see this behaviour anymore. So I'm guessing it has something to do with Klondike. Maybe NuGet.exe ignores it's Cache folder when downloading from a source other than NuGet.org?

My workaround is to keep my own Cache. I run a script to move the packages from %LocalAppData%\NuGet\Cache to C:\LocalCache and I added C:\LocalCache as my first package source. If the latest version of my package is already in C:\LocalCache, it will use that one and skip the download from our server.

[yishaigalatzer]

Can you please share a fiddler trace of your scenario? I have a guess that your package version might not be normalized? That means there might be extra zeros in the string.

If you are using a local feed, please read our blog about accelerating your local network share.

Last - I tried it with 3.4 rather than 3.3, please use that version (available on the download tab on nuget.org)

[LtKlaus]

I did try with 3.4 previously but got the following error so I went back to 3.3

An error occurred while retrieving package metadata for 'Moq' from source 'MyPackageSource'. For security reasons DTD is prohibited in this XML document. To enable DTD processing set the DtdProcessing property on XmlReaderSettings to Parse and pass the settings into XmlReader.Create method.

I ran my tests with 3.3 while Fiddler was running.

I deleted all package from the Cache and ran "nuget install Moq -o Packages". This is what Fiddler logged.

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
1   401 HTTP    tfs-live:5432   /api/   1,293       text/html   nuget:3316          
2   200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:3316          
3   401 HTTP    tfs-live:5432   /api/   341     text/html; charset=us-ascii nuget:3316          
4   307 HTTP    tfs-live:5432   /api/   0           nuget:3316          
5   401 HTTP    tfs-live:5432   /api/odata  1,293       text/html   nuget:3316          
6   401 HTTP    tfs-live:5432   /api/odata  341     text/html; charset=us-ascii nuget:3316          
7   200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:3316          
8   200 HTTP    tfs-live:5432   /api/odata  366     application/atomsvc+xml; charset=utf-8  nuget:3316          
9   401 HTTP    tfs-live:5432   /api/odata/$metadata    1,293       text/html   nuget:3316          
10  401 HTTP    tfs-live:5432   /api/odata/$metadata    341     text/html; charset=us-ascii nuget:3316          
11  200 HTTP    tfs-live:5432   /api/odata/$metadata    4,386       application/xml; charset=utf-8  nuget:3316          
12  401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  1,293       text/html   nuget:3316          
13  200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:3316          
14  401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  341     text/html; charset=us-ascii nuget:3316          
15  200 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  28,301      application/atom+xml; charset=utf-8 nuget:3316          
16  200 HTTP    tfs-live:5432   /api/packages/Moq/4.2.1510.2205/content 648,088 private application/zip nuget:3316          

I ran the same command again with the package still in the Cache. This is what Fiddler logged. While it was running I watched the Cache folder and confirmed it downloaded a new copy of the package.

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
1   200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:1208          
2   401 HTTP    tfs-live:5432   /api/   1,293       text/html   nuget:1208          
3   200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:1208          
4   200 HTTP    Tunnel to   www.nuget.org:443   0           nuget:1208          
5   401 HTTP    tfs-live:5432   /api/   341     text/html; charset=us-ascii nuget:1208          
6   307 HTTP    tfs-live:5432   /api/   0           nuget:1208          
7   401 HTTP    tfs-live:5432   /api/odata  1,293       text/html   nuget:1208          
8   401 HTTP    tfs-live:5432   /api/odata  341     text/html; charset=us-ascii nuget:1208          
9   200 HTTP    tfs-live:5432   /api/odata  366     application/atomsvc+xml; charset=utf-8  nuget:1208          
10  401 HTTP    tfs-live:5432   /api/odata/$metadata    1,293       text/html   nuget:1208          
11  401 HTTP    tfs-live:5432   /api/odata/$metadata    341     text/html; charset=us-ascii nuget:1208          
12  200 HTTP    tfs-live:5432   /api/odata/$metadata    4,386       application/xml; charset=utf-8  nuget:1208          
13  401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  1,293       text/html   nuget:1208          
14  401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  341     text/html; charset=us-ascii nuget:1208          
15  200 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?id='Moq'  28,301      application/atom+xml; charset=utf-8 nuget:1208          
16  200 HTTP    tfs-live:5432   /api/packages/Moq/4.2.1510.2205/content 648,088 private application/zip nuget:1208  

[rrelyea]

Initially, I want to focus on the DTD side of this. We want to understand any major change in behavior with DTD processing that we may have introduced with 3.4.

Please work with me so that I can repro this same DTD error.

Can you point me to a specific instance of a Klondike server that I can use? Or do I need to setup my own Klondike server?

We suspect the DTD issue is related to the v2 feed that your server is providing?

[rrelyea]

@LtKlaus - we have a private build of nuget.exe that we'd like to share with you. can you please email me at ..., and I'll get the build to you. we'd like to confirm that the DTD issue is fixed for you.

[rrelyea]

@LtKlaus said in email: Running this version of NuGet.exe doesn't produce the DTD error but it also fails to find the package.

This is what I get

C:\ConsoleApplication1>nuget install Moq -o Packages Feeds used: http://tfs-live:5432/api/

GET http://tfs-live:5432/api/FindPackagesById()?Id='Moq' NotFound http://tfs-live:5432/api/FindPackagesById()?Id='Moq' 585ms Unable to find package 'Moq'

When I use v3.3 of NuGet I get this

C:\ConsoleApplication1>nuget install Moq -o Packages Feeds used: http://tfs-live:5432/api/

Attempting to gather dependencies information for package 'Moq.4.2.1510.2205' with respect to project 'Packages', targeting 'Any,Version=v0.0' Attempting to resolve dependencies for package 'Moq.4.2.1510.2205' with DependencyBehavior 'Lowest' Resolving actions to install package 'Moq.4.2.1510.2205' Resolved actions to install package 'Moq.4.2.1510.2205' Adding package 'Moq.4.2.1510.2205' to folder 'C:\ConsoleApplication1\Packages' Added package 'Moq.4.2.1510.2205' to folder 'C:\ConsoleApplication1\Packages' Successfully installed 'Moq 4.2.1510.2205' to Packages

This is what I see in Fiddler when using v3.3

Result Protocol Host URL Body Caching Content-Type Process Comments Custom

1 401 HTTP tfs-live:5432 /api/ 1,293 text/html nuget:4476
2 401 HTTP tfs-live:5432 /api/ 341 text/html; charset=us-ascii nuget:4476
3 307 HTTP tfs-live:5432 /api/ 0 nuget:4476
4 401 HTTP tfs-live:5432 /api/odata 1,293 text/html nuget:4476
5 401 HTTP tfs-live:5432 /api/odata 341 text/html; charset=us-ascii nuget:4476
6 200 HTTP tfs-live:5432 /api/odata 366 application/atomsvc+xml; charset=utf-8 nuget:4476
7 401 HTTP tfs-live:5432 /api/odata/$metadata 1,293 text/html nuget:4476
8 401 HTTP tfs-live:5432 /api/odata/$metadata 341 text/html; charset=us-ascii nuget:4476
9 200 HTTP tfs-live:5432 /api/odata/$metadata 4,386 application/xml; charset=utf-8 nuget:4476
10 401 HTTP tfs-live:5432 /api/odata/FindPackagesById()?id='Moq' 1,293 text/html nuget:4476
11 401 HTTP tfs-live:5432 /api/odata/FindPackagesById()?id='Moq' 341 text/html; charset=us-ascii nuget:4476
12 200 HTTP tfs-live:5432 /api/odata/FindPackagesById()?id='Moq' 28,301 application/atom+xml; charset=utf-8 nuget:4476
13 200 HTTP tfs-live:5432 /api/packages/Moq/4.2.1510.2205/content 648,088 private application/zip nuget:4476

and this is what I see in Fiddler when using your version of NuGet

Result Protocol Host URL Body Caching Content-Type Process Comments Custom

1 401 HTTP tfs-live:5432 /api/FindPackagesById()?Id='Moq' 1,293 text/html nuget:3572
2 401 HTTP tfs-live:5432 /api/FindPackagesById()?Id='Moq' 341 text/html; charset=us-ascii nuget:3572
3 404 HTTP tfs-live:5432 /api/FindPackagesById()?Id='Moq' 1,245 text/html nuget:3572

[rrelyea]

Do you see this behavior for other packages (like "bootstrap" or "mvvmlight").

I've noticed that when I use those 2 packages with a local Klondike server, they are found on the default mirror server (nuget v2 feed), and then cached on my local Klondike server. When I do Moq -- it doesn't get cached on the server for some reason.

[rrelyea]

It appears that the redirection, may have been broken in our v3.4. If you change your package server specified as /api/odata instead of /api/ -- I think it will work for you with the private nuget.exe I sent you. I'm going to open a separate issue on the lack of redirection in v3.4.

Please change to */api/odata temporarily and validate if our cache behavior seems appropriate now (the original issue you complained about).

[LtKlaus]

When trying to install mvvmlight using your test version of NuGet, and my Klondike server as the only source, I get the same message saying it can't find the package. If I use NuGet v3.3 it finds and installs the package without issue.

[LtKlaus]

Using */api/odata in my source does allow me to install the packages using the version of v3.4 that you sent me.

[LtKlaus]

I also tested my initial issue with your version of NuGet to see if it would redownload stuff that's already in the cache. It took a bit of time to realize it's now using C:\Users\%UserName%.NuGet\packages\ for the cache, but it looks like it no longer downloads packages that are already in the cache.

When testing with v3.3 I could see a .tmp file in the cache while it was downloading the package and could also confirm the time stamp on the package changed each time I ran nuget install. The new cache is structured differently but I can confirm it only downloaded the package once and the time stamp hasn't changed.

Also when using your version of NuGet, in Fiddler I see the following if I install Moq and it isn't in the cache

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
1   401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  1,293       text/html   nuget:1788          
2   401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  341     text/html; charset=us-ascii nuget:1788          
3   200 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  28,378      application/atom+xml; charset=utf-8 nuget:1788          
4   200 HTTP    tfs-live:5432   /api/packages/Moq/4.2.1510.2205/content 648,088 private application/zip nuget:1788          

and this is what I see if Moq is already in the cache

#   Result  Protocol    Host    URL Body    Caching Content-Type    Process Comments    Custom  
1   401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  1,293       text/html   nuget:300           
2   401 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  341     text/html; charset=us-ascii nuget:300           
3   200 HTTP    tfs-live:5432   /api/odata/FindPackagesById()?Id='Moq'  28,378      application/atom+xml; charset=utf-8 nuget:300           

This shows that it isn't downloading the package when it's already in the cache. In my earlier comment that shows the logs from Fiddler when using v3.3, the last line of the log shows it's always downloading the package.

[rrelyea]

Ok. Closing this issue as fixed in 3.4 then. Thanks for the report. Spun off the discovery issue here: https://github.com/NuGet/Home/issues/2319