Open spongessuck opened 5 years ago
Thanks for raising the issue. Is there a way to reliably repro this? Are you able to post the error output?
I don't know whether this provides any assistance, but FWIW, I'm finding that the libman restore command works reliably with cdnjs, but referencing the provider unpkg (which works within the IDE), consistently fails via the restore command, the reported errors are: [LIB002]: The "bootstrap-wysiwyg@2.0.1" library could not be resolved by the "unpkg" provider [LIB002]: The "material-design-icons-font@2.2.3" library could not be resolved by the "unpkg" provider [LIB002]: The "roboto-fontface@0.4.5" library could not be resolved by the "unpkg" provider
I've attached my libman.json file: libman.json.txt
I've ran into similar feedback messages from Libman a while ago when I tried to deploy an app that used it. I haven't looked into it at all, but at the time, if I rebuilt the solution first, then tried the deploy again it seemed to work (implying maybe one has caching issues)? I can't provide much more detail than that in my instance.
I have a similar behavior. You add your package in via cdnjs (because unpkg almost never works, in fact I've never personally got it it to work). Some packages you select will come down, but others fail consistently. For instance, try to add the monaco-editor and it will consistently returns 404 for half of the files. I'm starting to think it's a CDN problem all together. Example:
Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/monaco-editor/0.15.6/min-maps/vs/loader.js.map"
However, if I navigate to that URL manually and it worked. But then I did it again and it 404'd. I reloaded that exact 404 and it worked. I opened up the Chrome utilities and reloaded every few seconds and about 20% of the requests returned a 200 with the file, the rest 404'd with the same URL. This is the behavior seen when trying to restore packages with a fair amount of files. -Something- fails and then the whole thing fails. The more files there are, the more chance something 404's and it can be different every time which is making it impossible to restore some packages via this mechanism.
I haven't used this a ton but I've observed it over a serious of a few weeks.
As of recent this problem became worse Error LIB010 Failed to download resource from "https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js" The 404 response from cdnjs is intermittent but it fails the build pipeline and costs man-hours in reaction to failed CI/CD build notifications and followup email notification about the build notifications { "version": "1.0", "defaultProvider": "cdnjs", "defaultDestination": "wwwroot/lib", "libraries": [ { "library": "twitter-bootstrap@3.4.0", "destination": "wwwroot/lib/bootstrap/" }]}
Are there any plans to implement a fallback mechanism?
Thank you.
I've just been downloading libraries with npm and adding filesystem references to files in node_modules.
On Mon, Dec 24, 2018, 11:04 freemstr <notifications@github.com wrote:
As of recent this problem became worse Error LIB010 Failed to download resource from " https://cdnjs.cloudflare.com/ajax/libs/twitter-bootstrap/3.4.0/js/npm.js" The 404 response from cdnjs is intermittent but it fails the build and costs man-hours in reaction to failed build notifications and followup email notification about the build notifications
Are there any plans to implement a fallback mechanism?
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aspnet/LibraryManager/issues/370#issuecomment-449750058, or mute the thread https://github.com/notifications/unsubscribe-auth/AGE0TkhVrJkc3kDmJydRdOfn6vuA-QS-ks5u8PsQgaJpZM4WbUgN .
@justcla At 23:00 my Azure Pipeline build failed. It succeeded several hours later with the same source project Build consist of downloading clean source and building the solution Error LIB002: The "@types/jquery@3.3.22" library could not be resolved by the "unpkg" provider libman.json { "version": "1.0", "defaultProvider": "cdnjs", "defaultDestination": "JS/lib", "libraries": [ { "library": "@types/jquery@3.3.22", "destination": "TS/lib/jquery/", "provider": "unpkg" } ]}
Could this be related? https://github.com/aspnet/LibraryManager/issues/241
Are there any plans to implement a fallback mechanism?
I think this would be useful. Builds failing because they cant get to the cdn could be a big problem. What if someone was try to release a patch for a client who's having an issue... it needs to be sorted.
Here the same problems. But also when it is in production that unpkg.com is sometimes very slow and unreachable. For builds this is a lesser issue locally, because we can retry this, but when having CI it will fail the builds. In production it could take 30 seconds to load the page.. Something is wrong with unpkg, you should expect CloudFlare to keep it stable.
Maybe the same bug here. Sometimes the packages can be restored, sometimes they cannot be resolved by "unpkg". I've read that unpkg is way better than cdnjs because of its architecture. While unpkg makes npmjs packages accessible, cdnjs is barely manageable anymore these days. If unpkg is just as bad, there are no more options and I'm keeping a copy of the JavaScript files in my Git repository again, just like in the past. It's not that they are too big for that. At least they're there where my code is, and not gone half of the time.
Also, why does LibMan have to restore these files from online at every build? The files are there, there's no reason to fail the build because they couldn't be downloaded. That's just stupid.
And strangely, ASPNETCDN is not one of the listed package providers. You'd think Microsoft have an option to use its own CDN.
@justcla I can frequently repro this when trying to restore material-design-icons
, which contains a large number of files. It fails on a different file every time. It seems that restore does not retry individual files, and the entire operation fails if any file fails.
The LIB002 is most likely due to GET requests going via HTTP protocol - in such cases unpkg returns status code 301. That's the behaviour we're seeing as per #491
LIB018 seems to be present in larger packages - the meta related to it becomes truncated as confirmed with Fiddler.
Are there any timeouts in play within Libman to prevent hangs during the restore process. It could also be race conditions. If unpkg takes more than 30s to return the full package payload does Libman simply kill the request and throw?
Oh, maybe it has to fetch on every build because of #490. It doesn't know what it's doing or what it has done. So it has to check every time. And it even fails either way. Can't detect new files and can't download anything, then blows up either the build or the application at runtime. It's on my list of things to throw out the window now.
We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.
I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.
We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.
I don't understand why the team opted to move to a library manager that supports fewer packages and is less reliable. Very frustrating.
Also running into the same error (also unpkg) and it's preventing production builds intermittently.
We routinely see 'The ... library could not be resolved by the "unpkg" provider' errors. It causes continuous builds to fail. It causes local builds to fail. It's pretty much a fail.
I am having the same issue as hempels (Visual studio Community 2019 16.3.6). It would be great if this was solved as for my project Libman seems to be a generally good option.
I got fed up with it. All of the packages I used unpkg for originally are now obtainable from jsDelivr, I believe it was added after unpkg. So now I'm just using jsDelivr instead. Still using libman mind you, I just no longer using unpkg as a package source.
@nickalbrecht jsDelivr is not available in the selection. How did you do that?
@ygoe Just update Library Manager to the most recent version, it was added at a later date so earlier version won't have it.
@ygoe which version of VS are you using? The JSDelivr provider wasn't included in VS until 16.0, but you can build a private version of the VSIX. See more here.
Tried with VS 2017. Not switched to VS 2019 yet. Waiting for a usable version (without several blocking bugs) with .NET Core 3 support (which should be 16.4).
You should also be able to use the CLI or the Build package if you don't have VS2019. I use Microsoft.Web.LibraryManager.Build so that my libraries restore as part of the build outside of VS. But we're kind of getting off topic :-P
All my install library are failing to restore when i run restore client side library for my libma.js file. I am recieving this error " library could not be resolved by the "cdnjs" provider" example: { "library": "twitter-bootstrap@3.3.6", "destination": "wwwroot/lib/twitter-bootstrap/" }, { "library": "font-awesome@4.1.0", "destination": "wwwroot/lib/font-awesome/" }, { "library": "toastr.js@2.1.4", "destination": "wwwroot/lib/toastr.js/" }, jquery works. others coud not be able to restore. I am using visual studio 2019.
Correct, around 8 hours ago this started happening. It is because of problems at cdnjs / Cloudflare. Looks like it should is resolved almost an hour ago: https://status.cdnjs.com/incidents/0gdmkm4tl98d
Issues solved. The package restore is working fine again.
I had a similar issue. The problem was with package name. Instead of:
{
"library": "twitter-bootstrap@4.4.1",
"destination": "wwwroot/lib/bootstrap/"
},
I was using
{
"library": "bootstrap@4.4.1",
"destination": "wwwroot/lib/bootstrap/"
},
The message was confusing. And it wasn't installing any of my packages.
The message should be something like The package "bootstrap@4.4.1" was not found in "cdnjs" repository
rather than The "bootstrap@4.4.1" library could not be resolved by the "cdnjs" provider
.
@denis-pujdak-adm-it If I were to rephrase your point, we should be emphasizing that the library name couldn't be found separately from the name@version? E.g.
The package "bootstrap@4.4.1" was not found in "cdnjs" repository.
vs.
The "bootstrap" library could not be resolved by the "cdnjs" provider
(removing the version here to emphasize the name)
@jimmylewis Hm. Anyway. That was just my thought based on my experience with the instalation.
I am also getting some intermittent errors when running dotnet build or dotnet watch run. most of the time it's fine but then every now and again I get this:
libman.json : error LIB002: The "bootstrap@4.4.1" library could not be resolved by the "unpkg" provider [/Users/solrevdev/code/projectnamehere/src/web/web.csproj]
The build failed. Fix the build errors and run again.
libman.json
{
"version": "1.0",
"defaultProvider": "cdnjs",
"libraries": [
{
"provider": "unpkg",
"library": "bootstrap@4.4.1",
"files": [
"dist/css/bootstrap.css",
"dist/css/bootstrap.css.map",
"dist/css/bootstrap.min.css",
"dist/css/bootstrap.min.css.map",
"dist/js/bootstrap.js",
"dist/js/bootstrap.js.map",
"dist/js/bootstrap.min.js",
"dist/js/bootstrap.min.js.map"
],
"destination": "wwwroot/lib/bootstrap/"
},
{
"library": "bootstrap-colorpicker@3.2.0",
"destination": "wwwroot/lib/bootstrap-colorpicker/dist"
},
{
"library": "jquery@3.4.1",
"files": ["jquery.min.js", "jquery.js", "jquery.min.map"],
"destination": "wwwroot/lib/jquery/dist/"
},
{
"library": "jquery-validate@1.19.1",
"files": ["jquery.validate.js", "jquery.validate.min.js"],
"destination": "wwwroot/lib/jquery-validation/dist"
},
{
"library": "jquery-validation-unobtrusive@3.2.11",
"files": [
"jquery.validate.unobtrusive.js",
"jquery.validate.unobtrusive.min.js"
],
"destination": "wwwroot/lib/jquery-validation-unobtrusive/"
},
{
"library": "lodash.js@4.17.15",
"files": ["lodash.js", "lodash.min.js"],
"destination": "wwwroot/lib/lodash/"
},
{
"library": "vue@2.6.11",
"files": ["vue.js", "vue.min.js"],
"destination": "wwwroot/lib/vue/"
}
]
}
OS:
▲ (macmini.local) ~ screenfetch -n
solrevdev@macmini
OS: 64bit Mac OS X 10.14.6 18G95
Kernel: x86_64 Darwin 18.7.0
Uptime: 5d 19h 7m
Packages: 439
Shell: zsh 5.8
Resolution: 2560x1440
DE: Aqua
WM: Quartz Compositor
WM Theme: Blue (Dark)
Disk: 293G / 525G (57%)
CPU: Intel Core i5-4278U @ 2.60GHz
GPU: Intel Iris
RAM: 8668MiB / 16384MiB
dotnet
▲ (macmini.local) ~ dotnet --info
.NET Core SDK (reflecting any global.json):
Version: 3.1.102
Commit: 573d158fea
Runtime Environment:
OS Name: Mac OS X
OS Version: 10.14
OS Platform: Darwin
RID: osx.10.14-x64
Base Path: /usr/local/share/dotnet/sdk/3.1.102/
Host (useful for support):
Version: 3.1.2
Commit: 916b5cba26
.NET Core SDKs installed:
2.1.804 [/usr/local/share/dotnet/sdk]
3.1.102 [/usr/local/share/dotnet/sdk]
.NET Core runtimes installed:
Microsoft.AspNetCore.All 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.All]
Microsoft.AspNetCore.App 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.AspNetCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.AspNetCore.App]
Microsoft.NETCore.App 2.1.16 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
Microsoft.NETCore.App 3.1.2 [/usr/local/share/dotnet/shared/Microsoft.NETCore.App]
To install additional .NET Core runtimes or SDKs:
https://aka.ms/dotnet-download
Hello, I am also getting the same intermittent errors. Is there any way to ask libman to retry the download? If not, could we think about a retry option, allowing a number of retrying? Regards,
Is there any way to ask libman to retry the download?
open libman.json, press space, press delete. Save and it'll retry :)
or you can right-click libman.json and choose "restore client-side libraries".
I am using the restore on build option. So I was more asking for an automatic way of doing a retry. It will be especially useful for our build pipeline as the build is regularly on error because of random fails from unpkg.
Which version of the NuGet package are you using? 2.0.48 and above should have a 5-attempt loop when downloading files.
I am using the restore on build option. So I was more asking for an automatic way of doing a retry. It will be especially useful for our build pipeline as the build is regularly on error because of random fails from unpkg.
What you really want is to ensure unpkg works, and a retry doesn't always make that happen. So I would recommend getting the updates locally, and committing them along with your source. That shifts the problem away from the build server's dependency on an unreliable 3rd party and onto the developer. Caching the files locally in this way might be a lot more practical if you're rebuilding often so you're not continually re-fetching the same files from the internet.
Anecdotally, JSDelivr also seems more reliable than unpkg if you don't want to check in files. YMMV.
It's been two years and nothing has been done about this?
It just keeps randomly failing during a build:
Is there a fallback mechanism? Is there a reliable alternative to unpkg source?
Is there a reliable alternative to unpkg source?
As @jimmylewis said in the previous comment, there's also jsDelivr provider which should be more reliable. Migration should be as easy as changing the provider type. If you get issues like this with jsDelivr as well, feel free to ping me - I'm on the jsDelivr team and we aim for 100% reliability.
Again, I've gotten around this issue by using npm to install dependencies,
then have libman configured to use the filesystem provider to copy the
files I need into a lib
directory that my project references. It's been
extremely reliable for local and CI builds.
On Thu, Apr 9, 2020, 12:44 Martin Kolárik notifications@github.com wrote:
Is there a reliable alternative to unpkg source?
As @jimmylewis https://github.com/jimmylewis said in the previous comment, there's also jsDelivr provider which should be more reliable. Migration should be as easy as changing the provider type. If you get issues like this with jsDelivr as well, feel free to ping me - I'm on the jsDelivr team and we aim for 100% reliability.
— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://github.com/aspnet/LibraryManager/issues/370#issuecomment-611630208, or unsubscribe https://github.com/notifications/unsubscribe-auth/ABQTITUT6D3QX6QW3I7WGD3RLX3QXANCNFSM4FTNJAGQ .
We have the libraries downloaded to a network fileshare and just configure libman to point to that.
So no having to check the libraries into source code, and no issues with the build.
It doesn't really make sense to re-download the same libraries every time we need to deploy an updated build and the build is no longer subject to breaking due to intermittent issues with libman package retrieval
Thanks to your comments, I am now using jsdelivr instead of unpkg and so far, it's working better.
@matthieume If you could get a network trace (e.g. with Fiddler) of when it's not working right, I'd be interested in taking a look. I'm curious to confirm that you're actually getting failures 5x on the same request. (Totally optional though, no pressure if you've moved on 😉)
@jimmylewis I am not sure why they don't appears on my Fiddler... But here is my libman.json, try to restore the package multiple time, you should have the error
{
"version": "1.0",
"defaultProvider": "jsdelivr",
"libraries": [
{
"provider": "cdnjs",
"library": "mdbootstrap@4.14.0",
"destination": "wwwroot/lib/mdbootstrap/"
},
{
"provider": "unpkg",
"library": "filepond-plugin-image-preview@4.6.1",
"destination": "wwwroot/lib/filepond-plugin-image-preview/"
},
{
"provider": "unpkg",
"library": "filepond-plugin-image-edit@1.6.0",
"destination": "wwwroot/lib/filepond-plugin-image-edit/"
},
{
"provider": "unpkg",
"library": "filepond@4.12.0",
"destination": "wwwroot/lib/filepond/"
},
{
"library": "@simonwep/pickr@1.5.1",
"destination": "wwwroot/lib/@simonwep/pickr/",
"files": [
"dist/pickr.min.js",
"dist/themes/nano.min.css"
]
},
{
"provider": "cdnjs",
"library": "toastr.js@2.1.4",
"destination": "wwwroot/lib/toastr.js/"
},
{
"library": "bootstrap-select@1.13.12",
"destination": "wwwroot/lib/bootstrap-select/"
},
{
"library": "daterangepicker@3.0.5",
"destination": "wwwroot/lib/daterangepicker/"
},
{
"library": "@fortawesome/fontawesome-free@5.12.1",
"destination": "wwwroot/lib/@fortawesome/fontawesome-free/"
},
{
"library": "bootstrap@4.4.1",
"destination": "wwwroot/lib/bootstrap/"
},
{
"provider": "cdnjs",
"library": "jquery@3.4.1",
"destination": "wwwroot/lib/jquery/"
},
{
"library": "davidshimjs-qrcodejs@0.0.2",
"destination": "wwwroot/lib/davidshimjs-qrcodejs/"
},
{
"provider": "cdnjs",
"library": "jqueryui@1.12.1",
"destination": "wwwroot/lib/jqueryui/"
},
{
"provider": "cdnjs",
"library": "moment.js@2.24.0",
"destination": "wwwroot/lib/moment.js/"
},
{
"provider": "cdnjs",
"library": "Chart.js@2.9.3",
"destination": "wwwroot/lib/Chart.js/"
},
{
"provider": "cdnjs",
"library": "popper.js@1.16.1",
"destination": "wwwroot/lib/popper.js/"
},
{
"provider": "unpkg",
"library": "filepond-plugin-file-validate-type@1.2.4",
"destination": "wwwroot/lib/filepond-plugin-file-validate-type/"
},
{
"provider": "unpkg",
"library": "filepond-plugin-image-crop@2.0.3",
"destination": "wwwroot/lib/filepond-plugin-image-crop/"
},
{
"provider": "unpkg",
"library": "filepond-plugin-image-transform@3.7.2",
"destination": "wwwroot/lib/filepond-plugin-image-transform/"
},
{
"library": "wowjs@1.1.3",
"destination": "wwwroot/lib/wowjs/"
},
{
"provider": "cdnjs",
"library": "timeago.js@4.0.2",
"destination": "wwwroot/lib/timeago.js/"
},
{
"provider": "cdnjs",
"library": "bodymovin@5.6.6",
"destination": "wwwroot/lib/bodymovin/",
"files": [
"lottie.min.js"
]
},
{
"provider": "unpkg",
"library": "datatables.net@1.10.20",
"destination": "wwwroot/lib/datatables/"
},
{
"provider": "unpkg",
"library": "datatables.net-bs4@1.10.20",
"destination": "wwwroot/lib/datatables-bs4/"
},
{
"provider": "unpkg",
"library": "datatables.net-buttons@1.6.1",
"destination": "wwwroot/lib/datatables-buttons/"
},
{
"provider": "unpkg",
"library": "datatables.net-buttons-bs4@1.6.1",
"destination": "wwwroot/lib/datatables-buttons-bs4/"
},
{
"provider": "unpkg",
"library": "datatables.net-colreorder@1.5.2",
"destination": "wwwroot/lib/datatables-colreorder/"
},
{
"provider": "unpkg",
"library": "datatables.net-colreorder-bs4@1.5.2",
"destination": "wwwroot/lib/datatables-colreorder-bs4/"
},
{
"provider": "unpkg",
"library": "datatables.net-fixedcolumns@3.3.0",
"destination": "wwwroot/lib/datatables-fixedcolumns/"
},
{
"provider": "unpkg",
"library": "datatables.net-fixedcolumns-bs4@3.3.0",
"destination": "wwwroot/lib/datatables-fixedcolumns-bs4/"
}
]
}
I'm getting this right now trying to restore angular.js
{
"version": "1.0",
"defaultProvider": "cdnjs",
"defaultDestination": "wwwroot/lib/",
"libraries": [
{
"library": "angular.js@1.7.8",
"files": [
"angular.js",
"angular-animate.js",
"angular-cookies.js",
"angular-route.js",
"angular-aria.js",
"angular-messages.js"
]
}
]
}
Any library that I have the specific files specified is failing.
Restore operation started... Restoring libraries for project XXXXXXXX "angular.js@1.7.8" does not contain the following: angular.js, angular-animate.js, angular-cookies.js, angular-route.js, angular-aria.js, angular-messages.js Valid files are Restore operation completed with errors
If you open the libman.json and attempt to edit the file list it has NO files retrieved from the server.
This has been happening to all of our builds today, both on local workstations as well as in Azure DevOps in the cloud.
@ExcaliburVT that's odd, your file works for me. Which version of LibMan are you using? Can you find the cache\cdnjs\angular.js\metadata.json
file in your LibMan cache and see if it has the expected files listed for the 1.7.8 version?
Libman has been unreliable for our company in last 2-3 weeks. Sporadic 'could not resolve package' messages. We sometimes have to rebuild like 10-20 times until it's working again.
On my computer, i'm getting random 'does not contain the following' messages, same as @ExcaliburVT . However, a colleague of mine does not have the same problem at the moment. Very strange behaviour.
@jimmylewis
It's because Libman has included shitty sources by default such as unpkg. There should only be sources with 99 percent uptime.
Using Microsoft.Web.LibraryManager.Build v 2.0.96, the latest version at the time was 2.1.50 but had pipeline issues. 2.1.76 is the latest stable.
My team is also running into this issue where LibMan will be unable to find files using the default provider (cdnjs) for some users, but not all. Is also affecting our build pipelines.
Here is an errors message example, these errors are LIB018s:
"jquery-validate@1.19.1" does not contain the following: additional-methods.js, additional-methods.min.js, jquery.validate.js, jquery.validate.min.js Valid files are Restore operation completed with errors
Our libman.json is pretty simple:
{
"version": "1.0",
"defaultProvider": "cdnjs",
"libraries": [
{
"library": "jquery@3.5.0",
"destination": "wwwroot/lib/jquery/"
},
{
"library": "twitter-bootstrap@4.4.1",
"destination": "wwwroot/lib/bootstrap/"
},
{
"library": "jquery-validate@1.19.1",
"files": [
"additional-methods.js",
"additional-methods.min.js",
"jquery.validate.js",
"jquery.validate.min.js"
],
"destination": "wwwroot/lib/jquery-validate/"
},
{
"library": "jquery-validation-unobtrusive@3.2.11",
"destination": "wwwroot/lib/jquery-validation-unobtrusive/"
},
{
"library": "jSignature@2.1.3",
"destination": "wwwroot/lib/jSignature/",
"files": [
"jSignature.min.js"
]
},
{
"library": "moment.js@2.24.0",
"destination": "wwwroot/lib/moment.js/",
"files": [
"moment.min.js"
]
}
]
}
I'm actually relieved that other people had the same problem we did, I was starting to think I was crazy. The issue seems to be resolved today as all our automated builds are succeeding but this is the 2nd time in a month that we basically had ALL builds (approx 20) fail during the entire day for basically the same reason.
We are using Visual Studio 2019 version 16.5.4 and do not have the libman command line installed.
@jimmylewis where would the Visual Studio libman cached be located? I looked through the documentation and installed the command line tool to use the cache list command but it does not indicate the file path.
Functional impact
Builds fail because required lib files don't exist in project.
Minimal repro steps
Expected result
Files that restore successfully once should always restore successfully
Actual result
LibMan intermittently fails with error LIB002 or LIB018, or doesn't load the correct contents of the package (i.e., index.html instead of dist/**).