search

apache / buildstream

BuildStream, the software integration tool
https://buildstream.build/
Apache License 2.0
83 stars 28 forks source link

[bst-1] unaliased-url annoying w/ source plugins like cpan, pypi, cargo, etc #1631

Open AdrianVovk opened 2 years ago

AdrianVovk commented 2 years ago

These source plugins have a remote package index (crates.io, etc) that they scan to find new versions of dependencies. Then from these package indexes they construct full download URLs, and then download sources from these URLs. From the plugin user's perspective, there is nothing to alias because there are no URL's involved. The user just says "get me package foo from pypi" and the plugin does that.

Buildstream complains that these sources aren't using aliases. Since these sources sometimes download hundreds of libraries for a build, this can end up spamming the build log with hundreds of lines of warnings.

Potential solutions:

This probably applies to bst-2 also, though I haven't tested it at all

gtristan commented 2 years ago

Indeed I can see how this would be annoying.

Potential solutions:

  • make every plugin provide a way to override the path to the index, and make every project that uses these plugins override these variables with an alias (This is clunky and a bit ugly, IMO. even then this can't work 100% of the time. What about package indexes that return data to map versions -> download urls, or something similar?)

This sounds ideal, these plugins should have some option for the user to provide an alias for the plugins to use in place of full URIs.

  • make a way for projects to opt out of the warning in general. Quick and dirty
  • make a way for plugins to opt out of the warning (i.e. "I know this URL I'm giving you won't have an alias; don't raise an error"). Probably the best middle-of-the-road solution IMO

We have the option to turn the warning into an error but I'm very hesitant to allow silencing these warnings completely.

At best, I would think it would be ideal to turn the individual warnings into a single warning, summarizing all of the instances where aliases are missing at startup time.

The main purpose of this warning is to ensure that the project has the capability to mirror the entirety of the build input, and keep a local copy so as to ensure the project is never dependent on the internet in order to repeat a build, which is a pretty important capability (which means it is doubly important for these sources which discover implied external sources to allow setting an alias for them).

nanonyme commented 2 years ago

FWIW pypi plugin does have a mechanism for mirroring. I'm also hesitant having the error disabled globally. On per element level, yes, for some cases mirroring is non-sense. Like, it might be your own project repo to begin with.

nanonyme commented 7 months ago

I would also note we ended up just making this warning fatal in freedesktop-sdk because not aliased source is clearly a bug when you want to mirror everything.