add new `upload-sequence` command

shubhbapna commented 2 days ago

Context

Fromager provides an end to end solution to build wheels from scratch as part of a trusted supply chain. However, one crucial part of it is using sources from a trusted artifact store. The downstream pattern of using fromager is to first bootstrap and get the build order file which is then used to upload the sdist to our private store after which we run build-sequence to actually build and consume the wheels. In the build-sequence step we need to fetch the sdists from our private trusted store instead of the ones in publicly hosted stores. So we have to run download-sequence first to ensure that build-sequence uses these sdists. This is because the sdist url in the build order file points to the public ones and we cannot have build-sequence download the sdists from there.

This has caused us to rely on separate service (which depends on fromager) to upload all these artifacts as well as ensure that download-sequence places the downloaded sdists in the right locations such that when build-sequence runs it thinks that it has already downloaded the sdists and doesn't attempt to download them from the public servers.

Feature Request

Add a upload-sequence command that takes a build order file, uploads the sdists from public servers to private servers and updates all the urls in the build order to point to the private servers.

This will ensure that the uploading service is a first class citizen of fromager and any changes in fromager that impact it are immediately caught within fromager itself
We don't have to rely on download-sequence placing the sdists correctly and build-sequence can use the sdists url directly from the build order file

dhellmann commented 2 days ago

Another aspect of this to consider for updating the URLs in the build-order file: We would have to assume that the tool that uploads the file knows (is told or can compute) the URL where it can be fetched. Is that always going to be true?

shubhbapna commented 2 days ago

Maybe we can make it a requirement?

dhellmann commented 1 day ago

We could probably make it a requirement. I expect the way that will work for us downstream after our upcoming migration is to upload the artifact using the OCI APIs, then resolve the package using the PyPI API and use that to get the URL.

shubhbapna commented 1 day ago

Haven't worked with OCI APIs before, maybe it returns the download url in its response?

python-wheel-build / fromager

add new `upload-sequence` command #509