search

macaroni-os / mark-issues

Macaroni Automated Repositories Kit Issues
4 stars 1 forks source link

[metatools] RFC - The regeneration of all kits from zero doesn't generate all blos #50

Open geaaru opened 2 months ago

geaaru commented 2 months ago

Preparing a new infrastructure as a clone of MARK's infra and/or a mirror is not possible because the static ebuilds (imported from external repositories, for example gentoo-staging) aren't correctly managed and the blos tree of these packages is not created.

The autogen packages don't suffer from this issue.

A temporary workaround for this problem is to prepare a secondary mirror node with a flat layout that contains the tarballs of all packages at the moment not autogenned. Using flat layout will simplify the developer's life in pushing/removing packages that are missing or removed from the tree.

geaaru commented 2 months ago

While we think about how we could replace metatools in the future it's good to reduce the number of imported and/or static ebuilds to clean our kit-fixups tree to be only autogen.

geaaru commented 2 months ago

Using this GENTOO_MIRRORS env is a workaround to fetch static ebuild tarballs:

GENTOO_MIRRORS="https://distfiles.macaronios.org https://dl.macaronios.org/repos/distfiles"

this could be added on /etc/portage/make.conf. We will try to fix metatools to have both automatically.

geaaru commented 1 week ago

About the static ebuild i saw that metatools supplies a distfile-kit-fetch tool from version 1.3.2:

From ChangeLog:

Add distfile-kit-fetch tool which you would run on the system you ran merge-kits on. It will try to grab all the non- autogenned distfiles and download all it can, ultra-fast-spider style, and store them locally in the BLOS. It is used like this: distfile-kit-fetch It will use the kit-cache data from a previous merge-kits run. This kit-cache data is stored in ~/repo_tmp/tmp/kit_cache. This tool also will make sure it has a locally-checked out kit-fixups repo in ~/repo_tmp/source-trees/kit-fixups and will utilize the thirdpartymirrors file located at core-kit/curated/profiles/ thirdpartymirrors to expand any mirror:// prefixes in ebuild SRC_URI strings. Additional work has been done on this tool to make it production-quality. For example, it won't stop running when it encounters a file download that errors out --> instead it will be greedy and try to keep downloading as many distfiles as it can.