Open michael-o opened 3 years ago
Why are you making 2 separate repositories? Poudriere won't support the pkg
idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.
Why are you making 2 separate repositories? Poudriere won't support the
pkg
idea of multiple remote repositories. Any packages needed for runtime dependencies need to be present and built in the same repository.
I will try to expand. While I could add this package into by base repository this would only solve the symptom not the problem. My idea is to have a base repo for all jails. Then, repos per jail (use case) type on top of base which may depend on packages in base or any other packages from the official ports tree. So to speak a layered design. I want to avoid recompiling ports which aren't necessary. My idea is very similar to one of @rootwyrm in #534.
Theoretically one could do the following:
122-release-amd64-default-ldadw-base
122-release-amd64-default-ldadw-custom
and poudriere will try to use these packages first and then build the rest from ports:
poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -M `122-release-amd64-default-ldadw-base` -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports
where -M
would mount previously prepared local repos in /usr/local/etc/pkg/repos
in the jail. Alternatively, there would be a directory for this run which could contain repos to include.
With that one can avoid duplication and introduce a layered design. WDYT?
It seems like that https://github.com/freebsd/poudriere/pull/797 achieves a, simpler, but same goal though limited for some reason to official FreeBSD repo and the arbitrary one.
I was able to very uglily trick poudriere with a modified version of #797:
common.sh
for:
cat >> "${MASTERMNT}/etc/pkg/poudriere.conf" <<-EOF
FreeBSD: {
url: ${packagesite},
mirror_type: "NONE",
pubkey: "NONE",
signature_type: "NONE"
}
EOF
poudriere bulk -j 122-release-amd64 -O ldadw-custom -z ldadw-custom -f /usr/local/etc/poudriere.d/122-release-amd64-default-ldadw-custom-ports -b https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base -c
but it still wanted to build too many packages, then I did the following:
root@deblndw011x:/usr/local/etc/poudriere.d
# cp -p 122-release-amd64-default-ldadw-base-make.conf 122-release-amd64-default-ldadw-custom-make.conf
# cp -rp 122-release-amd64-default-ldadw-base-options 122-release-amd64-default-ldadw-custom-options
It is not fully clear why I need to fake this overlay and why pkg simply doesn't install both RUN_DEPENDS
from the repo, but insists on the default options config. I guess it does not take it uses its own tree.
Is this a starting point we can discuss for layering? WDYT?
In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.
Thanks, I'll look into it after my vacation.
In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.
I was caught up again by this. Read your post. I understand your approach, but a see a few issues:
pkg autoremove
. This I don't want.For one usecase I have now tried -b URL
:
[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base
Bootstrapping pkg from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base, please wait...
This works, but since the repo config done in common.sh
I need to apply: https://github.com/freebsd/poudriere/issues/883#issuecomment-866791715 since URL does not gurantee that it is a mirror of FreeBSD
, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?
The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.
The build has now been completed. Well, packages have been downloaded, but many which are available at the supplied repo are still being rebuild. This I don't understand.
Found it. I still need to merge the port options from the repo to be used and the port options for this bulk. I will create an issue.
In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.
I was caught up again by this. Read your post. I understand your approach, but a see a few issues:
- Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by
pkg autoremove
. This I don't want.
To clarify something the dependencies are automatic but do not show in pkg autoremove
because the metaports are non-automatic and still installed. So pkg autoremove
shows nothing typically. pkg noauto
is quite clean and only shows pkg and the metaports.
The leaf packages only show up in pkg autoremove
if I trim them from the metaport and upgrade it.
Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.
samba# pkg noauto
pkg-1.18.4
local-fileserver-meta-20210811_12
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
samba# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.
roundcube# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
local-webmail-meta-20220802_3
roundcube# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.
prometheus# pkg noauto
pkg-1.18.4
prometheus-2.36.0_5
local-jail-meta-20220925_7
local-postfix-relay-meta-20210527_18,2
prometheus# pkg autoremove -n
Checking integrity... done (0 conflicting)
Nothing to do.
matrix# pkg noauto
pkg-1.18.4
local-jail-meta-20220925_7
local-matrix-meta-20221030_4
local-postfix-relay-meta-20210527_18,2
matrix# pkg autoremove -n
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 3 packages:
Installed packages to be REMOVED:
matrix-conduit: 0.4.0_2
py39-pip: 22.2.2
py39-virtualenv: 16.7.5
Number of packages to be removed: 3
The operation will free 69 MiB.
For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or PORTREVISION
change happens (and sane things like OPTIONS
). It won't force rebuild everything so much anymore.
For one usecase I have now tried
-b URL
:
Yeah that makes sense as a workaround for what you want.
[00:00:07] Package fetch: Looking for missing packages to fetch from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/${ABI}/ldadw-base Bootstrapping pkg from https://deblndw011x.ad001.siemens.net/FreeBSD/packages/FreeBSD:12:amd64/ldadw-base, please wait...
This works, but since the repo config done in
common.sh
I need to apply: #883 (comment) since URL does not gurantee that it is a mirror ofFreeBSD
, but could also be a custom one. Maybe I should create a followup ticket where a URL should not expect mirror, signature, etc. like the canonical repo?
Definitely please open a separate issue so it can be fixed and tracked.
@bdrewery Thanks, let me crunch on this and get back to you tomorrow. I think I have found yet another bug in Poudriere when building java/openjdk8
:
====================================================
A new configuration has been successfully created in
/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/build/bsd-x86_64-normal-server-release
using configure arguments '--with-boot-jdk=/usr/local/bootstrap-openjdk8 --with-build-number=b07 --with-update-version=342 --with-vendor-name='OpenJDK BSD Porting Team' --with-vendor-url=https://github.com/battleblow/jdk8u/ --with-vendor-bug-url=mumu --with-vendor-vm-bug-url=mumu --disable-freetype-bundling --disable-zip-debug-info --with-cacerts-file=/usr/ports/java/openjdk8/files/cacerts --with-cups=/usr/local --with-debug-level=release --with-freetype=/usr/local --with-freetype-include=/usr/local/include/freetype2 --with-jobs=20 --with-jvm-variants=server --with-milestone=fcs --with-package-path=/usr/local --with-zlib=system --enable-unlimited-crypto --disable-debug-symbols --with-giflib=system --x-includes=/usr/local/include --x-libraries=/usr/local/lib --x-libraries=/usr/local/lib --x-includes=/usr/local/include --prefix=/usr/local --localstatedir=/var --mandir=/usr/local/man --infodir=/usr/local/share/info/ --build=amd64-portbld-freebsd12.3'.
Configuration summary:
* Debug level: release
* JDK variant: normal
* JVM variants: server
* OpenJDK target: OS: bsd, CPU architecture: x86, address length: 64
Tools summary:
* Boot JDK: openjdk version "1.8.0_144" OpenJDK Runtime Environment (build 1.8.0_144-b01) OpenJDK 64-Bit Server VM (build 25.144-b01, mixed mode) (at /usr/local/bootstrap-openjdk8)
* Toolchain: clang (clang/LLVM)
* C Compiler: Version 10.0.1 (at /usr/bin/cc)
* C++ Compiler: Version 10.0.1 (at /usr/bin/c++)
Build performance summary:
* Cores to use: 20
* Memory limit: 32601 MB
===========================================================================
=======================<phase: build >============================
===== env: NO_DEPENDS=yes USER=nobody UID=65534 GID=65534
===> Building for openjdk8-8.342.07.1_1
gmake[1]: Entering directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1'
(cd /wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make && /usr/local/bin/gmake 3 VERBOSE="3" LOG_LEVEL="warn" -R -I /wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make/common "SPEC=/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/build/bsd-x86_64-normal-server-release/spec.gmk" -j20 -f SourceRevision.gmk create-source-revision-tracker)
gmake[2]: Entering directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make'
gmake[2]: *** No rule to make target '3'. Stop.
gmake[2]: Leaving directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1/make'
gmake[1]: *** [/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1//make/Main.gmk:207: create-source-revision-tracker] Error 2
gmake[1]: Leaving directory '/wrkdirs/usr/ports/java/openjdk8/work/jdk8u-jdk8u342-b07.1'
===> Compilation failed unexpectedly.
Try to set MAKE_JOBS_UNSAFE=yes and rebuild before reporting the failure to
the maintainer.
*** Error code 1
is choking on the spaces here in the vendor arg: https://github.com/freebsd/freebsd-ports/blob/main/java/openjdk8/Makefile#L70 patch:
root@deblndw011x:/var/poudriere/ports/default/java/openjdk8 (2022Q4-custom *)
# git diff -U0
diff --git a/java/openjdk8/Makefile b/java/openjdk8/Makefile
index 2638b33328ab..96e1e74d1156 100644
--- a/java/openjdk8/Makefile
+++ b/java/openjdk8/Makefile
@@ -70 +70 @@ CONFIGURE_ARGS+= --with-update-version=${JDK_UPDATE_VERSION}
-CONFIGURE_ARGS+= --with-vendor-name="OpenJDK BSD Porting Team" \
+CONFIGURE_ARGS+= --with-vendor-name="OpenJDK_BSD_Porting_Team" \
makes it work. Will report also tomorrow as well.
That's a make
thing Poudriere can't do much about.
That's a
make
thing Poudriere can't do much about.
But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.
That's a
make
thing Poudriere can't do much about.But it does work when I am in the jail or just doing the from a host. I will report anyway for others which might trip over. Hopefully, you can explain why BSD make (?) fails here in this setup.
Forget that, this is a problem in the build scripts of the OpenJDK, bad use of printf
. I have created a patch and will create a bug with the OpenJDK 8 port.
In https://blog.shatow.net/posts/2013-07-21-managing-role-based-freebsd-servers-with-meta-packages-and-poudriere/ I cover how I do this. I just make a separate meta package for each jail and then can use 1 unified repository. Of course this only works if you do not have packages with conflicting options.
I was caught up again by this. Read your post. I understand your approach, but a see a few issues:
- Since you use metaports the ports you actually need are then automatic ports which means subject to deletion by
pkg autoremove
. This I don't want.To clarify something the dependencies are automatic but do not show in
pkg autoremove
because the metaports are non-automatic and still installed. Sopkg autoremove
shows nothing typically.pkg noauto
is quite clean and only shows pkg and the metaports.The leaf packages only show up in
pkg autoremove
if I trim them from the metaport and upgrade it.Here is an example from some of my jails. The matrix jail metaport used to depend on conduit but no longer does so it wants to delete it.
samba# pkg noauto pkg-1.18.4 local-fileserver-meta-20210811_12 local-jail-meta-20220925_7 local-postfix-relay-meta-20210527_18,2 samba# pkg autoremove -n Checking integrity... done (0 conflicting) Nothing to do. roundcube# pkg noauto pkg-1.18.4 local-jail-meta-20220925_7 local-postfix-relay-meta-20210527_18,2 local-webmail-meta-20220802_3 roundcube# pkg autoremove -n Checking integrity... done (0 conflicting) Nothing to do. prometheus# pkg noauto pkg-1.18.4 prometheus-2.36.0_5 local-jail-meta-20220925_7 local-postfix-relay-meta-20210527_18,2 prometheus# pkg autoremove -n Checking integrity... done (0 conflicting) Nothing to do. matrix# pkg noauto pkg-1.18.4 local-jail-meta-20220925_7 local-matrix-meta-20221030_4 local-postfix-relay-meta-20210527_18,2 matrix# pkg autoremove -n Checking integrity... done (0 conflicting) Deinstallation has been requested for the following 3 packages: Installed packages to be REMOVED: matrix-conduit: 0.4.0_2 py39-pip: 22.2.2 py39-virtualenv: 16.7.5 Number of packages to be removed: 3 The operation will free 69 MiB.
For what it's worth I am trying to wrap up #822 this week. It will significantly reduce port rebuilds to only when a version or
PORTREVISION
change happens (and sane things likeOPTIONS
). It won't force rebuild everything so much anymore.
This adds up. I have, indeed, a very similar approach, but without the meta ports, so called "packages classes". I will consider this in the next round of improvements. The downside is that you have an extra layer which simplifies installation, but the actually ports you use aren't direct anymore and these are which I document. In anycase, both approaches work and have pros and cons. Thanks for sharing.
Prerequisites
I believe that this isn't possible yet, so seeking for advice or need to create a feature request.
I have a repo with custom ports which aren't and won't be in the FreeBSD ports tree. One port is not selfcontained:
My jails have a base set of ports which include both dependencies from
RUN_DEPENDS
. That repos is build with:Now I want to build this custom tree:
This of course will trigger the build of both runtime dependencies which I don't want since those are already in
ldadw-base
. So I have added both ports to ablacklist
file and poudriere tells me:My port is skipped. My question: Is there anyway to tell poudriere to continue with the port and assume that the dependencies will be provided at runtime? This of course does not apply to build dependencies, but this could also be solved by compiling them, but not exposing them in the repo metadata when hosted for pkg.
Any help is appreciated. The only workaround I see is to remove the
RUN_DEPENDS
or move this port toldadw-base
, but other maybe upcoming custom ports will suffer from the same issue.