.spec eligibility for autonomously compileable packages from automated Version bumps

dcasota commented 1 year ago

Is your feature request related to a problem? Please describe.

Hi,

Please introduce a srpms funnel for autonomously compiled packages from automated Version bumps.

The reason for this request: Balancing new manufacturer package releases and fixing existing package versions seems more and more to become a challenge at scale.

A few culprits however can be avoided.

improve patch management methods a. avoid version independent patch calls of version dependent patch content e.g. libsepol-3.4 + fix-validation-of-user.patch b. avoid version independent patch calls of patch file names with version dependencies e.g. coreutils-9.1 + coreutils-9.1-i18n-1.patch b. introduce a .spec pattern to test with enabled/disabled patches. Introduce a patches naming convention. e.g.
```
%if 0%{?check_with_patches}
%patches
Patch0: %{name}-patchdescription0-000.patch
Patch1: %{name}-patchdescription0-001.patch
Patch2: %{name}-patchdescription1-000.patch
%endif
```
avoid version dependent source files Avoid unchecked existence of files which might only exist in a specific version.
improve .spec integrity a. python3 Names with python3-%{name} often differ from the package name and in sections %prep, %files b. use of variables E.g. use name variable most consequently c. Source0 Most Source0 urls are outdated.
Look for package alternatives Packages not manufacturer-maintained for many years should be considered as eol / to be refactored.
describe how to use .spec calls when compiling a specific package For example: 0%{?with_check}, %{?_smp_mflags}

Taking an analytical approach to scouting and analyzing autonomously compiled packages helps to liberate time for other purposes than increased .spec eligibility for aVb.

Describe the solution you'd like

Add .spec guidelines how to make .spec files most eligible for autonomously compileable packages from automated Version bumps (aVb).

Consider using packages statistics. The ratio "compiled new packages from aVb"/("total packages" - "same latest version") per timeframe indicates the vitality of an implementation+ecosystem.

With a dedicated srpms funnel, the team gets more choice to consider implementing a patch, a cve fix, etc. and to refer to a newer package version from aVb.

Here basic Ph 5.0 x86_64 statistics after some aVb tests of my own.

total packages	977
failed detection, various reasons	208
same latest version	371
downloaded manufacturer updates	398
-------------	-------------
compiled new packages from aVb	96
test cases new packages content	(todo)

For testing purposes, for each of those 398 detected manufacturer updates not included in latest Photon OS, its source has been downloaded and the .spec file has been created. For example, libpipeline-1.5.7.tar.gz with libpipeline-1.5.7.spec. The recent amount of 96 successfully compiled new packages from aVb contributes to the real benefits: time-savings, more granular choice and agility.

Describe alternatives you've considered

No response

Additional context

Sun May 21 2023 Ph 5.0 downloaded manufacturer updates

(Sun May 21 2023 Ph 3.0 downloaded manufacturer updates)

(Sun May 21 2023 Ph 4.0 downloaded manufacturer updates)

-- Mon May 29 2023 Ph 5.0 downloaded manufacturer updates testing SPECS - rename to .tar

(Mon May 29 2023 Ph 4.0 downloaded manufacturer updates)

(Mon May 29 2023 Ph 3.0 downloaded manufacturer updates)

Sun Jun 3 2023 Ph 5.0 downloaded manufacturer updates

(Sun Jun 3 2023 Ph 4.0 downloaded manufacturer updates)

(Sun Jun 3 2023 Ph 3.0 downloaded manufacturer updates)

Wed Jun 15 2023 Ph 5.0 downloaded manufacturer updates

(Wed Jun 15 2023 Ph 4.0 downloaded manufacturer updates)

(Wed Jun 15 2023 Ph 3.0 downloaded manufacturer updates)

dcasota commented 1 year ago

Advance and protect the (open-source) profession. Open issue(s) with no discussion -> close ticket.

dcasota commented 2 months ago

@Vasavisirnapalli

There is an increased amount of differences of same packages in Ph3/4/5/6. Most of them can be justified with certainty as false-positive.
The amount of packages eligible for an update, justified by a new vendor release, actually increased to over six hundred, for sure with a few false positives.

Differences

Differences Ph4.0 to Ph5.0

Spec                photon-4.0  photon-5.0
atk.spec            2.38.0-2    2.38.0-1
bazel.spec          6.1.2-4     5.3.2-5
calico.spec         3.26.4-3    3.26.1-5
calico-bgp-daemon.spec      0.2.2-27    0.2.2-16
calico-libnetwork.spec      1.1.3-26    1.1.3-15
c-ares.spec         1.19.1-2    1.18.1-1
cassandra.spec          4.0.10-4    4.0.8-5
cloud-init.spec         24.02.2002  24.1.4-1
cloud-network-setup.spec    0.2.2-9     0.2.2-7
cni.spec            1.1.1-11    1.1.1-8
conmon.spec         2.1.7-3     2.1.7-1
coredns.spec            1.11.1-5    1.11.1-2
coreutils.spec          09.01.2004  09.01.2003
cpio.spec           2.13-9      2.13-8
curl.spec           8.7.1-3     8.1.2-4
cython3.spec            3.0a6-2     0.29.32-2
distcc.spec         03.04.2004  03.04.2003
docker.spec         24.0.5-7    24.0.5-6
docker-compose.spec     2.26.1-3    2.20.2-4
edgex.spec          2.2.0-17    2.2.0-12
elixir.spec         1.16.3-1    1.14.2-2
erlang.spec         26.2.5-1    25.1.2-3
etcd.spec           3.5.12-4    3.5.12-1
flannel.spec            0.22.0-8    0.22.0-7
fuse-overlayfs-snapshotter.spec 1.0.6-7     1.0.6-5
gdb.spec            13.02.2003  11.02.2009
gdk-pixbuf.spec         2.42.0-6    2.42.0-4
glide.spec          0.13.3-26   0.13.3-15
gnuplot.spec            5.4.6-1     5.4.5-3
go.spec             1.21.12-1   1.20.12-1
gst-plugins-bad.spec        1.22.7-3    1.21.3-1
gstreamer.spec          1.22.7-1    1.21.3-4
gstreamer-plugins-base.spec 1.22.7-2    1.21.3-2
gtk3.spec           3.23.3-11   3.23.3-9
harfbuzz.spec           7.0.1-3     7.0.1-2
heapster.spec           1.5.4-27    1.5.4-16
httpd.spec          2.4.62-1    2.4.59-1
httpd-mod_jk.spec       1.2.48-12   1.2.48-11
ImageMagick.spec        7.1.1.11-4  7.1.0.47-4
iptables.spec           1.8.9-4     1.8.9-3
jq.spec             01.06.2003  01.06.2002
kafka.spec          3.4.0-4     3.3.1-5
kbd.spec            2.3.0-2     2.2.0-1
kubernetes.spec         1.27.13-3   1.27.3-9
kubernetes-dashboard.spec   2.7.0-13    2.7.0-11
kubernetes-metrics-server.spec  0.3.7-25    0.3.7-14
less.spec           654-1       608-3
libcgroup.spec          3.0.0-2     3.0.0-1
libldb.spec         2.7.2-2     2.6.1-3
libmspack.spec          0.10.1alpha-3   0.10.1alpha-2
libnss-ato.spec         20201005-2  20201005-1
libslirp.spec           4.7.0-2     4.7.0-1
libssh.spec         0.10.6-1    0.10.5-2
libssh2.spec            1.11.0-2    1.10.0-4
libtiff.spec            4.5.1-5     4.5.1-1
libuv.spec          1.45.0-2    1.44.2-1
libX11.spec         1.8.5-2     1.8.5-1
linux-firmware.spec     20230320-1  20230207-1
Linux-PAM.spec          1.5.3-3     1.5.3-1
logrotate.spec          3.21.0-1    3.20.1-1
memcached.spec          1.6.22-1    1.6.15-3
mesa.spec           23.0.0-3    23.0.0-2
monitoring-plugins.spec     2.3.1-3     2.3.1-2
netcat.spec         0.7.1-6     0.7.1-5
network-event-broker.spec   0.3-10      0.3-9
nghttp2.spec            1.57.0-2    1.48.0-7
nginx.spec          1.26.2-1    1.25.2-5
ntp.spec            4.2.8p18-1  4.2.8p17-2
openjdk11.spec          11.0.22-1   11.0.20-6
openjdk17.spec          17.0.10-1   17.0.8-2
openresty.spec          1.21.4.3-1  1.21.4.1-9
open-vm-tools.spec      12.4.5-1    12.3.5-5
pcstat.spec         Jan 28      Jan 19
photon-os-container-builder.spec    0.1.1-9 0.1.1-7
photon-os-installer.spec    02.07.2004  02.07.2001
pixman.spec         0.40.0-2    0.40.0-1
pkg-config.spec         0.29.2-5    0.29.2-3
pmd-ng.spec         0.1-8       0.1-7
podman.spec         4.5.1-7     4.5.1-5
postgresql13.spec       13.16-1     13.14-7
postgresql14.spec       14.13-1     14.12.2001
postgresql15.spec       15.08.2001  15.07.2001
python3-pip.spec        23.3.2-2    23.3.2-1
python-altgraph.spec        0.17-2      0.17.2-2
python-argparse.spec        1.4.0-2     1.4.0-1
python-backports_abc.spec   0.5-5       0.5-1
python-boto.spec        2.49.0-4    2.49.0-3
python-certifi.spec     2023.11.17-2    2022.6.15-2
python-ConcurrentLogHandler.spec       0.9.23-1 0.9.20-2
python-cryptography.spec    41.0.7-2    38.0.1-2
python-etcd.spec        0.4.5-8     0.4.5-7
python-pycryptodome.spec    3.20.0-1    3.12.0-2
python-pycryptodomex.spec   3.20.0-1    3.15.0-1
python-PyJWT.spec       2.8.0-1     2.6.0-1
python-pyOpenSSL.spec       23.2.0-2    22.0.0-3
python-pyvmomi.spec     8.0.2.0.1-1 7.0.3-1
python-PyYAML.spec      5.4.1-3     5.4.1-2
python-setuptools-rust.spec 1.5.2-2     1.5.2-1
python-urllib3.spec     1.26.19-1   1.25.11-4
python-zope.event.spec      4.5.0-3     4.5.0-2
runc.spec           1.1.12-4    1.1.12-1
samba-client.spec       4.18.8-3    4.17.5-11
selinux-policy.spec     36.5-8      36.5-6
slirp4netns.spec        1.2.0-5     1.2.0-1
socat.spec          2.0.0.b9-3  1.7.4.4-2
stig-hardening.spec     01.05.2002  01.02.2004
strongswan.spec         5.9.8-4     5.9.8-3
sudo.spec           1.9.15p5-1  1.9.14p3-2
syslog-ng.spec          4.3.1-6     4.3.1-5
sysstat.spec            12.7.2-1    12.7.1-1
telegraf.spec           1.28.1-6    1.28.1-2
traceroute.spec         2.1.3-1     2.1.0-5
unbound.spec            1.21.0-1    1.17.0-3
vernemq.spec            2.0.1-2     1.12.6.2-5
vim.spec            9.1.0682-1  9.0.2142-1
wavefront-proxy.spec        13.04.2002  13.04.2001
wireshark.spec          4.2.7-1     4.0.12-1

Differences 3.0 to 4.0

Spec            photon-3.0  photon-4.0
apache-ant.spec     1.10.12-3   1.10.10-5
autofs.spec     5.1.6-2     5.1.6-1
calico-bgp-daemon.spec  0.2.2-30    0.2.2-27
ddclient.spec       3.9.1-1     3.9.0-2
distrib-compat.spec 0.1-12      0.1-2
erlang-sd_notify.spec   01.01.2004  01.01.2003
geos.spec       3.8.1-2     3.8.1-1
glide.spec      0.13.3-27   0.13.3-26
go-md2man.spec      2.0.0-26    2.0.0-25
heapster.spec       1.5.4-30    1.5.4-27
inotify-tools.spec  3.13-3      3.13-2
ipxe.spec       20180717-2  1.20.1-3
json_spirit.spec    04.08.2004  04.08.2003
kubernetes-dns.spec 1.22.23-6   1.22.20-7
libjpeg-turbo.spec  2.1.0-3     2.1.0-2
libpng.spec     1.6.40-1    1.6.37-2
libtiff.spec        4.5.1-6     4.5.1-5
linuxptp.spec       4.0-2       4.0-1
lxcfs.spec      4.0.6-2     4.0.5-2
lz4.spec        1.9.3-2     1.9.2-2
mercurial.spec      5.9.3-1     5.5.1-2
nginx-ingress.spec  3.2.0-6     2.3.0-15
pcstat.spec     Jan 33      Jan 28
photon-upgrade.spec 01.01.2002  01.01.2001
runit.spec      2.1.2-6     2.1.2-5
stig-hardening.spec 01.07.2001  01.05.2002
synce4l.spec        1.0.0-1     0.8.0-1
sysdig.spec     0.30.2-6    0.30.2-1
tpm2-abrmd.spec     2.4.1-2     2.3.3-3
tzdata.spec     2023c-1     2022g-1
unixODBC.spec       2.3.12-1    2.3.9-2
unzip.spec      6.0-18      6.0-16
xerces-c.spec       3.2.5-1     3.2.4-1
xmlstarlet.spec     1.6.1-3     1.6.1-2
xtrans.spec     1.4.0-2     1.4.0-1

Packages eligible for automatic version bumps

The Photon OS team does not have the luxury to maintain over one thousand packages manually. I mentioned this trend quite a few times during the last three years. It is a relative performance indicator in comparison e.g. to open-source azurelinux releases.

photonos-urlhealth-5.0_202409072346.xlsx

The idea to improve throughput is a converter app which does several things per package

version bumps
- crafts the dockerfile
- parses the required packages per spec file and creates each package .yml file
- depend a bot + schedule
- successful version bumps are eligible for approval (e.g. #VEBA for: if this, then this)
cve bumps
- parse available cve sources (e.g. github) and crafts the spec file with patches
- depend a bot + schedule
- successful cve bumps are eligible for approval (e.g. #VEBA for: if this, then this)

YustasSwamp commented 2 months ago

Hi @dcasota, your concern is valid. We are moving towards such an automation. Common branch and automatic version bump will play a key role here.

dcasota commented 2 months ago

Hi Alexey, glad to hear that. github.com/advisories with cve entries is growing at burnout speed. In August, the photon os team members in charge mastered a bunch of kernel issues with severity high and meanwhile 45 new cves with severity high or critical were published. Take care.

edited: KPI = Time to remediate in days vs. number of vulnerabilities

dcasota commented 2 months ago

@YustasSwamp A few questions. The actual phase of automation progress is highly interesting. Thinking about realizing the plan below until YE.

Collaboration might become a challenge. Does the Photon OS team review pull requests regularly? Is it welcome that volunteers participate with pull requests? I don't want to create a Sisyphean task therefore the question.

A. Stage a Photon OS package builder bot environment 1) Setup VMware Edge Compute Stack 2) Install and run your own Photon OS package builder bot node scheduler 3) Per successful PR from D) --> merge PR to upstream github.com/vmware/photon/pulls

B. Version bump bot 1) run data scraping (github, pypi, etc.) 2) create new spec+ rpm package per scraped vendor package source update 3) run package installation tests 4) create automatic pull request

C. CVE patch bot 1) run data scraping (advisories, patches) 2) create updated spec+ rpm package per cve patch source 3) run package installation tests 4) create automatic pull request

D. Pull Request validation bot 1) run various checks (spec files, signature, etc.) 2) plugin and run quality tests per package 3) run Photon OS builder check 4) create automatic pull request review report

vmware / photon