search

rockstor / rockstor-rpmbuild

RPM build specifications used to create the distributed 'rockstor' package from rockstor-core and rockstor-jslibs GitHub releases.
GNU General Public License v3.0
2 stars 3 forks source link

Improve BuildRequires use #28

Closed phillxnet closed 1 year ago

phillxnet commented 1 year ago

We currently make almost no use of the BuildRequires section. It is proposed that we bring this key rpmbuild feature into play more to enable a better out-of-the-box experience re rpmbuild runs on say Minimal-VM images. Our new Poetry based install method requires that we build many of our required dependencies, and so we have specified these within the final install dependencies section. But this ignores the initial rpmbuild requirements that can be well served by BuildRequires.

These changes should be address first in the testing branch.

phillxnet commented 1 year ago

The results of some initial experimentation with this issue look promising:

lbuildvm:/opt/rockstor-rpmbuild # rpmbuild -ba -v rockstor.spec
warning: Downloading https://github.com/rockstor/rockstor-core/archive/refs/tags/5.0.1-0.tar.gz to /usr/src/packages/SOURCES/5.0.1-0.tar.gz
warning: Downloading https://github.com/rockstor/rockstor-jslibs/archive/refs/tags/5.0.1.tar.gz to /usr/src/packages/SOURCES/5.0.1.tar.gz
error: Failed build dependencies:
        dbus-1-devel is needed by rockstor-5.0.1-0.x86_64
        gcc-c++ is needed by rockstor-5.0.1-0.x86_64
        glib2-devel is needed by rockstor-5.0.1-0.x86_64
        postgresql13 is needed by rockstor-5.0.1-0.x86_64
        postgresql13-server is needed by rockstor-5.0.1-0.x86_64
        postgresql13-server-devel is needed by rockstor-5.0.1-0.x86_64
        python3-devel is needed by rockstor-5.0.1-0.x86_64

This is a far faster and more informative fail than during the Poetry venv creation for the %check run of all out tests.

I'll continue with abstracting all that is needed to build - on top of rpm-build's own dependencies, which do overlap with our requirements a little:

rpm-build dependencies (Leap 15.5 Minimal_VM image)

lbuildvm:/opt # zypper install --no-recommends rpm-build
...
The following 31 NEW packages are going to be installed:
  binutils bzip2 cpp cpp7 dwz gcc gcc7 gettext-tools glibc-devel glibc-locale libasan4 libatomic1 libcilkrts5 libctf-nobfd0 libctf0 libgomp1 libisl15 libitm1
  liblsan0 libmpc3 libmpfr6 libmpx2 libmpxwrappers2 libtsan0 libubsan0 libxcrypt-devel linux-glibc-devel make patch python-rpm-macros rpm-build

31 new packages to install.
Overall download size: 51.8 MiB. Already cached: 0 B. After the operation, additional 370.6 MiB will be used.
Continue? [y/n/v/...? shows all options] (y): y
phillxnet commented 1 year ago 
zypper in --no-recommends dbus-1-devel gcc-c++ glib2-devel postgresql13 postgresql13-server postgresql13-server-devel python3-devel

Followed by:

lbuildvm:/opt/rockstor-rpmbuild # rpmbuild -ba -v rockstor.spec
...
+ /usr/bin/git init -q
/var/tmp/rpm-tmp.aVuAjQ: line 47: /usr/bin/git: No such file or directory
error: Bad exit status from /var/tmp/rpm-tmp.aVuAjQ (%prep)

So missing git: we use the git options available in %prep to prepare our code, adding accordingly to build requirements: they are likely also a recommended to rpm-build itself but I want to have the minimum required here to improve our chances of doing test rpm builds (and consequently running our tests) on small hosts.

zypper in --no-recommends git
phillxnet commented 1 year ago

We have of course to start, and possibly configure HBA access to postgresql before running rpmbuild. In our README.md we suggest using a rockstor install: but this limits the ease (read complicates) our ability to source test rpm build hosts.


  File "/usr/src/packages/BUILD/rockstor-core-5.0.1-0/.venv/lib64/python3.6/site-packages/psycopg2/__init__.py", line 127, in connect
    conn = _connect(dsn, connection_factory=connection_factory, **kwasync)
django.db.utils.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: No such file or directory

        Is the server running locally and accepting connections on that socket?

error: Bad exit status from /var/tmp/rpm-tmp.1nTwWV (%check)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.1nTwWV (%check)
``
phillxnet commented 1 year ago

Short of finding how we BuildRequires on a systemd service running we will have to advise:

systemctl start postgresql

but we then have, of course, to configure HBA on the local postgresql - as we do in initrock:

rpmbuild -ba -v rockstor.spec
...
django.db.utils.OperationalError: connection to server on socket "/var/run/postgresql/.s.PGSQL.5432" failed: FATAL:  Peer authentication failed for user "rocky"

error: Bad exit status from /var/tmp/rpm-tmp.9b3k9N (%check)

RPM build errors:
    Bad exit status from /var/tmp/rpm-tmp.9b3k9N (%check)
phillxnet commented 1 year ago

See the modifications used by our initrock (rockstor-pre.service) to acomodate our testing requirements of postgresql:

https://github.com/rockstor/rockstor-core/commit/500265bc670e7d056bfb274129c2b67a3249e884

phillxnet commented 1 year ago

To enable the use of this repo to help with testing rpmbuld function: we might consider adding a script that will allow configure Postgres sufficiently to run our tests - a requirement for a successful rpm package build. This may help with setting up transient workers to test-build rpm packages (and thus run our tests also) at key points in development.

phillxnet commented 1 year ago

HBA Host Based Authentication (Postgresql)

Akin to our behaviour in initrock (rockstor-pre.servcie invoked): https://github.com/rockstor/rockstor-core/blob/5252197d35cde9ca7ad95bfafd4ac859da9d6103/src/rockstor/scripts/initrock.py#L66C21-L66C62

we can instantiate pg_hba.conf as follows (dedicated test systems only): Note that this allows local host access as per rockstor's expectation to specifically named databases. N.B. the following example is for the testing branch:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/pg_hba.conf --output /var/lib/pgsql/data/pg_hba.conf
phillxnet commented 1 year ago

DB setup as per initrock (minimal for tests) - WILL DESTROY EXISTING DB.

To run our tests on a non prior rockstor instance we need to establish db as per initrock. Briefely abstracted:

DB tune:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/postgresql_tune.sql --output /tmp/postgresql_tune.sql 
su - postgres -c "psql -w -f /tmp/postgresql_tune.sql"

DB setup:

curl https://raw.githubusercontent.com/rockstor/rockstor-core/testing/conf/postgresql_setup.sql --output /tmp/postgresql_setup.sql
su - postgres -c "psql -w -f /tmp/postgresql_setup.sql"

In our proof of concept test-running Minimal-VM we have errors concerning pre-existing postgresql:

lbuildvm:/opt/rockstor-rpmbuild # su - postgres -c "psql -w -f /tmp/postgresql_setup.sql"
DROP DATABASE
DROP DATABASE
psql:/tmp/postgresql_setup.sql:3: NOTICE:  database "test_smartdb" does not exist, skipping
DROP DATABASE
psql:/tmp/postgresql_setup.sql:4: NOTICE:  database "test_storageadmin" does not exist, skipping
DROP DATABASE
psql:/tmp/postgresql_setup.sql:5: ERROR:  cannot drop the currently open database
DROP ROLE
CREATE ROLE
CREATE DATABASE
CREATE DATABASE
psql:/tmp/postgresql_setup.sql:10: ERROR:  database "postgres" already exists

However we have our slightly more than minimal db tune/setup for establishing a throw-away test runner via a %check rpmbuild:

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
.F......F.............E......E.........................F.....E.FF.....F.................................................F..EE.F....................F.......F.................................................................................................
...
phillxnet commented 1 year ago

OS setup related failures that could constitute incomplete OS for test runs (remit of issue re BuildRequires):

All of the above very likely represent unit test improvements to be had regarding unwanted/unrequired OS sensitivity during test runs. I.e. insufficient mocking of OS components.

The above likely false dependencies/requirements, that might otherwise be satisfied by additional BuildRequires entries, have been spun-off to the now linked issue:

"Improve OS independence re unit tests" https://github.com/rockstor/rockstor-core/issues/2633

EDIT: pull request for spin-off issue: https://github.com/rockstor/rockstor-core/pull/2637

phillxnet commented 1 year ago

Closing as: Fixed by #39

phillxnet commented 1 year ago

Follow-up: With the above Postgresql DB Tune & DB Setup preperations, we have:

5.0.1-0 release

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
.F......F.............E......E.........................F.....E.FF.....F.................................................F..EE.F....................F.......F.................................................................................................

5.0.2-0

+ poetry run django-admin test
Creating test database for alias 'default'...
Creating test database for alias 'smart_manager'...
System check identified no issues (0 silenced).
......................................................................................................................................................................................................................................................................
----------------------------------------------------------------------
Ran 262 tests in 19.366s

OK
...
Wrote: /usr/src/packages/SRPMS/rockstor-5.0.2-0.src.rpm
Wrote: /usr/src/packages/RPMS/x86_64/rockstor-5.0.2-0.x86_64.rpm

Where the latter host OS, in addition to Minimal_VM pre-installs, only has our BuildRequires installed. Tested using a recent OS image: openSUSE-Leap-15.5-Minimal-VM.x86_64-kvm-and-xen.qcow2 Downloaded via: wget https://download.opensuse.org/distribution/leap/15.5/appliances/openSUSE-Leap-15.5-Minimal-VM.x86_64-kvm-and-xen.qcow2

phillxnet commented 1 year ago

Test install of rpm built in last comment. Followin 15.5 adapted instructions re repos here: https://rockstor.com/docs/howtos/rpm_install.html

OBS home repo for our shellinabox

zypper --non-interactive addrepo --refresh -p105 https://download.opensuse.org/repositories/home:/rockstor/15.5/ home_rockstor

Requires

Our additional package requires in 15.5 are currently as follows:

zypper --no-gpg-checks install /usr/src/packages/RPMS/x86_64/rockstor-5.0.2-0.x86_64.rpm  

The following 146 NEW packages are going to be installed:
  adcli at avahi catatonit check-create-certificate containerd cron cronie cryptsetup cups-config cyrus-sasl cyrus-sasl-gssapi cyrus-sasl-plain deltarpm device-mapper dmraid dnf
  dnf-data dnf-plugins-core dnf-yum docker e2fsprogs ed fontconfig hdparm hwdata keyutils kpartx krb5-client libaio1 libarchive13 libavahi-client3 libavahi-common3 libavahi-core7
  libbasicobjects0 libcares2 libcollection4 libcomps0 libcups2 libdaemon0 libdevmapper-event1_03 libdhash1 libdnf2 libdns_sd libduktape206 libevent-2_1-8 libfontconfig1 libfreeipmi17
  libgd3 libHX28 libini_config5 libipmimonitoring6 libjbig2 libjpeg8 libldb2 liblmdb-0_9_30 libltdl7 libmodulemd2 libndp0 libneon27 libnscd1 libnss_nis2 libpath_utils1
  libpolkit-agent-1-0 libpolkit-gobject-1-0 libpowerman0 libpwquality1 libref_array1 librepo0 libsensors4 libsnmp40 libsss_certmap0 libsss_idmap0 libsss_nss_idmap0 libtalloc2 libtdb1
  libtevent0 libtiff5 libupsclient1 liburing1 libusb-0_1-4 libwebp7 libX11-6 libX11-data libXau6 libxcb1 libXpm4 mailx nano net-snmp NetworkManager NetworkManager-branding-openSUSE
  nfs-client nfsidmap nfs-kernel-server nginx nss-mdns ntp nut nut-drivers-net perl-SNMP perl-Term-ReadKey polkit postfix python3-curses python3-dnf python3-dnf-plugins-core python3-gpg
  python3-hawkey python3-libcomps python3-libdnf python3-libmodulemd python3-python-dateutil python3-sssd-config realmd realmd-lang rockstor rpcbind runc samba samba-client
  samba-client-libs samba-libs samba-winbind samba-winbind-libs shellinabox smartmontools snmp-mibs sssd sssd-ad sssd-common sssd-dbus sssd-krb5-common sssd-ldap sssd-tools sudo
  systemtap-runtime system-user-daemon system-user-lp system-user-upsd thin-provisioning-tools typelib-1_0-Modulemd-2_0 usbutils wpa_supplicant ypbind yp-tools

146 new packages to install.
Overall download size: 85.8 MiB. Already cached: 0 B. After the operation, additional 331.9 MiB will be used.