Improved installer - Githubissues

DevonianTeuchter commented 3 months ago

Update to the installer primarily to handle install direct from file but includes:

analyzes install packages to determine if any are local files
implements capability to allow for custom repositories, currently complete GitHub mirror or a local directory; defaults to z/OS Open Tools main GH repo
generates install graph in advance to allow for space calculation up-front
implements pre/post phases to allow scripts to run at certain times. Default scripts: -- auto-cache-clean - clean package cache after installation transaction completes [enabled by default] -- man-db - run man-db automagically after install/alt/remove transactions [enabled if man-db package is installed] -- caveats - display any "caveats" that a package might have after installation [a caveat being something an end user has to either run, configure or be aware of when using a certain package]
implements an a/all option for y/n questions to automatically answer yes
generates a package database, updated during install/alt/removal
performance improvements to queries, utilising package db and jq
split install into "upgrade" and "install" scripts
split query into "query" and "list" scripts
multiple bug fixes and shellcheck updates
removes [broken] customized filesystem type logic; always use default of usr/local/zopen for package installation

Should resolve/improve: #733 #717 #701 #694 #688 #675 #673 #661 #643 #633 #619 #617 #605 #567 #566 #558 #533 #472 #431 #381 #339 #240

IgorTodorovskiIBM commented 3 months ago

Impressive amount of changes! This will take me a few days to review!

v1gnesh commented 3 months ago

With a minor edit to the last line w.r.t issues closed, we can make GH auto-close them on the merge of this PR. https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue#linking-a-pull-request-to-an-issue-using-a-keyword

DevonianTeuchter commented 3 months ago

With a minor edit to the last line w.r.t issues closed, we can make GH auto-close them on the merge of this PR.

My choice of words was deliberate to ensure this didn't happen ;-) As there is a lot of churn and some of the issues are subjective, explicitly checking those issues to see if the submitter is happy with the update is advised.

DevonianTeuchter commented 3 months ago

Investigating test harness failures...

lbdyck commented 3 months ago

Impressive

IgorTodorovskiIBM commented 3 months ago

I cloned this branch and ran zopen init on my existing file system and got this? Are we able to get it to work with existing file systems?

. ./.env
zopen init -y
Initializing zopen framework
- Enter fully-qualified path to your zopen root directory (default: /home/itodoro/zopen):

- Binaries will be symlinked under "/home/itodoro/zopen/usr/local/bin". Libraries will be symlinked under "/home/itodoro/zopen/usr/lib"
- Packages will be installed and maintained under the directory '/home/itodoro/zopen/'.
- Collecting usage statistics: true
- Re-initializing; zopen-config will be re-created
Generating File System
- Setting certificate environment variable
- Root FS available at /home/itodoro/zopen
Generating zopen-config
- Created config in /home/itodoro/zopen/etc/zopen-config
Installing zopen pre-requisites
/home/itodoro/projects/meta_new/packages/curl-8.4.0.20231011_155722.zos.pax.Z
- Sourcing environment
Processing z/OS Open Tools configuration...DONE
Updating CA Certificate
Getting latest CA Certificate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  224k  100  224k    0     0  1515k      0 --:--:-- --:--:-- --:--:-- 1515k
Re-creating /home/itodoro/zopen/etc/pki/tls/certs/cacert.pem
- zopen bootstrapping complete
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./bump/./bump'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./bzip2/./bzip2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./cli/./cli'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./expat/./expat'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./expect/./expect'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./findutils/./findutils'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gettext/./gettext'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gperf/./gperf'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./gpg/./gpg'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./help2man/./help2man'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libgdbm/./libgdbm'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libmd/./libmd'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libpcre/./libpcre'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libpcre2/./libpcre2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./libxml2/./libxml2'. Bad package install?
***WARNING: No metadata.json found in '/home/itodoro/zopen/usr/local/zopen/./llama/./llama'. Bad package install?
jq: invalid JSON text passed to --argjson
Use jq --help for help with command-line options,
or see the jq manpage, or online docs  at https://stedolan.github.io/jq
***ERROR: Could not add metadata for '' to install tracker. Run zopen --re-init to attempt regeneration.

IgorTodorovskiIBM commented 3 months ago

On a new zopen file system:

zopen init
Initializing zopen framework
- Enter fully-qualified path to your zopen root directory (default: /home/itodoro/zopen):
/home/itodoro/zopen-russell
- Binaries will be symlinked under "/home/itodoro/zopen-russell/usr/local/bin". Libraries will be symlinked under "/home/itodoro/zopen-russell/usr/lib"
- Packages will be installed and maintained under the directory '/home/itodoro/zopen-russell/'.
- Collecting usage statistics: true
Do you want to continue? [y/n]
y
Generating File System
- Setting certificate environment variable
- Root FS available at /home/itodoro/zopen-russell
Generating zopen-config
- Created config in /home/itodoro/zopen-russell/etc/zopen-config
Installing zopen pre-requisites
/home/itodoro/projects/meta_new/packages/curl-8.4.0.20231011_155722.zos.pax.Z
Setting up curl...
Setup completed.
Setting up jq...
Setup completed.
- Sourcing environment
Updating CA Certificate
Getting latest CA Certificate
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  224k  100  224k    0     0  1458k      0 --:--:-- --:--:-- --:--:-- 1465k
Re-creating /home/itodoro/zopen-russell/etc/pki/tls/certs/cacert.pem
- zopen bootstrapping complete
- Configured zopen to use stable releaseline packages
find: FSUM6512 unable to access "./*/.": EDC5129I No such file or directory.
***ERROR: Unable to update the package db

I have z/OS Open Tools find in my PATH

DevonianTeuchter commented 3 months ago

I cloned this branch and ran zopen init on my existing file system and got this? Are we able to get it to work with existing file systems?

I had tested migrating from 0.8.0 and 0.8.2 - but that was using the --refresh rather than re-init options... let me check that migration path

find: FSUM6512 unable to access "./*/.": EDC5129I No such file or directory. ***ERROR: Unable to update the package db

Another thing to investigate...

DevonianTeuchter commented 2 months ago

@IgorTodorovskiIBM I've pushed a further commit that should resolve the issue when running in a new environment (ie. zopen init). slight issue is that when using the checked out version, it will automatically update meta to the latest in the repo so you then have to reapply the clone into the zopen filesystem. No easy way around that as copying the installing meta into the zopen filesystem could leave the system with an older meta than in the repo going forward, depengind on how quickly a packaged release is made available [if that makes sense]. Some of the changes in the commit are to fix the custom repo handling also.

DevonianTeuchter commented 1 month ago

If anyone has a chance [and is feeling up for a challenge!], the new installer for zopen is available for testing. It has to be cloned from git since it cannot be pushed to the repos until tested/merged: git clone --single-branch -b new_installer https://github.com/ZOSOpenTools/meta.git
and there's a few hoops to jump through [since it's not available in the repos, zopen has to be tricked to use it] If interested, please see below for more information

The updated zopen will generate metadata when first "upgraded" so this needs more than just copying the files across - there needs to be a refresh. Once cloned: = Installing into a clean test environment:

cd into the cloned/meta/bin directory
run ./zopen init -y ""
Run the instruxtions below to integrate the new meta in place of the "latest" as installed during the initialization = An existing zopen environment can be upgraded by:
move the cloned directory to <zopen-rootfs>/usr/local/zopen/meta/meta.git
run zopen alt --select meta and pick the git version of zopen (which should be listed)
run zopen init --refresh - this should generate a new package database. This is basically what happens under-the-covers during a meta upgrade, just manually run. There shouldn't be any errors when running the commands - at least, it works on my system anyway! :-) It should be possible to move back to an older meta by using zopen alt --select meta, the new files/directories generated in <zopenrootfs>/etc/zopen will however need to be manually deleted (as the older meta releases have no knowledge of these files) Note that for testing purposes, cloning directly into a directory meta.[xyz] under usr/local/zopen/meta means it is possible to test different versions using zopen alt to switch; using a symlink within the u/l/z/meta directory structure will not currently work due to how the "in-use" version is detected Any and all feedback would be appreciated - as this is a bit more than a bug fix or basic feature addition, having extra testing would be useful!

IgorTodorovskiIBM commented 1 month ago

[ITODORO@ZOSCAN2B ~/projects]$ zopen install ~/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
curl: (37) Couldn't open file /home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Could not download from file:///home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z. Correct any errors and potentially retry
[ITODORO@ZOSCAN2B ~/projects]$ zopen install $HOME/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
curl: (37) Couldn't open file /home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Could not download from file:///home/itodoro/projects//home/itodoro/zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z. Correct any errors and potentially retry
[ITODORO@ZOSCAN2B ~/projects]$ zopen install zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: The following ports could not be installed:
***ERROR:     zopen/var/cache/zopen/tmux-heads.3.3a.20240308_143819.zos.pax.Z
***ERROR: Check port name(s), remove any extra 'port' suffixes and retry command.
[ITODORO@ZOSCAN2B ~/projects]$

ZOSOpenTools / meta

Improved installer #742