Bug: failed first-time deploy

[freephile]

TASK [mediawiki : (Re-)build SemanticMediaWiki data for: demo] 14:06

/bin/sh: bash /opt/.deploy-meza/smw-rebuild-all.sh \"demo\" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log: No such file or directory"

[freephile]

Explanation

This error ONLY happens on a first-time deploy on a brand new environment.

Here's why: deploy will create the demo wiki on the first deploy if no wiki exists on the target.

deploy will also indiscriminately run "Rebuild SMW data" for every wiki, and it apparently happens before the demo wiki is even built (meaning it lacks the database tables associated with SMW or at least - from the error message - lacks any log history of prior SMW rebuilds).

You can't re-build what isn't there. And it doesn't make sense to rebuild something that is brand new. It doesn't even make sense to re-build SMW data indiscriminately for every deploy.

So, ultimately, the best practice would be to separate SMW tasks into a separate playbook that can be called in a controlled, scheduled, well-planned way rather than a process that is tightly coupled to 'deploy' which tends to be called on an ad-hoc basis. In other words, 'deploy' is too powerful and tries to do everything under the sun rather than focus on a single or even 'main' responsibility.

Solution

Fortunately, right now there is a tag smw-data that is attached to the Semantic MediaWiki data rebuilding tasks. So, the immediate fix to this problem is to simply use that tag in the first deploy of a new environment.

sudo meza deploy monolith --skip-tags smw-data,search-index -vvv

This needs to be added to the documentation for the CMTE procedure and any general reference about using Meza deploy.

[freephile]

Where is smw-data tag found

src/playbooks/rebuild-smw-and-index.yml: 22 tags: 23: - smw-data 24

src/roles/autodeployer/tasks/main.yml: 7 # overwrite: true 8: # extra_options: "--skip-tags smw-data,search-index" 9

src/roles/mediawiki/tasks/main.yml: 584 tags: 585: - smw-data 586

[freephile]

Solution Verification

3.1 - Start the RL8 Clone VM

### 3.1.5 [userx@localhost ~]$ cat /etc/os-release NAME="Rocky Linux" VERSION="8.9 (Green Obsidian)" ID="rocky" ID_LIKE="rhel centos fedora" VERSION_ID="8.9" PLATFORM_ID="platform:el8" PRETTY_NAME="Rocky Linux 8.9 (Green Obsidian)" ANSI_COLOR="0;32" LOGO="fedora-logo-icon" CPE_NAME="cpe:/o:rocky:rocky:8:GA" HOME_URL="https://rockylinux.org/" BUG_REPORT_URL="https://bugs.rockylinux.org/" SUPPORT_END="2029-05-31" ROCKY_SUPPORT_PRODUCT="Rocky-Linux-8" ROCKY_SUPPORT_PRODUCT_VERSION="8.9" REDHAT_SUPPORT_PRODUCT="Rocky Linux" REDHAT_SUPPORT_PRODUCT_VERSION="8.9" ### 3.1.6 sudo systemctl status fapolicyd Unit fapolicyd.service could not be found. ### 3.1.7 cat /proc/sys/crypto/fips_enabled 0 ### 3.1.8 sudo sestatus SELinux status: enabled SELinuxfs mount: /sys/fs/selinux SELinux root directory: /etc/selinux Loaded policy name: targeted Current mode: permissive Mode from config file: permissive Policy MLS status: enabled Policy deny_unknown status: allowed Memory protection checking: actual (secure) Max kernel policy version: 33 ### 3.1.9 ### 3.1.10 sudo cat /etc/fstab ``` # # /etc/fstab # Created by anaconda on Sun Dec 24 03:35:09 2023 # # Accessible filesystems, by reference, are maintained under '/dev/disk/'. # See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info. # # After editing this file, run 'systemctl daemon-reload' to update systemd # units generated from this file. # /dev/mapper/rl-root / xfs defaults 0 0 UUID=612247c4-ab60-4616-8fb7-22a9edb345e7 /boot xfs defaults 0 0 /dev/mapper/rl-swap none swap defaults 0 0 ``` ### 3.1.11 ### 3.1.12 sudo egrep 'python38|ansible' /etc/yum.repos.d/Rocky-AppStream.repo exclude=ansible python38 ### 3.1.13 repo id repo name appstream Rocky Linux 8 - AppStream baseos Rocky Linux 8 - BaseOS epel Extra Packages for Enterprise Linux 8 - x86_64 extras Rocky Linux 8 - Extras

[freephile]

3.2 - Install Meza on the RL8 Clone VM

I followed each step in the procedure as written. The only thing 'special' that I had to do here was to be sure to use the 'dev' branch `export MEZA_BRANCH_NAME='grc-atf-dev'` And of course I had to specify **smw-data** in skip-tags `sudo meza deploy monolith --skip-tags smw-data,search-index -vvv`

[freephile]

Sadly, this deploy did error - with a different error that we've seen before: the "wiki_id recursive templating error" aka wiki_id is undefined.

The confusing part of this error is that 'list_of_wikis' is defined and is a list of one: ['demo']

TASK [mediawiki : Check list of wikis] *****************************************
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:485
ok: [localhost] => {
    "list_of_wikis": [
        "demo"
    ]
}

TASK [mediawiki : Set fact - initiate empty list of wikis to rebuild smw and search data] ***
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:490
ok: [localhost] => {
    "ansible_facts": {
        "wikis_to_rebuild_data": []
    },
    "changed": false
}

TASK [Ensure defined wikis exist] **********************************************
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:495
ERROR! An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansib

Filesystem

Looking at what's been dropped onto the target already, we can see the conf dir and .deploy dir, but not yet the web dir.

[userx@localhost opt]$ ls -al /opt/conf-meza/public/wikis/
total 0
drwxr-xr-x. 3 meza-ansible wheel   18 Mar 23 19:46 .
drwxr-xr-x. 6 meza-ansible wheel  175 Mar 23 19:33 ..
drwxr-xr-x. 5 apache       apache 139 Mar 23 19:46 demo

[userx@localhost opt]$ ls -al /opt/.deploy-meza/public/wikis/demo/
total 56
drwxr-xr-x. 5 root root   139 Mar 23 19:46 .
drwxr-xr-x. 3 root root    18 Mar 23 19:46 ..
-rwxr-xr-x. 1 root root 41342 Mar 23 19:46 favicon.ico
-rwxr-xr-x. 1 root root 10922 Mar 23 19:46 logo.png
drwxr-xr-x. 2 root root    23 Mar 23 19:46 postLocalSettings.d
drwxr-xr-x. 2 root root    39 Mar 23 19:46 preLocalSettings.d
drwxr-xr-x. 2 root root    22 Mar 23 19:46 samlAuthorizations.d
-rwxr-xr-x. 1 root root     0 Mar 23 19:46 .smw.json

[userx@localhost opt]$ ls -al /opt/htdocs/
total 20
drwxrwxr-x.  5 apache       apache  138 Mar 23 19:46 .
drwxr-xr-x.  9 meza-ansible wheel   131 Mar 23 19:34 ..
-rwxr-xr-x.  1 root         root   1234 Mar 23 19:42 .htaccess
-rwxr-xr-x.  1 root         root    242 Mar 23 19:42 index.php
drwxrwxr-x. 17 meza-ansible wheel  4096 Mar 23 19:46 mediawiki
-rwxr-xr-x.  1 root         root    270 Mar 23 19:42 office.html
-rwxr-xr-x.  1 root         root    786 Mar 23 19:42 robots.txt
drwxr-xr-x.  4 meza-ansible wheel   103 Mar 23 19:26 ServerPerformance
drwxrwxr-x.  5 meza-ansible wheel   124 Mar 23 19:46 WikiBlender

[freephile]

The failure is avoided if we explicitly pass wiki_id as an extra var on the command line:

✅ [SOLVED] (WORKAROUND) sudo meza deploy monolith --skip-tags smw-data,search-index -vvv -e wiki_id=demo

But, since 'list_of_wikis' is defined (as seen in the debug output) and passed at evaluation time into the included task as a list, there's something we're not understanding about how that list is seen by Ansible and creates a templating error.

[freephile]

Procedure to RESET a test VM

Made some code changes and want to re-try the 'first-time deploy'?

The best approach is to have a snapshot or clone of a pristine system that you can just start over with. Failing that, you could try to delete most of what Meza lays down.

Here's how you can delete the 'demo' wiki to re-run a "first-time deploy":

sudo meza delete wiki-promptless monolith demo ; \
sudo rm -rf /opt/data-meza/backups/monolith/demo ; \
sudo rm -rf /opt/.deploy-meza/public/wikis/demo ; \
sudo rm -rf /opt/htdocs/demo ; \
sudo rm -rf /opt/htdocs/wikis/demo/ ; \

And possibly go even more nuclear PLEASE DON'T DO THIS IF YOU DON'T KNOW WHAT YOU'RE DOING! YOU MIGHT BE DELETING FILES THAT YOU DON'T WANT TO LOSE

sudo rm -rf /opt/conf-meza/ ; \
sudo rm -rf /opt/htdocs/ ; \
sudo rm -rf /opt/.deploy-meza/

sudo meza deploy monolith --skip-tags smw-data,search-index -vvv -e wiki_id=demo

PLAY RECAP ***** localhost : ok=454 changed=59 unreachable=0 failed=0 skipped=336 rescued=0 ignored=1

[freephile]

However, there continues to be a problem with 'create wiki' even after the demo wiki exists. After a successful initial 'deploy' which creates the 'demo' wiki, I had inconsistent results with 'create wiki' I created wikis

here is a "next run" create wiki

sudo meza create wiki-promptless monolith test 'The Test Wiki'

TASK [Re-sync config to app servers] ***

TASK [sync-configs : Ensure app servers have local config directory] *** ok: [localhost]

TASK [sync-configs : Ensure app servers have config from controller] *** changed: [localhost]

TASK [sync-configs : Ensure app servers local config directory and contents configured] *** changed: [localhost]

TASK [Use verify-wiki to make sure new wiki is setup] ** ERROR! An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>,

I tried to fix this up with

[userx@localhost meza]$ sudo git diff
diff --git a/src/roles/create-wiki-wrapper/tasks/main.yml b/src/roles/create-wiki-wrapper/tasks/main.yml
index 04b79f7..3dd947d 100644
--- a/src/roles/create-wiki-wrapper/tasks/main.yml
+++ b/src/roles/create-wiki-wrapper/tasks/main.yml
@@ -29,6 +29,8 @@
 - name: Use verify-wiki to make sure new wiki is setup
   include_role:
     name: verify-wiki
+  vars:
+    wiki_id: "{{ wiki_id }}"

Which seemed to work to make 'create-wiki' succeed, but then I found later problems or inconsistent / improper behavior because the newly created wikis weren't fully setup.

Continuing with the Test / Verification procedure

• Make sure you can login to the 'test' wiki as Admin with password 'wiki' sudo WIKI=test php /opt/htdocs/mediawiki/maintenance/createAndPromote.php --force --sysop --bureaucrat Admin wiki
• Browse to https://localhost to see the landing page
• Click through to the 'Test' wiki and perform the content tests

Although create-wiki worked without error, the Pretty URL redirect is not present yet (https://localhost/test -> 'Not Found'), and the site is unstyled if accessed directly (https://localhost/test/Main_Page). Both of these are a clear indication that 'create-wiki-wrapper' did not fully deploy the wikis.

I figured that a 'deploy' would at least cure the condition.

But meza deploy exits with error

TASK [mediawiki : Set fact - initiate empty list of wikis to rebuild smw and search data] *** task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:490 ok: [localhost] => { "ansible_facts": { "wikis_to_rebuild_data": [] }, "changed": false }

TASK [Ensure defined wikis exist] ** task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:495 ERROR! An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class

(repeat)

'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: An unhandled exception occurred while templating '{{ wiki_id }}'. Error was a <class 'ansible.errors.AnsibleError'>, original message: recursive loop detected in template string: {{ wiki_id }}. maximum recursion depth exceeded statically imported: /opt/meza/src/roles/verify-wiki/tasks/transfer-backup-to-db-master.yml statically imported: /opt/meza/src/roles/verify-wiki/tasks/import-wiki-sql.yml redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db statically imported: /opt/meza/src/roles/verify-wiki/tasks/transfer-backup-to-db-master.yml statically imported: /opt/meza/src/roles/verify-wiki/tasks/import-wiki-sql.yml redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db statically imported: /opt/meza/src/roles/verify-wiki/tasks/transfer-backup-to-db-master.yml statically imported: /opt/meza/src/roles/verify-wiki/tasks/import-wiki-sql.yml redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db redirecting (type: modules) ansible.builtin.mysql_db to community.mysql.mysql_db

PLAY RECAP ***** localhost : ok=354 changed=41 unreachable=0 failed=0 skipped=238 rescued=0 ignored=1

Exiting with return code 2

Need better environment

This is when I decided I need a better development and test environment than using VirtualBox and so I'm creating a true Rocky Linux VM with a Digital Ocean droplet.

[freephile]

I created a 4GB RL 8 droplet from the Digital Ocean web panel

1.5.11

sudo dnf update -y This is a big update

Skip 1.6 Guest Additions

We can skip Guest Additions because this is a true VM, not Virtualbox. But, in the step where epel-release is installed, (and in the Meza code that installs epel-release), I think we probably need to also run /usr/bin/crb enable to enable the CRB repository. Many EPEL packages require the CodeReady Builder (CRB) repository. If you install epel-release manually, the post-install scriptlet will advise this.

1.7 Tweaks

Make selinux permissive

sudo sed -i -e 's;SELINUX=enforcing$;SELINUX=permissive;g' /etc/selinux/config (confirm before and after with sestatus)

3.2 Install Meza

Results in a much different system with Ansible 2.15.3, Python 3.11.5 (not 3.6.8)

[root@rockylinux-s-2vcpu-4gb-nyc3-01 opt]# cd /opt/meza/config/
[root@rockylinux-s-2vcpu-4gb-nyc3-01 config]# ansible --version
ansible [core 2.15.3]
  config file = /opt/meza/config/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.5 (main, Oct 25 2023, 16:19:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True

Development Tools

I installed Ansible Dev Tools to get ansible-lint and others pip3 install ansible-dev-tools

[freephile]

The failure is avoided if we explicitly pass wiki_id as an extra var on the command line:

✅ [SOLVED] (WORKAROUND) sudo meza deploy monolith --skip-tags smw-data,search-index -vvv -e wiki_id=demo

[freephile]

ansible -m debug -a "var=hostvars['localhost']" localhost

(for a complete list of ansible host variables do ansible -m setup localhost)

shows that python is 3.11 and Ansible version is 2.15.3 on the DO Droplet

localhost | SUCCESS => {
    "hostvars['localhost']": {
        "ansible_check_mode": false,
        "ansible_config_file": "/opt/meza/config/ansible.cfg",
        "ansible_connection": "local",
        "ansible_diff_mode": false,
        "ansible_facts": {},
        "ansible_forks": 5,
        "ansible_inventory_sources": [
            "/etc/ansible/hosts"
        ],
        "ansible_playbook_python": "/usr/bin/python3.11",
        "ansible_python_interpreter": "/usr/bin/python3.11",
        "ansible_verbosity": 0,
        "ansible_version": {
            "full": "2.15.3",
            "major": 2,
            "minor": 15,
            "revision": 3,
            "string": "2.15.3"
        },
        "group_names": [],
        "groups": {
            "all": [],
            "ungrouped": []
        },
        "inventory_hostname": "localhost",
        "inventory_hostname_short": "localhost",
        "omit": "__omit_place_holder__956afa642a2566c50f2dee9d77116b326ec077ef",
        "playbook_dir": "/opt/meza/config"
    }
}

[freephile]

For docs on include_role, see https://docs.ansible.com/ansible/latest/collections/ansible/builtin/include_role_module.html

There was a bug in 2018 (Ansible 2.6.0) that when you used include_role: when used with_items in name would execute with the last item only. That bug was fixed in 2.6 and backported to 2.5 so certainly not at issue here.

And there was a bug related to delegation that was pretty similar to what we're seeing. That bug is solved and not exactly pertinent to our case.

[freephile]

Ansible Releases are intentionally confusing. Because RedHat sells premium subscriptions to Enterprise customers the "community ansible" versus "Ansible Automation Platform" are versioned separately, named separately, behave differently etc. https://docs.ansible.com/ansible/latest/reference_appendices/release_and_maintenance.html Buyer Beware

In any event, 'with_items' has changed since 2.5 and although it is not deprecated, the 'loop' construct is favored. https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html https://docs.ansible.com/ansible/latest/playbook_guide/playbooks_loops.html#with-items https://docs.ansible.com/ansible/latest/collections/ansible/builtin/items_lookup.html

Contrary to the still widely used with_items:, a bare loop: does not apply an automatic flatten(level=1) on the passed arguments.

Importantly, it seems that Python and Ansible are upgraded in the Digital Ocean droplet instances. So, the discussion about upgrading Python is very relevant https://github.com/freephile/meza/discussions/8

[freephile]

I re-ran the getmeza script after having deleted most of what Meza installs.

The output clearly indicates that the Meza 'requirements' for Ansible 2.9 were overwritten by a newer Ansible Core

sudo bash /opt/meza/src/scripts/getmeza.sh
DigitalOcean Agent                                                                                                                                          83 kB/s | 3.3 kB     00:00    
DigitalOcean Droplet Agent                                                                                                                                  57 kB/s | 3.3 kB     00:00    
Package centos-release-ansible-29-1-2.el8.noarch is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Last metadata expiration check: 0:00:02 ago on Wed 27 Mar 2024 12:56:55 PM UTC.
Package python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
Last metadata expiration check: 0:00:05 ago on Wed 27 Mar 2024 12:56:55 PM UTC.
Package git-2.39.3-1.el8_8.x86_64 is already installed.
Error: 
 Problem: problem with installed package ansible-core-2.15.3-1.el8.x86_64
  - package ansible-core-2.15.3-1.el8.x86_64 from @System conflicts with ansible < 2.10.0 provided by ansible-2.9.27-1.el8.noarch from centos-ansible-29
  - package ansible-core-2.15.3-1.el8.x86_64 from appstream conflicts with ansible < 2.10.0 provided by ansible-2.9.27-1.el8.noarch from centos-ansible-29
  - conflicting requests
(try to add '--allowerasing' to command line to replace conflicting packages or '--skip-broken' to skip uninstallable packages or '--nobest' to use not only best candidate packages)
Last metadata expiration check: 0:00:09 ago on Wed 27 Mar 2024 12:56:55 PM UTC.
Package python3-libselinux-2.9-8.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!
meza-ansible already exists
meza-ansible home-dir in correct location
meza command installed. Use it:
  sudo meza deploy monolith

[freephile]

I blew away that droplet and created a new one.

Here's the abbreviated setup for a new 8.9 Rocky Linux droplet

dnf install -y epel-release

sestatus
sed -i -e 's;SELINUX=enforcing$;SELINUX=permissive;g' /etc/selinux/config
sestatus

sed -i -e 's;countme=1$;countme=1\nexclude=ansible python38;g' /etc/yum.repos.d/Rocky-AppStream.repo

# update Rocky Linux packages
dnf -y upgrade

dnf install -y git

git config --global user.name "Greg Rundlett"
git config --global user.email greg.rundlett@gmail.com

# use a non-standard repository and branch
export MEZA_REPOSITORY_URL='https://github.com/freephile/meza.git'
export MEZA_BRANCH_NAME='REL1_29'

git clone $MEZA_REPOSITORY_URL /opt/meza

## chmod not needed. Already mode 0755
chmod a+rx /opt/meza

bash /opt/meza/src/scripts/getmeza.sh

# update the system again
## This upgrades Ansible and Python because they are 'excluded' but NOT versionlocked
dnf -y upgrade

cd /opt/meza/config

ansible --version
ansible-community --version
python --version

meza deploy monolith -vvv

dnf -y upgrade

Upgraded:
  NetworkManager-1:1.40.16-13.el8_9.x86_64                NetworkManager-libnm-1:1.40.16-13.el8_9.x86_64                          NetworkManager-team-1:1.40.16-13.el8_9.x86_64           
  NetworkManager-tui-1:1.40.16-13.el8_9.x86_64            PackageKit-1.1.12-7.el8.x86_64                                          PackageKit-glib-1.1.12-7.el8.x86_64                     
  audit-3.0.7-5.el8.x86_64                                audit-libs-3.0.7-5.el8.x86_64                                           authselect-1.2.6-2.el8.x86_64                           
  authselect-compat-1.2.6-2.el8.x86_64                    authselect-libs-1.2.6-2.el8.x86_64                                      bind-export-libs-32:9.11.36-11.el8_9.x86_64             
  c-ares-1.13.0-9.el8_9.1.x86_64                          chkconfig-1.19.2-1.el8.x86_64                                           cloud-init-23.1.1-11.el8_9.1.0.2.noarch                 
  cockpit-bridge-300.1-1.el8_9.x86_64                     cockpit-system-300.1-1.el8_9.noarch                                     cockpit-ws-300.1-1.el8_9.x86_64                         
  crypto-policies-20230731-1.git3177e06.el8.noarch        crypto-policies-scripts-20230731-1.git3177e06.el8.noarch                cryptsetup-libs-2.3.7-7.el8.x86_64                      
  curl-7.61.1-33.el8.x86_64                               dbus-1:1.12.8-26.el8.x86_64                                             dbus-common-1:1.12.8-26.el8.noarch                      
  dbus-daemon-1:1.12.8-26.el8.x86_64                      dbus-libs-1:1.12.8-26.el8.x86_64                                        dbus-tools-1:1.12.8-26.el8.x86_64                       
  device-mapper-8:1.02.181-13.el8_9.x86_64                device-mapper-libs-8:1.02.181-13.el8_9.x86_64                           dmidecode-1:3.3-5.el8.x86_64                            
  dnf-4.7.0-19.el8.noarch                                 dnf-data-4.7.0-19.el8.noarch                                            dnf-plugins-core-4.0.21-23.el8.noarch                   
  dracut-049-228.git20230802.el8.x86_64                   dracut-config-generic-049-228.git20230802.el8.x86_64                    dracut-config-rescue-049-228.git20230802.el8.x86_64     
  dracut-network-049-228.git20230802.el8.x86_64           dracut-squash-049-228.git20230802.el8.x86_64                            elfutils-debuginfod-client-0.189-3.el8.x86_64           
  elfutils-default-yama-scope-0.189-3.el8.noarch          elfutils-libelf-0.189-3.el8.x86_64                                      elfutils-libs-0.189-3.el8.x86_64                        
  epel-release-8-19.el8.noarch                            file-5.33-25.el8.x86_64                                                 file-libs-5.33-25.el8.x86_64                            
  findutils-1:4.6.0-21.el8.x86_64                         fuse-libs-2.9.7-17.el8.x86_64                                           fwupd-1.7.8-2.el8.rocky.0.2.x86_64                      
  glibc-2.28-236.el8_9.12.x86_64                          glibc-all-langpacks-2.28-236.el8_9.12.x86_64                            glibc-common-2.28-236.el8_9.12.x86_64                   
  glibc-gconv-extra-2.28-236.el8_9.12.x86_64              gnutls-3.6.16-8.el8_9.1.x86_64                                          grub2-common-1:2.02-150.el8.rocky.0.2.noarch            
  grub2-efi-x64-1:2.02-150.el8.rocky.0.2.x86_64           grub2-pc-1:2.02-150.el8.rocky.0.2.x86_64                                grub2-pc-modules-1:2.02-150.el8.rocky.0.2.noarch        
  grub2-tools-1:2.02-150.el8.rocky.0.2.x86_64             grub2-tools-extra-1:2.02-150.el8.rocky.0.2.x86_64                       grub2-tools-minimal-1:2.02-150.el8.rocky.0.2.x86_64     
  grubby-8.40-48.el8.x86_64                               hwdata-0.314-8.19.el8.noarch                                            iproute-6.2.0-5.el8_9.x86_64                            
  iputils-20180629-11.el8.x86_64                          irqbalance-2:1.9.2-1.el8.x86_64                                         kbd-2.0.4-11.el8.x86_64                                 
  kbd-legacy-2.0.4-11.el8.noarch                          kbd-misc-2.0.4-11.el8.noarch                                            kernel-tools-4.18.0-513.18.1.el8_9.x86_64               
  kernel-tools-libs-4.18.0-513.18.1.el8_9.x86_64          kexec-tools-2.0.26-8.el8.x86_64                                         kpartx-0.8.4-39.el8.x86_64                              
  krb5-libs-1.18.2-26.el8.x86_64                          libX11-1.6.8-6.el8.x86_64                                               libX11-common-1.6.8-6.el8.noarch                        
  libblkid-2.32.1-43.el8.x86_64                           libblockdev-2.28-4.el8.x86_64                                           libblockdev-crypto-2.28-4.el8.x86_64                    
  libblockdev-fs-2.28-4.el8.x86_64                        libblockdev-loop-2.28-4.el8.x86_64                                      libblockdev-mdraid-2.28-4.el8.x86_64                    
  libblockdev-part-2.28-4.el8.x86_64                      libblockdev-swap-2.28-4.el8.x86_64                                      libblockdev-utils-2.28-4.el8.x86_64                     
  libcap-2.48-6.el8_9.x86_64                              libcurl-7.61.1-33.el8.x86_64                                            libdnf-0.63.0-17.el8_9.x86_64                           
  libfastjson-0.99.9-2.el8.x86_64                         libfdisk-2.32.1-43.el8.x86_64                                           libgcc-8.5.0-20.el8.x86_64                              
  libgomp-8.5.0-20.el8.x86_64                             libldb-2.7.2-3.el8.x86_64                                               libmaxminddb-1.2.0-10.el8_9.1.x86_64                    
  libmount-2.32.1-43.el8.x86_64                           libnghttp2-1.33.0-5.el8_8.x86_64                                        libsmartcols-2.32.1-43.el8.x86_64                       
  libsolv-0.7.20-6.el8.x86_64                             libsoup-2.62.3-4.el8.x86_64                                             libssh-0.9.6-13.el8_9.x86_64                            
  libssh-config-0.9.6-13.el8_9.noarch                     libsss_autofs-2.9.1-4.el8_9.5.x86_64                                    libsss_certmap-2.9.1-4.el8_9.5.x86_64                   
  libsss_idmap-2.9.1-4.el8_9.5.x86_64                     libsss_nss_idmap-2.9.1-4.el8_9.5.x86_64                                 libsss_sudo-2.9.1-4.el8_9.5.x86_64                      
  libstdc++-8.5.0-20.el8.x86_64                           libtalloc-2.4.0-3.el8.x86_64                                            libtdb-1.4.8-3.el8.x86_64                               
  libtevent-0.14.1-3.el8.x86_64                           libudisks2-2.9.0-16.el8.x86_64                                          libuuid-2.32.1-43.el8.x86_64                            
  libxml2-2.9.7-18.el8_9.x86_64                           mdadm-4.2-8.el8.x86_64                                                  memstrack-0.2.5-2.el8.x86_64                            
  microcode_ctl-4:20230808-2.20231009.1.el8_9.x86_64      ncurses-6.1-10.20180224.el8.x86_64                                      ncurses-base-6.1-10.20180224.el8.noarch                 
  ncurses-libs-6.1-10.20180224.el8.x86_64                 nspr-4.35.0-1.el8_8.x86_64                                              nss-3.90.0-6.el8_9.x86_64                               
  nss-softokn-3.90.0-6.el8_9.x86_64                       nss-softokn-freebl-3.90.0-6.el8_9.x86_64                                nss-sysinit-3.90.0-6.el8_9.x86_64                       
  nss-util-3.90.0-6.el8_9.x86_64                          numactl-libs-2.0.16-1.el8.x86_64                                        openssh-8.0p1-19.el8_9.2.x86_64                         
  openssh-clients-8.0p1-19.el8_9.2.x86_64                 openssh-server-8.0p1-19.el8_9.2.x86_64                                  openssl-1:1.1.1k-12.el8_9.x86_64                        
  openssl-libs-1:1.1.1k-12.el8_9.x86_64                   pam-1.3.1-27.el8.x86_64                                                 pixman-0.38.4-3.el8_9.x86_64                            
  platform-python-3.6.8-56.el8_9.3.rocky.0.x86_64         platform-python-pip-9.0.3-23.el8.rocky.0.noarch                         procps-ng-3.3.15-14.el8.x86_64                          
  python3-audit-3.0.7-5.el8.x86_64                        python3-cffi-1.11.5-6.el8.x86_64                                        python3-cryptography-3.2.1-7.el8_9.x86_64               
  python3-dnf-4.7.0-19.el8.noarch                         python3-dnf-plugins-core-4.0.21-23.el8.noarch                           python3-hawkey-0.63.0-17.el8_9.x86_64                   
  python3-libdnf-0.63.0-17.el8_9.x86_64                   python3-libs-3.6.8-56.el8_9.3.rocky.0.x86_64                            python3-libxml2-2.9.7-18.el8_9.x86_64                   
  python3-magic-5.33-25.el8.noarch                        python3-perf-4.18.0-513.18.1.el8_9.x86_64                               python3-pip-wheel-9.0.3-23.el8.rocky.0.noarch           
  python3-pytz-2017.2-11.el8.noarch                       python3-requests-2.20.0-4.el8.noarch                                    python3-rpm-4.14.3-28.el8_9.x86_64                      
  python3-setools-4.3.0-5.el8.x86_64                      python3-syspurpose-1.28.40-1.el8_9.rocky.0.2.x86_64                     python3-unbound-1.16.2-5.el8_9.2.x86_64                 
  python3-urllib3-1.24.2-5.el8_9.2.noarch                 qemu-guest-agent-15:6.2.0-40.module+el8.9.0+1654+f4df84c4.2.x86_64      rng-tools-6.16-1.el8.x86_64                             
  rocky-gpg-keys-8.9-1.8.el8.noarch                       rocky-release-8.9-1.8.el8.noarch                                        rocky-repos-8.9-1.8.el8.noarch                          
  rpm-4.14.3-28.el8_9.x86_64                              rpm-build-libs-4.14.3-28.el8_9.x86_64                                   rpm-libs-4.14.3-28.el8_9.x86_64                         
  rpm-plugin-selinux-4.14.3-28.el8_9.x86_64               rpm-plugin-systemd-inhibit-4.14.3-28.el8_9.x86_64                       rsyslog-8.2102.0-15.el8.x86_64                          
  selinux-policy-3.14.3-128.el8_9.1.noarch                selinux-policy-targeted-3.14.3-128.el8_9.1.noarch                       shadow-utils-2:4.6-19.el8.x86_64                        
  sos-4.6.1-1.el8.noarch                                  sqlite-libs-3.26.0-19.el8_9.x86_64                                      sssd-client-2.9.1-4.el8_9.5.x86_64                      
  sssd-common-2.9.1-4.el8_9.5.x86_64                      sssd-kcm-2.9.1-4.el8_9.5.x86_64                                         sssd-nfs-idmap-2.9.1-4.el8_9.5.x86_64                   
  sudo-1.9.5p2-1.el8_9.x86_64                             systemd-239-78.el8.x86_64                                               systemd-libs-239-78.el8.x86_64                          
  systemd-pam-239-78.el8.x86_64                           systemd-udev-239-78.el8.x86_64                                          tpm2-tss-2.3.2-5.el8.x86_64                             
  tuned-2.21.0-1.el8_9.noarch                             tzdata-2024a-1.el8.noarch                                               udisks2-2.9.0-16.el8.x86_64                             
  unbound-libs-1.16.2-5.el8_9.2.x86_64                    util-linux-2.32.1-43.el8.x86_64                                         virt-what-1.25-4.el8.x86_64                             
  volume_key-libs-0.3.11-6.el8.x86_64                     which-2.21-20.el8.x86_64                                                xfsprogs-5.0.0-12.el8.x86_64                            
  yum-4.7.0-19.el8.noarch                                 yum-utils-4.0.21-23.el8.noarch                                          zlib-1.2.11-25.el8.x86_64                               
Installed:
  grub2-tools-efi-1:2.02-150.el8.rocky.0.2.x86_64        kernel-4.18.0-513.18.1.el8_9.x86_64      kernel-core-4.18.0-513.18.1.el8_9.x86_64   kernel-modules-4.18.0-513.18.1.el8_9.x86_64  
  linux-firmware-20230824-120.git0e048b06.el8_9.noarch   python3-setuptools-39.2.0-7.el8.noarch  

dnf -y install git

git-2.39.3-1.el8_8.x86_64 is already installed.

getmeza.sh

[root@rockylinux-s-2vcpu-4gb-nyc3-01 opt]# bash /opt/meza/src/scripts/getmeza.sh
Last metadata expiration check: 0:03:15 ago on Wed 27 Mar 2024 02:20:59 PM UTC.
Dependencies resolved.
============================================================================================================================================================================================
 Package                                                        Architecture                          Version                                   Repository                             Size
============================================================================================================================================================================================
Installing:
 centos-release-ansible-29                                      noarch                                1-2.el8                                   extras                                7.4 k
Installing dependencies:
 centos-release-configmanagement                                noarch                                1-1.el8                                   extras                                7.6 k

Transaction Summary
============================================================================================================================================================================================
Install  2 Packages

Total download size: 15 k
Installed size: 2.2 k
Downloading Packages:
(1/2): centos-release-ansible-29-1-2.el8.noarch.rpm                                                                                                         754 kB/s | 7.4 kB     00:00    
(2/2): centos-release-configmanagement-1-1.el8.noarch.rpm                                                                                                   718 kB/s | 7.6 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                       202 kB/s |  15 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                    1/1 
  Installing       : centos-release-configmanagement-1-1.el8.noarch                                                                                                                     1/2 
  Installing       : centos-release-ansible-29-1-2.el8.noarch                                                                                                                           2/2 
  Verifying        : centos-release-ansible-29-1-2.el8.noarch                                                                                                                           1/2 
  Verifying        : centos-release-configmanagement-1-1.el8.noarch                                                                                                                     2/2 

Installed:
  centos-release-ansible-29-1-2.el8.noarch                                                  centos-release-configmanagement-1-1.el8.noarch                                                 

Complete!
CentOS Configmanagement SIG - ansible-29                                                                                                                    2.8 MB/s | 2.1 MB     00:00    
Last metadata expiration check: 0:00:01 ago on Wed 27 Mar 2024 02:24:17 PM UTC.
Dependencies resolved.
============================================================================================================================================================================================
 Package                                 Architecture                       Version                                                             Repository                             Size
============================================================================================================================================================================================
Installing:
 python36                                x86_64                             3.6.8-38.module+el8.5.0+671+195e4563                                appstream                              18 k
Installing dependencies:
 python3-pip                             noarch                             9.0.3-23.el8.rocky.0                                                appstream                              19 k
Enabling module streams:
 python36                                                                   3.6                                                                                                            

Transaction Summary
============================================================================================================================================================================================
Install  2 Packages

Total download size: 38 k
Installed size: 16 k
Downloading Packages:
(1/2): python3-pip-9.0.3-23.el8.rocky.0.noarch.rpm                                                                                                          270 kB/s |  19 kB     00:00    
(2/2): python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64.rpm                                                                                             243 kB/s |  18 kB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                       288 kB/s |  38 kB     00:00     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                    1/1 
  Installing       : python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64                                                                                                               1/2 
  Running scriptlet: python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64                                                                                                               1/2 
  Installing       : python3-pip-9.0.3-23.el8.rocky.0.noarch                                                                                                                            2/2 
  Running scriptlet: python3-pip-9.0.3-23.el8.rocky.0.noarch                                                                                                                            2/2 
  Verifying        : python3-pip-9.0.3-23.el8.rocky.0.noarch                                                                                                                            1/2 
  Verifying        : python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64                                                                                                               2/2 

Installed:
  python3-pip-9.0.3-23.el8.rocky.0.noarch                                                python36-3.6.8-38.module+el8.5.0+671+195e4563.x86_64                                               

Complete!
Last metadata expiration check: 0:00:04 ago on Wed 27 Mar 2024 02:24:17 PM UTC.
Package git-2.39.3-1.el8_8.x86_64 is already installed.
Dependencies resolved.
============================================================================================================================================================================================
 Package                                         Architecture                          Version                                       Repository                                        Size
============================================================================================================================================================================================
Installing:
 ansible                                         noarch                                2.9.27-1.el8                                  centos-ansible-29                                 17 M
Installing dependencies:
 sshpass                                         x86_64                                1.09-4.el8                                    appstream                                         29 k
Installing weak dependencies:
 python3-jmespath                                noarch                                0.9.0-11.el8                                  appstream                                         44 k

Transaction Summary
============================================================================================================================================================================================
Install  3 Packages

Total download size: 17 M
Installed size: 96 M
Downloading Packages:
(1/3): sshpass-1.09-4.el8.x86_64.rpm                                                                                                                        561 kB/s |  29 kB     00:00    
(2/3): python3-jmespath-0.9.0-11.el8.noarch.rpm                                                                                                             804 kB/s |  44 kB     00:00    
(3/3): ansible-2.9.27-1.el8.noarch.rpm                                                                                                                       32 MB/s |  17 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        26 MB/s |  17 MB     00:00     
CentOS Configmanagement SIG - ansible-29                                                                                                                    1.0 MB/s | 1.0 kB     00:00    
Importing GPG key 0x6E8B7E8A:
 Userid     : "CentOS Config Management SIG (https://wiki.centos.org/SpecialInterestGroup/ConfigManagementSIG) <security@centos.org>"
 Fingerprint: C75A FB57 D5C0 F238 CB15 BEC8 1AE1 10FA 6E8B 7E8A
 From       : /etc/pki/rpm-gpg/RPM-GPG-KEY-CentOS-SIG-ConfigManagement
Key imported successfully
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                    1/1 
  Installing       : sshpass-1.09-4.el8.x86_64                                                                                                                                          1/3 
  Installing       : python3-jmespath-0.9.0-11.el8.noarch                                                                                                                               2/3 
  Installing       : ansible-2.9.27-1.el8.noarch                                                                                                                                        3/3 
  Running scriptlet: ansible-2.9.27-1.el8.noarch                                                                                                                                        3/3 
  Verifying        : ansible-2.9.27-1.el8.noarch                                                                                                                                        1/3 
  Verifying        : python3-jmespath-0.9.0-11.el8.noarch                                                                                                                               2/3 
  Verifying        : sshpass-1.09-4.el8.x86_64                                                                                                                                          3/3 

Installed:
  ansible-2.9.27-1.el8.noarch                                 python3-jmespath-0.9.0-11.el8.noarch                                 sshpass-1.09-4.el8.x86_64                                

Complete!
Last metadata expiration check: 0:00:28 ago on Wed 27 Mar 2024 02:24:17 PM UTC.
Package python3-libselinux-2.9-8.el8.x86_64 is already installed.
Dependencies resolved.
Nothing to do.
Complete!

Add ansible master user
Ansible user: meza-ansible
Generating public/private rsa key pair.
Your identification has been saved in /opt/conf-meza/users/meza-ansible/.ssh/id_rsa.
Your public key has been saved in /opt/conf-meza/users/meza-ansible/.ssh/id_rsa.pub.
The key fingerprint is:
SHA256:WXkOiNvAh/T5gnlCudkQLjUWvgl3mwa+BzZ3Gehh4iQ meza-ansible@rockylinux-s-2vcpu-4gb-nyc3-01
The key's randomart image is:
+---[RSA 3072]----+
|      B.         |
|     B B + .     |
|    E / X = .    |
|     X ^ B *     |
|      / S + .    |
|     . O o       |
|      . .        |
|       .         |
|                 |
+----[SHA256]-----+
Adding the following to /etc/sudoers.d/meza-ansible:
meza-ansible ALL=(ALL) NOPASSWD: ALL
meza command installed. Use it:
  sudo meza deploy monolith

After 'getmeza.sh' The ansible and python versions are 'correct'

ansible --version ; \ ansible-community --version ; \ python --version ; ansible 2.9.27 config file = /etc/ansible/ansible.cfg configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules'] ansible python module location = /usr/lib/python3.6/site-packages/ansible executable location = /usr/bin/ansible python version = 3.6.8 (default, Jan 15 2024, 23:09:02) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] bash: ansible-community: command not found Python 3.6.8

However, if you 'dnf -y upgrade' your system, you will upgrade Ansible and Python. Ansible will become 2.15.3 System Python will still be 3.6.8 but Ansible Python will be 3.11.5

dnf -y upgrade
DigitalOcean Agent                                                                                                                                           74 kB/s | 3.3 kB     00:00    
DigitalOcean Droplet Agent                                                                                                                                   92 kB/s | 3.3 kB     00:00    
Dependencies resolved.
============================================================================================================================================================================================
 Package                                                   Architecture                         Version                                       Repository                               Size
============================================================================================================================================================================================
Upgrading:
 ansible                                                   noarch                               8.3.0-1.el8                                   epel                                     41 M
Installing dependencies:
 ansible-core                                              x86_64                               2.15.3-1.el8                                  appstream                               3.6 M
 mpdecimal                                                 x86_64                               2.5.1-3.el8                                   appstream                                92 k
 python3.11                                                x86_64                               3.11.5-1.el8_9                                appstream                                29 k
 python3.11-cffi                                           x86_64                               1.15.1-1.el8                                  appstream                               292 k
 python3.11-cryptography                                   x86_64                               37.0.2-5.el8                                  appstream                               1.1 M
 python3.11-libs                                           x86_64                               3.11.5-1.el8_9                                appstream                                10 M
 python3.11-pip-wheel                                      noarch                               22.3.1-4.el8                                  appstream                               1.4 M
 python3.11-ply                                            noarch                               3.11-1.el8                                    appstream                               134 k
 python3.11-pycparser                                      noarch                               2.20-1.el8                                    appstream                               146 k
 python3.11-pyyaml                                         x86_64                               6.0-1.el8                                     appstream                               213 k
 python3.11-setuptools-wheel                               noarch                               65.5.1-2.el8                                  appstream                               719 k

Transaction Summary
============================================================================================================================================================================================
Install  11 Packages
Upgrade   1 Package

Total download size: 59 M
Downloading Packages:
(1/12): python3.11-3.11.5-1.el8_9.x86_64.rpm                                                                                                                 96 kB/s |  29 kB     00:00    
(2/12): mpdecimal-2.5.1-3.el8.x86_64.rpm                                                                                                                    277 kB/s |  92 kB     00:00    
(3/12): python3.11-cffi-1.15.1-1.el8.x86_64.rpm                                                                                                             2.0 MB/s | 292 kB     00:00    
(4/12): ansible-core-2.15.3-1.el8.x86_64.rpm                                                                                                                6.5 MB/s | 3.6 MB     00:00    
(5/12): python3.11-cryptography-37.0.2-5.el8.x86_64.rpm                                                                                                     4.6 MB/s | 1.1 MB     00:00    
(6/12): python3.11-pip-wheel-22.3.1-4.el8.noarch.rpm                                                                                                         13 MB/s | 1.4 MB     00:00    
(7/12): python3.11-ply-3.11-1.el8.noarch.rpm                                                                                                                1.3 MB/s | 134 kB     00:00    
(8/12): python3.11-pyyaml-6.0-1.el8.x86_64.rpm                                                                                                              2.0 MB/s | 213 kB     00:00    
(9/12): python3.11-libs-3.11.5-1.el8_9.x86_64.rpm                                                                                                            23 MB/s |  10 MB     00:00    
(10/12): python3.11-pycparser-2.20-1.el8.noarch.rpm                                                                                                         623 kB/s | 146 kB     00:00    
(11/12): python3.11-setuptools-wheel-65.5.1-2.el8.noarch.rpm                                                                                                3.7 MB/s | 719 kB     00:00    
(12/12): ansible-8.3.0-1.el8.noarch.rpm                                                                                                                      52 MB/s |  41 MB     00:00    
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Total                                                                                                                                                        31 MB/s |  59 MB     00:01     
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
  Preparing        :                                                                                                                                                                    1/1 
  Installing       : python3.11-setuptools-wheel-65.5.1-2.el8.noarch                                                                                                                   1/13 
  Installing       : python3.11-pip-wheel-22.3.1-4.el8.noarch                                                                                                                          2/13 
  Installing       : mpdecimal-2.5.1-3.el8.x86_64                                                                                                                                      3/13 
  Installing       : python3.11-3.11.5-1.el8_9.x86_64                                                                                                                                  4/13 
  Running scriptlet: python3.11-3.11.5-1.el8_9.x86_64                                                                                                                                  4/13 
  Installing       : python3.11-libs-3.11.5-1.el8_9.x86_64                                                                                                                             5/13 
  Installing       : python3.11-ply-3.11-1.el8.noarch                                                                                                                                  6/13 
  Installing       : python3.11-pycparser-2.20-1.el8.noarch                                                                                                                            7/13 
  Installing       : python3.11-cffi-1.15.1-1.el8.x86_64                                                                                                                               8/13 
  Installing       : python3.11-cryptography-37.0.2-5.el8.x86_64                                                                                                                       9/13 
  Installing       : python3.11-pyyaml-6.0-1.el8.x86_64                                                                                                                               10/13 
  Installing       : ansible-core-2.15.3-1.el8.x86_64                                                                                                                                 11/13 
  Upgrading        : ansible-8.3.0-1.el8.noarch                                                                                                                                       12/13 
  Cleanup          : ansible-2.9.27-1.el8.noarch                                                                                                                                      13/13 
  Running scriptlet: ansible-2.9.27-1.el8.noarch                                                                                                                                      13/13 
  Verifying        : ansible-core-2.15.3-1.el8.x86_64                                                                                                                                  1/13 
  Verifying        : mpdecimal-2.5.1-3.el8.x86_64                                                                                                                                      2/13 
  Verifying        : python3.11-3.11.5-1.el8_9.x86_64                                                                                                                                  3/13 
  Verifying        : python3.11-cffi-1.15.1-1.el8.x86_64                                                                                                                               4/13 
  Verifying        : python3.11-cryptography-37.0.2-5.el8.x86_64                                                                                                                       5/13 
  Verifying        : python3.11-libs-3.11.5-1.el8_9.x86_64                                                                                                                             6/13 
  Verifying        : python3.11-pip-wheel-22.3.1-4.el8.noarch                                                                                                                          7/13 
  Verifying        : python3.11-ply-3.11-1.el8.noarch                                                                                                                                  8/13 
  Verifying        : python3.11-pycparser-2.20-1.el8.noarch                                                                                                                            9/13 
  Verifying        : python3.11-pyyaml-6.0-1.el8.x86_64                                                                                                                               10/13 
  Verifying        : python3.11-setuptools-wheel-65.5.1-2.el8.noarch                                                                                                                  11/13 
  Verifying        : ansible-8.3.0-1.el8.noarch                                                                                                                                       12/13 
  Verifying        : ansible-2.9.27-1.el8.noarch                                                                                                                                      13/13 

Upgraded:
  ansible-8.3.0-1.el8.noarch                                                                                                                                                                
Installed:
  ansible-core-2.15.3-1.el8.x86_64                 mpdecimal-2.5.1-3.el8.x86_64               python3.11-3.11.5-1.el8_9.x86_64                     python3.11-cffi-1.15.1-1.el8.x86_64     
  python3.11-cryptography-37.0.2-5.el8.x86_64      python3.11-libs-3.11.5-1.el8_9.x86_64      python3.11-pip-wheel-22.3.1-4.el8.noarch             python3.11-ply-3.11-1.el8.noarch        
  python3.11-pycparser-2.20-1.el8.noarch           python3.11-pyyaml-6.0-1.el8.x86_64         python3.11-setuptools-wheel-65.5.1-2.el8.noarch     

Complete!
[root@rockylinux-s-2vcpu-4gb-nyc3-01 opt]# ansible --version ; ansible-community --version ; python --version ;
ansible [core 2.15.3]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.5 (main, Oct 25 2023, 16:19:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
Ansible community version 8.3.0
Python 3.6.8

And if we move to the 'config' directory of meza, we'll pickup the ansible.cfg file there:

ansible --version ; \
> ansible-community --version ; \
> python --version ;
ansible [core 2.15.3]
  config file = /opt/meza/config/ansible.cfg
  configured module search path = ['/root/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3.11/site-packages/ansible
  ansible collection location = /root/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.11.5 (main, Oct 25 2023, 16:19:59) [GCC 8.5.0 20210514 (Red Hat 8.5.0-20)] (/usr/bin/python3.11)
  jinja version = 3.1.2
  libyaml = True
Ansible community version 8.3.0
Python 3.6.8

deploy breaks on smw-rebuild-all.sh

In this environment, the traditional meza deploy monolith -vvv breaks on SMW-Rebuild-all

TASK [mediawiki : (Re-)build SemanticMediaWiki data for: demo] *****************
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:582
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: meza-ansible
<localhost> EXEC /bin/sh -c '( umask 77 && mkdir -p "` echo /tmp/${USER}/ansible `"&& mkdir "` echo /tmp/${USER}/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672 `" && echo ansible-tmp-1711551166.4157615-113841-270060465983672="` echo /tmp/${USER}/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672 `" ) && sleep 0'
Using module file /usr/lib/python3.11/site-packages/ansible/modules/command.py
<localhost> PUT /opt/conf-meza/users/meza-ansible/.ansible/tmp/ansible-local-76444qt02fwc6/tmpyxfk3q8b TO /tmp/meza-ansible/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672/AnsiballZ_command.py
<localhost> EXEC /bin/sh -c 'chmod u+x /tmp/meza-ansible/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672/ /tmp/meza-ansible/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672/AnsiballZ_command.py && sleep 0'
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-dvbwwsfxnnpsjujfawqrqvmelloxstir ; /usr/libexec/platform-python /tmp/meza-ansible/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672/AnsiballZ_command.py'"'"' && sleep 0'
<localhost> EXEC /bin/sh -c 'rm -f -r /tmp/meza-ansible/ansible/ansible-tmp-1711551166.4157615-113841-270060465983672/ > /dev/null 2>&1 && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "'bash /opt/.deploy-meza/smw-rebuild-all.sh \"demo\" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log'\n",
    "delta": "0:00:00.006433",
    "end": "2024-03-27 14:52:46.752216",
    "invocation": {
        "module_args": {
            "_raw_params": "'bash /opt/.deploy-meza/smw-rebuild-all.sh \"demo\" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log'\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true
        }
    },
    "rc": 127,
    "start": "2024-03-27 14:52:46.745783"
}

STDERR:

/bin/sh: bash /opt/.deploy-meza/smw-rebuild-all.sh "demo" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log: No such file or directory

MSG:

non-zero return code

[freephile]

It breaks because the logfile doesn't exist

The reason the SMW task fails is actually because the logfile doesn't exist yet. So, changing the play to create the log file, and ONLY run when it creates the log file accomplishes two goals:

• unbreak initial deploys
• separate true SMW rebuild data into a different playbook (meza/src/playbooks/rebuild-smw-and-index.yml already exists and does exactly that.)

Since there isn't any data to rebuild for a newly born app server with a demo wiki, we could/should re-factor this more. The right thing to do is almost certainly to delete this task from the MediaWiki role.

[freephile]

How to rebuild SMW data and Elasticsearch indexes

The meza command meza maint-rebuild monolith will run the meza/src/playbooks/rebuild-smw-and-index.yml playbook.

This command needs to be added to https://www.mediawiki.org/wiki/Meza/Commands

[freephile]

With major upgrades to Ansible and Python, you also get New Bugs

The good news is that we finally managed to get a simple meza deploy monolith to work without errors on an initial system build. The bad news is that it is quite easy to inadvertently upgrade Python and Ansible by leaps and bounds - introducing a whole new potential for errors and bugs. We immediately discovered that the Visual Editor would not work for comments. (Although uploads worked, edits worked, search worked.)

Error converting between wikitext and HTML for VisualEditor

The VE API seems fine https://159.65.253.2/demo/api.php?action=visualeditor&format=json&paction=parse&page=Main_Page

Anyway, we also saw that 'create wiki' wasn't fully deploying newly created wikis. And the system itself even was unstable with episodes of 100% memory usage forcing us to hard reboot the server twice.

Modern Meza

We would like very much to use this "Modern Meza", but in reality we need to carefully review all the changes in Python and Ansible over the past several years to ensure that Meza adopts, implements and conforms to the new features, standards and expectations of such an environment.

You can see that Ansible 2.9 was released on Oct. 31 2019 Since then, there have been 6 major releases: 2.10, 2.11, 2.12, 2.13, 2.14 and now 2.15 We need to go through the Porting Guides for 2.10 - Ansible 8 Keep in mind, that for each minor point release (e.g. 2.10) there are multiple step releases (e.g. 2.10.0, 2.10.1, 2.10.2, 2.10.3, 2.10.4, 2.10.5, 2.10.6, 2.10.7) indicated by the fact that the CHANGELOG for 2.10 is over 8,000 lines

[freephile]

Fixing this bug

With a slight 'pun intended' we need to 'fix' (as in not changeable) the versions of Ansible and Python packages so that they are not accidentally upgraded. In the Debian / Ubuntu world, this is called "package pinning".

In the past, Meza only 'excluded' the new packages which is like pretending they don't exist. We need to versionlock the packages in RockyLinux / RHEL so that they upgrade only when we explicitly intend to. (versionlock for DNF is the equivalent to pinning in apt.)

[freephile]

This issue is still happening, despite the documented behavior of 'creates' (it is supposed to create the non-existent file).

Here is the first-run deploy output I got after following the Meza/Install on existing server procedure (that I wrote)

TASK [mediawiki : (Re-)build search index for: demo] ***************************************************************************************************************************************
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:574
Using module file /usr/lib/python3.6/site-packages/ansible/modules/commands/command.py
Pipelining is enabled.
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: meza-ansible
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-pvzimopmxkiabnbscofnhrwoccmfhohm ; /usr/bin/python'"'"' && sleep 0'
changed: [localhost] => {
    "changed": true,
    "cmd": "bash /opt/.deploy-meza/elastic-rebuild-all.sh \"demo\"",
    "delta": "0:00:03.907182",
    "end": "2024-03-28 20:16:15.113647",
    "invocation": {
        "module_args": {
            "_raw_params": "bash /opt/.deploy-meza/elastic-rebuild-all.sh \"demo\"",
            "_uses_shell": true,
            "argv": null,
            "chdir": null,
            "creates": null,
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 0,
    "start": "2024-03-28 20:16:11.206465"
}

STDOUT:

Rebuilding index for demo
  Output log:
    /opt/data-meza/logs/search-index/demo.2024-03-28_20:16:11.log
elastic-build-index completed for "demo" at 2024-03-28_20:16:15

TASK [mediawiki : (Re-)build SemanticMediaWiki data for: demo] *****************************************************************************************************************************
task path: /opt/meza/src/roles/mediawiki/tasks/main.yml:584
Using module file /usr/lib/python3.6/site-packages/ansible/modules/commands/command.py
Pipelining is enabled.
<localhost> ESTABLISH LOCAL CONNECTION FOR USER: meza-ansible
<localhost> EXEC /bin/sh -c 'sudo -H -S -n  -u root /bin/sh -c '"'"'echo BECOME-SUCCESS-kjphqujcqssoncclcflpdhdlhpbxzwoa ; /usr/bin/python'"'"' && sleep 0'
fatal: [localhost]: FAILED! => {
    "changed": true,
    "cmd": "'bash /opt/.deploy-meza/smw-rebuild-all.sh \"demo\" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log'\n",
    "delta": "0:00:00.005758",
    "end": "2024-03-28 20:16:15.507938",
    "invocation": {
        "module_args": {
            "_raw_params": "'bash /opt/.deploy-meza/smw-rebuild-all.sh \"demo\" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log'\n",
            "_uses_shell": true,
            "argv": null,
            "chdir": "/opt/data-meza/logs/smw-rebuilddata/",
            "creates": "smw-rebuild-all.log",
            "executable": null,
            "removes": null,
            "stdin": null,
            "stdin_add_newline": true,
            "strip_empty_ends": true,
            "warn": true
        }
    },
    "rc": 127,
    "start": "2024-03-28 20:16:15.502180"
}

STDERR:

/bin/sh: bash /opt/.deploy-meza/smw-rebuild-all.sh "demo" > /opt/data-meza/logs/smw-rebuilddata/smw-rebuild-all.log: No such file or directory

MSG:

non-zero return code

[freephile]

I changed the shell command argument into a single line thinking I saw some issue with the 'folding' of the string (but there isn't any problem... the full command is right there in the error message).

I don't know yet whether that command formatting change affects anything because the task is completely skipped in a second deploy (no 'smw-rebuild-all.log' string in the deploy-output log).

Although the first deploy failed mid-way, the 'when' conditions apparently change causing the task to be skipped. The smw-rebuild-all.log file itself does NOT get created in the second deploy (or multiple deploys after that).

[freephile]

Although I have a fully reproducible process for deploying Meza and even adding SAML auth, there are a couple stubborn issues:

1. First-time deploy (hard to reproduce because you only get one crack at it) Current work-around is to --skip-tags smw-data OR, simply deploy again.
2. Create wiki doesn't fully build the wiki. You must also run a deploy after the create-wiki in order to fully build the wiki. "Not fully built" means the Blender page lists the wiki, but traversing the link results in a 404. IF you navigate manually to the wiki, it will NOT be styled with MediaWiki CSS. The playbook (and conditions) must be logged and traced to review why it was skipped.
   1. An ancillary problem to this is that "create wiki" does NOT log output so it's difficult to even review what Meza did in the playbook. The current workaround for no logs is to manually use "tee": 'create wiki-promptless | tee /tmp/create.log'
3. A third issue that is truly out of scope for this project is that the Blender Landing page gets broken if you add SAML auth due to a custom Meza feature called "Non MediaWiki Auth" that is baked into the SAML Auth setup. This feature needs to be separated from the SAML auth feature and possibly removed entirely (or updated to leverage the SAML framework used BY MediaWiki).