Upgrading to 1.39.8 Ironbank

fishpeopleapps commented 1 month ago

Describe the situation

Summary: Multiple issues when attempting to upgrade to newer canasta version using Ironbank's hardened image resulting in broken staging environment, incorrect permission, etc.

Description: Hello! I am the developer on the Space-Wiki. We recently attempted to update to PHP 8.1 and the new hardened image (https://ironbank.dso.mil/repomap/details;registry1Path=opensource%252Fcanastawiki%252Fcanasta) tag 1.39.7. We started this process back in April and continue to have problems so any insight, at all, would be greatly appreciated. Initially our hosting company said we had to use a custom image, due to this error: https://hub.docker.com/_/php. Our "custom image" consists of the Ironbank canasta's image and manually installed required packages.

The custom image seems to "work" but these are some of the errors that appear in ArgoCD's logs:

Screenshots: rsync: [generator] failed to set times on "/mediawiki/images": Operation not permitted (1) rsync error: some files/attrs were not transferred (see previous errors) (code 23) at main.c(1338) [sender=3.2.7] rm -rf /mw_origin_files rm: cannot remove /mw_origin_files: Permission denied

++ find /maintenance-scripts/ -maxdepth 1 -mindepth 1 -type f -name '*.sh' stat: cannot statx '/mediawiki/sitemap': No such file or directory

AH00558: apache2: Could not reliably determine the server's fully qualified domain name, using 10.42.90.109. Set the 'ServerName' directive globally to suppress this message [15-Jul-2024 21:44:33] ERROR: failed to open error_log (/var/log/php8.1-fpm.log): Permission denied (13) [15-Jul-2024 21:44:33] ERROR: failed to post process the configuration [15-Jul-2024 21:44:33] ERROR: FPM initialization failed

Warning: wikis.yaml does not exist. Running general jobs.

Here's the full log: argoCDLogs.txt

Final Thoughts -- By posting here I am hoping for a friendly wiki person to please point me in the correct direction of anything I might find useful for understanding what's going on so I can fix the problem. I noticed a lot of changes between run-all and run-apache. Is there a reason we cannot use run-apache or this file was changed?

Steps to reproduce the issue (if applicable): n/a

Expected behavior

According to Ironbank and our hosting company, nothing is wrong, so I imagine to them the expected behavior is that the upgrade went smoothly and everything functions normally.

System info

Please complete the following information:

MediaWiki version (e.g. 1.39.7) - 1.39.1 trying to upgrade to 1.39.7
Canasta version - whatever comes with that image
Host operating system - linux
Do you have sudo/root permissions on the host OS? no

Sanity checks

Only applies to troubleshooting requests.

Have you checked the documentation on canasta.wiki for how to address this? Yes
Have you checked prior issues on GitHub yet? Yes
Are you following all Canasta approaches and have avoided doing things such as running docker exec directly on the container, removing the Caddy/Varnish containers, adding unauthorized files to the Docker container after startup, etc.? IDK

Thank you for your help and expertise, it is greatly appreciated.

yaronkoren commented 1 month ago

Hello @fishpeopleapps - sorry for the delay in responding. And thank you for the detailed bug report. The fact that you don't have sudo/root permissions definitely seems like an issue - it may explain at least the permissions-related issues you are seeing. Can you get this access?

To answer your question about run-all: run-apache.sh was renamed to run-all.sh, because the original name was a misnomer: it does a lot more than run Apache. But the script basically does the same thing that it did before - there have been a bunch of small changes to the code, but as far as I remember nothing substantial.

fishpeopleapps commented 1 month ago

No apologies needed, thank you so much for your time @yaronkoren ! This issue has been ongoing since 22/April so I am incredibly grateful for any insight you can offer.

I inquired about sudo/root permissions and this was the response I received: Hosting Company: As I recall, the image has root permissions already for the apache config scripts, so I'd think the script can be run as root instead of the non-root user Ironbank: The base container from Iron Bank is setup to run as root (as apache changes to a non-root user on its own). The cluster also has to be configured (usually via security policy) to allow privileged containers (preferably on dedicated nodes with no other workloads, etc). Hosting Company: The container is currently allowed to run as root in the cluster.

I wanted to add a couple of things as well.

We have not been able to pull down our staging environment locally. We can pull the wiki image via docker desktop and look around, using the 'exec' tab for commands, but that is the extend. We can add commands to either the apache script* or as args in our deployment.yaml file.
*We previously pulled out the run-all.sh script and mounted it outside of Ironbank so we could run commands/scripts in the run_autoupdate(), but yesterday I thought maybe returning to a baseline and using the Ironbank image with as little deviation as possible might be best for troubleshooting. I have attached the updated error log after I made this change: baselineArgoLogs.txt

Here is the contents of the deployment.yaml file - `apiVersion: apps/v1 kind: Deployment metadata: name: space-wiki spec: template: spec: containers:

name: space-wiki command: ['/bin/sh'] args:
- '-c'
- | echo "Running apache script" /run-all.sh echo "run all complete" volumes:
name: config-script configMap: name: local-settings-staging`

-Kim

fishpeopleapps commented 1 month ago

Thoughts on this error in the logs? PHP Fatal error: Declaration of GetMediawikiSettings::finalSetup() must be compatible with Maintenance::finalSetup(?MediaWiki\Settings\SettingsBuilder $settingsBuilder = null) in /getMediawikiSettings.php on line 142

Brought it up to Ironbank and they aren't sure but are open to test/change things.

yaronkoren commented 1 month ago

@fishpeopleapps - sorry for the long delay. That was apparently a bug in Canasta, which was fixed in this commit. Maybe it works better now, with the latest code?

fishpeopleapps commented 3 weeks ago

@yaronkoren That did fix the php issue. I noticed Ironbank didn't include skins.yaml and extensions-skins.php in their hardened image. They were able to add extensions-skins.php, but not the skins.yaml because "While skins.yaml is in the github, the file does not exist in the upstream container. I ran "find / -name skins.yaml" inside the upstream container and it did not return any results. I also used grep to search for any references in files/scripts to that file and again, there were no hits."

Right now we have this error and I am wondering if it could be related to not having the skins.yaml file in the upstream container?

Fatal error: Uncaught Exception: Unable to open file /var/www/mediawiki/w/skins/chameleon/skin.json: filemtime(): stat failed for /var/www/mediawiki/w/skins/chameleon/skin.json in /var/www/mediawiki/w/includes/registration/ExtensionRegistry.php:199 Stack trace: #0 /var/www/mediawiki/w/includes/GlobalFunctions.php(86): ExtensionRegistry->queue() #1 /mediawiki/config/LocalSettings.php(476): wfLoadSkin() #2 /var/www/mediawiki/w/CanastaDefaultSettings.php(82): require_once('...') #3 /var/www/mediawiki/w/LocalSettings.php(8): require_once('...') #4 /var/www/mediawiki/w/includes/Setup.php(218): require_once('...') #5 /var/www/mediawiki/w/includes/WebStart.php(86): require_once('...') #6 /var/www/mediawiki/w/index.php(44): require('...') #7 {main} thrown in /var/www/mediawiki/w/includes/registration/ExtensionRegistry.php on line 199

yaronkoren commented 2 weeks ago

@fishpeopleapps - Sorry again for the delay. Are they using the latest version of Canasta? skins.yaml is right here.

fishpeopleapps commented 2 weeks ago

@yaronkoren we are using the hardened image from ironbank which is 1.39.8 and the source code can be found here: https://repo1.dso.mil/dsop/opensource/canastawiki/canasta

When I pointed out that file and how its missing their response was - "While skins.yaml is in the github, the file does not exist in the upstream container. I ran "find / -name skins.yaml" inside the upstream container and it did not return any results. I also used grep to search for any references in files/scripts to that file and again, there were no hits."

fishpeopleapps commented 2 weeks ago

In the hardening_manifest.yaml

apiVersion: v1

name: opensource/canastawiki/canasta

tags:
  - "1.39.8"
  - "latest"

args:
  BASE_IMAGE: "opensource/debian/debian"
  BASE_TAG: "12.6"

labels:
  org.opencontainers.image.title: "canastawiki/canasta"
  org.opencontainers.image.description: "Canasta is an all-in-one MediaWiki package for sysadmins that makes it easy to manage MediaWiki, add extensions, and load starter content and data structures."
  org.opencontainers.image.licenses: "MIT License"
  org.opencontainers.image.url: "https://github.com/CanastaWiki/Canasta"
  org.opencontainers.image.vendor: "Canasta"
  org.opencontainers.image.version: "1.39.8"
  mil.dso.ironbank.image.keywords: "opensource,canasta,canastawiki,debian,php,mediawiki"
  mil.dso.ironbank.image.type: "opensource"
  mil.dso.ironbank.product.name: "opensource/canastawiki/canasta"

resources:
- tag: ghcr.io/canastawiki/canasta:1.39.8-latest
  url: docker://ghcr.io/canastawiki/canasta@sha256:68e823d715231b35b1348f200d797aad45aa6ee7c488031ec45de7d721015cc1

fishpeopleapps commented 2 weeks ago

I asked for specifics from IB and they responded: "When I mean upstream, I mean that the official canasta wiki container does not contain that file. "

yaronkoren commented 2 weeks ago

@fishpeopleapps - okay, sorry for the confusion about skins.yaml. We believe that this was actually just an issue with the installing of the Chameleon skin (because it, alone among skins and extensions, starts with a lowercase letter), and that the recent patch #433 solves it. If you get the latest code again, hopefully everything will work now.

fishpeopleapps commented 2 weeks ago

@yaronkoren @jeffw16 Example - I commented out the chameleon skin and this is the new error. We are really having a problem with getting Canasta to work.

PHP Fatal error:  Uncaught Exception: Unable to open file /var/www/mediawiki/w/skins/pivot/skin.json: filemtime(): stat failed for /var/www/mediawiki/w/skins/pivot/skin.json in /var/www/mediawiki/w/includes/registration/ExtensionRegistry.php:199
Stack trace:
#0 /var/www/mediawiki/w/includes/GlobalFunctions.php(86): ExtensionRegistry->queue()
#1 /mediawiki/config/LocalSettings.php(481): wfLoadSkin()
#2 /var/www/mediawiki/w/CanastaDefaultSettings.php(82): require_once('...')
#3 /var/www/mediawiki/w/LocalSettings.php(8): require_once('...')
#4 /var/www/mediawiki/w/includes/Setup.php(218): require_once('...')
#5 /var/www/mediawiki/w/maintenance/doMaintenance.php(83): require_once('...')
#6 /getMediawikiSettings.php(164): require('...')
#7 {main}

yaronkoren commented 2 weeks ago

Thank you for your continued patience. For the problem with loading the Pivot skin - please make sure they are calling wfLoadSkin( 'Pivot' ) and not wfLoadSkin( 'pivot' ) - in other words, Chameleon is the only skin or extension that starts with a lowercase letter, as far as I know.

For the permission problems - I don't know what is causing that. Maybe the loading issue is leading to that as well?

yaronkoren commented 1 week ago

@fishpeopleapps - okay, we just checked in PR #434, which fixes some problems that may have caused this error. If you can, please get them to upgrade to this latest code, and try it again.

fishpeopleapps commented 1 week ago

@yaronkoren While this fixes the skin issues, our other issues persist. Our staging environment is still down. It has been challenging to receive support due to our unique environment.

I understand from your website that the Canasta Project does not guarantee the same functionality for Ironbank Canasta, nor does it provide official support. However it's not working at all, Ironbank does not provide development support with PHP/Wiki, and discussions have started about the USSF moving away from Canasta.

Could you please share your insights on this matter and the extent of support that might be available for the hardened image from Ironbank? This information would be valuable as I provide updates to leadership. They would like to better understand the scope of assistance that is available to us.

Thank you.

yaronkoren commented 1 week ago

I understand your (and others') frustration, and I certainly hope that you don't move away from Canasta. (And I, too, would like to see Ironbank Canasta fully working.) As far as support: I'm happy to continue providing suggestions (and bug fixes) here. That may or may not be enough to solve the problems you are seeing. If you/they prefer to get more hands-on support, there are any number of consultants, an consulting companies - including my own, WikiWorks - who could be hired to do direct troubleshooting.

fishpeopleapps commented 1 week ago

Thank you, I will pass along WikiWorks as a consulting option. I appreciate your quick feedback!

CanastaWiki / Canasta