Fedora resources are all 404's; container has database problems

[rosiel]

I'm not sure if these are related.

1 - I can't write to Fedora. When I create a node then click the Fedora URI, it's a 404. When I try to upload a file, if I'm uploading at the default monthly folder which hasn't been created yet (2023-09), I get the error that everyone's been posting in Slack, upload directory fedora://2023-09 for the file field field_media_file could not be created or is not accessible.. And - what's potentially even worse, if I take out that folder so I'm writing straight to the top level e.g. fedora://dead_bug.gif, then my file appears to upload and the media saves, though trying to access the file (manually or for derivatives) yeilds a 404.

2 - The fedora container has a bunch of logs (only visible in Docker Desktop as far as I can tell) complaining about a lack of database connection:

2023-09-25 17:19:22 SQL State  : 90020
2023-09-25 17:19:22 Error Code : 90020
2023-09-25 17:19:22 Message    : Database may be already in use: "Locked by another computer: 172.22.0.14". 
Possible solutions: close all other connection(s); use the server mode [90020-210]

[rosiel]

In diving into where this database connection may be, I see some environment variables in the fcrepo container that look promising:

/opt/tomcat # env | grep DB | sort
DB_DRIVER=mysql
DB_HOST=
DB_MYSQL_HOST=mariadb
DB_MYSQL_PORT=3306
DB_NAME=default
DB_PASSWORD=password
DB_PORT=
DB_POSTGRESQL_HOST=postgresql
DB_POSTGRESQL_PORT=5432
DB_ROOT_PASSWORD=password
DB_ROOT_USER=root
DB_USER=default
FCREPO_DB_NAME=fcrepo
FCREPO_DB_PASSWORD=password
FCREPO_DB_USER=fcrepo

I assume that DB_DRIVER selects the MYSQL values for HOST and PORT (because they're otherwise missing). This would imply there's a fcrepo database in the mariadb container that's busy.

In the MariaDB container,

MariaDB [(none)]> show databases;
+--------------------+
| Database           |
+--------------------+
| drupal_default     |
| information_schema |
| matomo             |
| mysql              |
| performance_schema |
| sys                |
+--------------------+
6 rows in set (0.006 sec)

There does not seem to be a database set up for Fedora.

[highermath]

I had thought that the standard for Fedora was to use Postgres. Could your Fedora be set up to use that, even it if might not exist?

[nigelgbanks]

By default, unless otherwise configured, Fedora will use its file storage mechanism rather than a database.

https://github.com/Islandora-Devops/isle-buildkit/blob/c9ab7e77665a390a0a73124686339cf9a08881b1/fcrepo6/Dockerfile#L57

This seemed to have worked for years? I merged this 3 weeks ago that updated the FCREPO by a minor version https://github.com/Islandora-Devops/isle-buildkit/commit/740f6878fac5e66d32f8d6171c2b028be081c479

This could have dropped that feature? Or it behaves differently now https://wiki.lyrasis.org/display/FEDORA6x/Database+Cache

I would have hoped the automatic tests would have caught that though.

[nigelgbanks]

We don't have an automatic test for the file backend...

https://github.com/Islandora-Devops/isle-buildkit/tree/main/fcrepo6/tests

[nigelgbanks]

I've added a test, it appears to be working.

https://github.com/Islandora-Devops/isle-buildkit/commit/20ce60742bf65341613d3c867fc505cbeb328efa

[nigelgbanks]

I've actually seen this bug before, some time ago....

https://github.com/Islandora/documentation/issues/2094

[nigelgbanks]

Seems to be handled by pull #23