search

Islandora-Collaboration-Group / ISLE

Islandora Enterprise (ISLE) is a community project that addresses two of the most significant pain-points in Islandora: installation and maintenance.
https://Islandora-Collaboration-Group.github.io/ISLE/
GNU General Public License v3.0
32 stars 34 forks source link

Insufficient TemporaryStore disk space halting FEDORA #401

Open McFateM opened 3 years ago

McFateM commented 3 years ago

Issue description

In my local instance of ISLE the FEDORA container frequently crashes with an ERROR like this:

ERROR 2020-12-18 17:03:20.119 [localhost-startStop-1] (BrokerService) Temporary Store limit is 51200 mb, whilst the temporary data directory: /usr/local/fedora/data/activemq-data/localhost/tmp_storage only has 49895 mb of usable space
Killed

These ERRORs are often accompanied by warnings that don't appear to kill the container, like so:

WARN 2020-12-18 17:03:20.093 [localhost-startStop-1] (BrokerService) Store limit is 102400 mb, whilst the data directory: /usr/local/fedora/data/activemq-data/localhost/KahaDB only has 49895 mb of usable space

For an issue, describe steps to reproduce the issue

System setup (OS information, software versions, etc):

Steps:

  1. Launch my local ISLE stack using docker-compose up -d. The container appears to be OK.
  2. After the stack is up, check for seven healthy containers, like docker ps.
  3. In my local workflow the next step involves importing a previously saved MySQL Drupal database, like so:
# Import the named .sql export
docker exec -i isle-mysql-ld mysql -uroot -pmydigitalgrinnellpassword20 digital_grinnell < ${VOLUME}/Backup-Data/local-export.sql
What's the expected result?

Seven healthy containers with a new imported Drupal database.

What's the actual result?

In my case, the isle-fedora-ld container stops and the logs show the ERROR and WARN messages I've included above.

noahwsmith commented 3 years ago

@McFateM if you can find this file I can help you get a PR in to lower the temp requirement: https://stackoverflow.com/a/27549226/324361

McFateM commented 3 years ago

Yup, that is the thread I found yesterday. Going to have a look in the config for it now. Thanks!

McFateM commented 3 years ago

Well, I couldn't find ANY of the config files mentioned in that thread, or others I followed. But, I did find a lot of conversation from a long time ago suggesting that ActiveMQ has the ability to adjust/reconfigure itself automatically. So, I decided to do some experimentation with my docker-compose.local.yml file and ActiveMQ, something I have been doing for a long time.

Recognizing that running out of ActiveMQ space is problematic, in the past I have always redirected my /usr/local/fedora/data/activemq-data directory, even before ISLE. Lately I stopped doing that because the default docker-compose.local.yml simply allows ISLE to let Docker handle that volume, like so:

     - isle-fedora-activemq:/usr/local/fedora/data/activemq-data

Hoping to gain some insight into this I changed that line to read: 

      - ~/DG-LOCAL/activeMQ:/usr/local/fedora/data/activemq-data

That change eliminated ALL of the ActiveMQ space-related warnings and ERRORs that I was seeing. Yay! But FEDORA is still crashing at the same point in the ISLE spin-up, where I am trying to engage the MySQL container to import a large Drupal database.

That makes some sense since I suspect ActiveMQ is involved in messaging FEDORA as MySQL makes progress. Seems reasonable, eh? In any event, I've since turned my attention back to those other, seemingly inconsequential INFO messages that isle-fedora-ld is still throwing, namely blocks like this:

2020-12-18T19:13:59Z 933a5b205d3b /usr/local/bin/confd[10035]: INFO /usr/local/fedora/server/config/fedora.fcfg has mode -rw-r--r-- should be -rwxr-----
2020-12-18T19:13:59Z 933a5b205d3b /usr/local/bin/confd[10035]: INFO /usr/local/fedora/server/config/fedora.fcfg has md5sum 731d01cff04366b0a24c7bc8eb529e44 should be 9661d23ad3263aefbc39d07ef96dbff4
2020-12-18T19:13:59Z 933a5b205d3b /usr/local/bin/confd[10035]: INFO Target config /usr/local/fedora/server/config/fedora.fcfg out of sync
2020-12-18T19:13:59Z 933a5b205d3b /usr/local/bin/confd[10035]: INFO Target config /usr/local/fedora/server/config/fedora.fcfg has been updated

I'm seeing lots of these and wondering exactly why? So my plan here is to rewind my workflow back to a stable state and repeat my attempt to upgrade from ISLE v1.5.1 to v1.5.2, step-by-step to try and determine where these curious issues are coming from.

g7morris commented 3 years ago

Hi @SummittDweller aka @McFateM aka Mark ;)

Would you like to submit a PR to the Fedora image https://github.com/Islandora-Collaboration-Group/isle-fedora with that fix for the next ISLE release?

We'd love to give you credit and a big thanks for this!

Hope you are well.

Cheers Gavin