Jira fails to start due to corrupted plugin cache

[jhult]

Description

Jira fails to start sometimes with the following error:

System plugins failed to start: We haven't been able to start all the required system plugins

We couldn't start the following plugins: ...

Stoping and starting sometimes fixes the issue.

This doesn't appear to be the fault of the Docker container.

Has anyone else run into this?

Additional Reading

• https://jira.atlassian.com/browse/JRASERVER-27314
• https://confluence.atlassian.com/jirakb/troubleshooting-jira-startup-failed-error-394464512.html#TroubleshootingJIRAStartupFailedError-Cache
• https://community.atlassian.com/t5/Jira-Software-questions/JIRA-access-problem-after-update/qaq-p/654160

What I've tried

• I have tried increasing the plugin timeout. While this does seem to help, it still has issues the majority of the time.

Possible solutions

I'm wondering if we can or should add some sort of script to clear the plugin cache such as this:

rm -rf /var/atlassian/jira/plugins/.bundled-plugins/ rm -rf /var/atlassian/jira/plugins/.osgi-plugins/

Steps to Reproduce

1. Stop Jira
2. Start Jira

Expected behavior: [What you expect to happen] Jira starts successfully

Actual behavior: [What actually happens] Startup hangs

Reproduces how often: [What percentage of the time does it reproduce?] It is not 100% of the time but seems often, maybe 80% +.

Versions

7.x

[blacklabelops]

I had been thinking about the correct handling of the script.

I could add such script but I am unsure where to apply the script.

Triggering the script when Jira is up seems to be unhealthy.

Triggering the script for each startup seems also to be unhealthy.

What would be helpful?

[jhult]

I'm not sure I know the best way (or place) to do this.

I'm okay with doing it on each startup but others might not be. Maybe we could have sort of flag in docker-compose.yml which would enable this?

[Firefishy]

@jhult Did the rm -rf stuff solve the issue?

This hints at the issue possibly being linked to permissions.

@blacklabelops Can you expand on following?

Triggering the script for each startup seems also to be unhealthy.

[jhult]

rm -rf does not always solve the issue. Often, it does. However, sometimes I have to manually go empty the entire plugins folder and then move back JARs a few at a time.

[Firefishy]

It is also worth noting that a docker restart does NOT reset the restarted container, any new files created within the container will still be there after the restart. The ENTRYPOINT should make some attempt to clean up the container.

[blacklabelops]

After some googling I would say that removing those folders is safe, just increases your startup time.

[blacklabelops]

Proposal: Env with value true or false. When true then cleanup script is triggered inside entrypoint.

[blacklabelops]

Documentation: https://github.com/blacklabelops/jira#jira-startup-plugin-purge