Grails plugin portal goes down every time three days and must be reset

[niravassar]

Grails plugins page at https://plugins.grails.org/ shows 0 plugins sometimes and we have to go in an restart it. then it works again.

I took at look at this in depth and will post some notes

[niravassar]

How does the application Work

• SyncPluginsJobService - job runs every hour, grailsPluginService.refresh()

• GrailsPluginService -

  • clearNotUpdatedSince() - gets the list of plugins and trims off if they are static havent been touched.
  • then refreshFromGitHub() - try find the latest ones and tweet it out.
  • fetchGithubRepository() - gets the details of one repository with a http get call.
  • there is GRAILS_PLUGINS_METADATA_BASE_URL = "https://grails.github.io" and static final String GRAILS_PLUGINS_METADATA_PATH = "/grails-plugins-metadata/grails-plugins.json" which is where the list may come from.
    • in refreshFromGithub() there is a http call that gets the this url. then it GETs for the url https://grails.github.io/grails-plugins-metadata/grails-plugins.json
    • then it will run procedd per the body of the plugin

• GrailsPluginRepositoryService - gets infor from a plugin like package name and owner, and readme etc

• PluginController.index calls grailsPluginsRepository.findByQuery, which will call findAll if no query is sent - this just sorts it.

in the job, there is process(). this calls grailsPluginRepository. and this gets info about 1 plugin.

[niravassar]

Theory on Whats Going on

• actions to retrieve githiub repo information and scrape readme ascii are in grails async tasks. There are many of them and embedded upon each other consecutively.
• when these tasks don't complete too many threads line up. there is a lot of tasks going on simultaneous and the http calls done come back all the time
• it also calls fetchGithubRepository() and then fetchGithubReadme() even if first one fails, which conjure up even more failures

Recreate on Local

• i was able to recreate this on local by making the job SyncPluginsJobService run every 15 to 30 seconds.
• after 3 min i get an error and then heap error.

[niravassar]

Minimally Invasive Proposed Solution

• I am going to create PR that simply extends the job that pings all the repos and extend the periodic time. Right now it is 1hr, and then usually results in the app breaking in like 3 days. I am going to set it at 12 hours to see if this allows jobs time to resolve it self and either extend the "Good" period between breaks or eliminate it all together. here is the PR https://github.com/grails/grails3-plugins/pull/34

[niravassar]

More involved solution

Untangle the tasks by only creating one task per plugin. Each plugin will have one task and the http calls will be consecutively run to get repo information, readme information and owner information. They will only execute if the previous one completes successfully and will be synchronous.

However, the tasks between plugins will be aysnchrous as the next plugin will fire out a task as well. here is the PR https://github.com/grails/grails3-plugins/pull/35