site-maven-plugin fails with "API Rate Limit Exceeded" when project is large

[jprante]

Hi,

when I try to create a site of a project https://github.com/jprante/elasticsearch with large javadocs, it fails badly.

The first tries produced 403 errors after long runs (15 minutes estimated):

Failed to execute goal com.github.github:site-maven-plugin:0.6:site (default) on project elasticsearch: Error creating blob: forbidden (403)

Okay, that confused me a bit, but I assume it failed because of too many blobs. So I reduced from >10,000 javadoc files down to ~7,000 by dropping test-javadocs.

My last try resulted in an API rate limit exceeded.

[INFO] --- site-maven-plugin:0.6:site (default) @ elasticsearch ---
[INFO] Creating 6,848 blobs
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 15:28.961s
[INFO] Finished at: Tue May 15 01:11:26 CEST 2012
[INFO] Final Memory: 38M/523M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.github.github:site-maven-plugin:0.6:site (default) on project elasticsearch: Error creating blob: API Rate Limit Exceeded for jprante (403) -> [Help 1]

DryRun works without problems.

How can I succeed (without breaking down the project into smaller pieces)?

Or, related question, what is the maximum number of blobs that can be created with github site maven plugin at once? Is it possible to batch / compress blobs?

Thank you for your time and help,

Jörg

[5AMSolutions]

Overall, the plugin seems to be great though it's hamstrung by GH's API Rate Limits.

This error is very annoying. Our project (CONNECT-Solution/CONNECT) is quite massive and needs to perform numerous API calls far more than the 5000 request/hr per GH API v3 Rate Limit. I've forked and looked at the com.github.github:site-maven-plugin src in hopes of finding a way to leverage: batching of blobs but don't see anything and/or the "Unauthenticated rate limited requests" by passing along "?client_id=xxxxxxxxxxxxxx&client_secret=yyyyyyyyyyyyyyyyyyyyy". Sadly, nothing leaped out at me immediately though what do you expect when you're staring at entirely foreign code.

Alternatively, I wonder if one could simply commit our site files to "refs/heads/gh-pages" using the good ole' git cmdline client though I'm not yet enough of a Git and GH expert to know.

[5AMSolutions]

I think using a traditional GIT SCM to publish to gh-pages branch would be wisest and more reliable as no Rate Limits are imposed. I saw this link http://maven.apache.org/plugins/maven-scm-publish-plugin/ and will explore more. The 10K' approach would be to do a 'mvn site:site site:stage' and stage to a temp directory then use the scm plugin to publish to the gh-pages branch.

[justinhrobbins]

+1 i have same issue when using this plugin

[mdoering]

same here. A shame as it worked really well without much configuration hassles

[yegor256]

+1 same here

[bguerin]

+1 same here too