Warning in Dashboard on More than 1000 Items in any GitHub Tree

[jaswrks]

A fork of: https://github.com/websharks/wp-kb-articles/issues/12#issuecomment-76542204

Important Note

While WPKBA now supports an unlimited number of files/directories, there is an important issue worth noting about the GitHub API. It pertains to the max number of items in any given directory tree.

Any tree that contains more than 1000 items is going to create a problem. You can have two gazillion directories with 1000 items each. However, if you have 1 directory with 1001 items this will cause an issue, because any single API call to retrieve a tree will return at most 1000 items.

For that reason, it is suggested that you use date-based subdirectories to prevent any single tree from containing more than 1000 items. 1000 being the most that a single API call will return.

The limit of 1000 is per-directory (i.e., tree) and not a total for all subdirectories. In other words, you can have one folder with 1000 subfolders, each with 1000 more subfolders, each with 1000 files. The key is that you don't allow more than 1000 sub-directories or files inside any given directory, regardless of it's depth.

To clarify further, a "tree" is a directory (regardless of it's depth). Each tree returns a list of items that consist of other trees (subdirectories), or files inside the tree. Trees can be nested as deep as you like, in any quantity that you like. However, there can never be more than 1000 items in any given tree.

That means that even the following directory structure may eventually hit a limit over time:

- tutorials/
  - 0101/
  - 0102/
  - ... up to 1000 items max in the `tutorials` tree.

In this example, tutorials is a tree, 0101 is another tree, and 0102 is another tree. Each of these can contain up to 1000 items each (i.e., subdirectories or files).

---

@raamdev writes...

Is there any way to check how many items exist in a given subdirectory so that site owners can be warned on the front-end that a particular directory contains more than 1000 items?

@jaswsinc writes...

Yes, there is a truncated flag that we could look for. https://developer.github.com/v3/git/trees/

[jaswrks]

Suggested directory structure for large KB repos.

• tutorials/
  • 2015/
  • 01/
  • 02/
  • 03/
  • ... up to 12
  • 2016/
  • 01/
  • 02/
  • 03/
  • ... up to 12
• questions/
  • 2015/
  • 01/
  • 02/
  • 03/
  • ... up to 12
  • 2016/
  • 01/
  • 02/
  • 03/
  • ... up to 12
• videos/
  • 2015/
  • 01/
  • 02/
  • 03/
  • ... up to 12
  • 2016/
  • 01/
  • 02/
  • 03/
  • ... up to 12

[raamdev]

Suggested directory structure for large KB repos.

You could also just start with putting articles in questions/2015/ and then if you find the number of articles at some point that year approaching 1000, start putting them in a new sub-directory for the current month. That's what I'd do if I didn't feel it it was likely that a particular site was going to hit 1000 articles that year.

That strategy would allow for long-term growth of a KB, as you'd be able to separate the articles into year sub-directories without immediately going to the year/month/ level.

[jaswrks]

Agree, that seems easier.