amis92
commented
4 years ago This work is in full tracked via https://github.com/orgs/BSData/projects/3
The current state is as follows:
https://github.com/BSData/gallery together with https://github.com/BSData/publish-catpkg provide a functional replacement for the appspot webapp backend.
- Gallery contains a list of registered repositories ("registry")
- Gallery runs a scheduled workflow to update details of registered repos and saves them as "index". Details include:
- repository latest release (including catpkg.json asset)
- repository details like "archived" status
- When "index" changes, a
bsdata.catpkg-gallery.jsonasset is uploaded to a Gallery release. This file contains list of repositories in the format recognized by BattleScribe as aRepositorySource. - Specific repositories in RepositorySource (
catpkg-gallery.json) point tocatpkg.jsonassets of the latest release of a given repository. catpkg.jsonrelease assets are uploaded (along with all other necessary files) by thepublish-catpkgAction. It should be run as per README (on release publish and edit).catpkg.jsonformat is recognized by BattleScribe as the RepositoryDetails which provides information about all datafiles - their URLs, names, and revisions. Based on revisions, the app chooses which files to download. Files are added to release assets by thepublish-catpkgAction.
SamwiseOfCat
The problem
We're not scalable. Full stop.
The current design of our organization is not scalable to the increasing popularity, needs and update frequency. The system has been failing intermittently (#208) since almost the beginning, but it was good enough.
Now it's not.
The proposal
We need to decentralize this operation. Keep liability and maintenance at minimum. As of writing this post, BSData has 119 repositories. There already was a proposal to deal with the issue: #112 - it was done at 49 repositories.
This means that datafiles should not be BSData-org exclusive. We need to allow indexing of third-party repositories as well. So that a person can setup their own repository and in some semi-automated way add it to the index. This is one part.
The other is that #112 needs to be solved - and the recipe is easier than ever thanks to GitHub Actions. BSData will provide an action that will do all necessary operations so that the BSData indexer only needs to read some manifest file of that repository/release. Each repository will run action by itself, thus offloading the indexer.
The work
Of course, being a volunteer-run entity, there's close to no momentum. I'd love to gain some support from community in refining and implementing this proposal.
I'll be slowly working on that myself, if anyone is wanting to contribute, please do.