Keep project data in dotfiles in corresponding project repos and pull it during site build

[strogonoff]

Currently project data in YAML format must be included in project site source, and additionally duplicated in hub site source.

What we could instead do is the following approach:

• Keep structured project data in a dotfile .open-project.yml in project source repository root. This seems fairly convenient and is similar to, e.g., how Gitlab pulls its CI configuration. I believe GitHub Pages does a similar thing.
• In hub site and project sites configuration, simply link to the repos. Let the site build process (could be done within a Jekyll plugin) pull that data and generate static pages similar to the existing structure or in accordance with the designs.

Proper implementation here would likely also deal with #2 in one blow.

[ronaldtse]

Yes this works indeed. The only issue is that the hub pages need to be rebuilt whenever a project site gets updated, but that's a given due to the role of the hub pages.

[strogonoff]

@ronaldtse yeah, this implies some hook to trigger site suite to be rebuilt whenever .open-project.yml gets updated, and we would want to be more efficient and avoid rebuilding sites for projects which data was not modified.

Seems like it’s doable with CI: trigger project site build when a specified branch is updated, and trigger hub site build when that branch is updated in any of project repos. There may be a bit of duplication where we’ll have to link to same list of project source repos in (1) hub site Jekyll plugin configuration, and (2) hub site CI configuration, but that doesn’t seem especially dire.

[ronaldtse]

Yes, we can even just do git submodule of project sites within the hub site repo. And all we need is to update the config.yml file to include those posts (I think...).

[strogonoff]

I guess we have two options:

1. Configure hub site with links to project repos, and let build process pull the data from those repos, possibly specific branches. (IMO instead of having an “open project site” repo for each project we could use a branch inside actual project codebase repos and keep the data there. Again, similar to one of approaches to GitHub Pages setup.)
2. Submodule project site data. I’m not sure if we’ll be able to submodule a specific branch (it seems a bit messy), but it’s an option.

I prefer the first one for now.

[strogonoff]

And all we need is to update the config.yml file to include those posts (I think...).

It’s currently a bit more complicated but yeah, the current end idea is that we’d just configure a list of projects in _config.yml and let jekyll build pull data for those projects, possibly with local caching. That way project data exists in one place, and thanks to commonly used design pattern (a dotfile in repo root) it’d be easy to understand the setup.

Edit: submodules eliminate the need for local caching, on flip side they add complexity to the setup but if that only affects the hub site it may be acceptable.

[strogonoff]

This is implemented as of 1.0.0. Current implementation does not involve dotfile per se, but works similarly.

Below is an illustration of how it works given one hub, one project, and one software and spec (with the caveat that spec fetching currently is not used for any specific feature).

[ronaldtse]

@strogonoff thanks for the beautiful diagram 😉

One note that if you have time can you go through our list of repos -- many things (i.e. new/renamed repos) have updated since we first started. Thanks!