Investigate GHA and/or submodules for maintaining ioos.us menu bars?

[mwengren]

@MathewBiddle and I have been working on using GHA to synchronize the menu bars on our 'IOOS DMAC Documentation Portal' GitHub Pages site (https://ioos.github.io).

xref: https://github.com/ioos/documentation-theme-jekyll/issues/3

In that site, we use specific branches in the https://github.com/ioos/documentation-theme-jekyll repo to maintain a common Jekyll 'theme' and certain configuration files used by the theme, which are then referenced as git submodules in the downstream repositories that host DMAC documentation content via GH Pages.

For example:

• Site: https://ioos.github.io/ioos-metadata/
• Repo: https://github.com/ioos/ioos-metadata

Could we use a similar approach to synchronize the menu bar content in the various *.ioos.us repos that we host on GitHub (e.g. https://github.com/ioos/comt-landing, https://github.com/ioos/hfradar-dac-landing)?

Referring to this menu bar at the top of each ioos.us sub-site:

I'm not too familiar with how these sites work (NodeJS?), I know there's a external rendering process and the ioos.us sites aren't actually hosted on GitHub of course, but would just the menubar content sync via submodule be possible here?

We recently added GHA support to automate the submodule synchronization between the Jekyll content sites, see: https://github.com/ioos/ioos-documentation-jekyll-skeleton/blob/gh-pages/.github/workflows/sync_theme.yml. This job is run on a daily schedule and also any time new commits are pushed to each documentation repo.

If the menubar code could be kept in a separate branch of this repo, then I would think a separate submodule/GHA sync technique might work.

As motivation, I noticed we have at least one dead link in the current menu bar that needs to be updated (under Viewers > COMT):

http://oceansmap.com/comt/

[nguyandy]

Hey Micah, I love the idea! I looked at the other *.ioos.us sites in the repo and it looks like they're all built using Node/Pug as a templating engine. Good news there is that the repos seem to be consistent in this aspect. I'm not familiar with Pug, but I don't see how implementing something like this would be an issue.

One thing to note is that there isn't automated deployment process for the ioos landing pages, the current process is a bit extra IMO (build docker image, push to Dockerhub, pull and deploy on ECR). Since we're just serving up static content, would it make sense to move away from this process and host these sites on Github pages or S3+Cloudfront?

[mwengren]

Hi @nguyandy. Thanks!

My understanding of git submodules is that they can serve in effect as a folder symlink does on unix, but not a file symlink, if that makes sense.

So, as long as the top menu content (header.html and whatever else) could be housed in a subfolder, presumably under https://github.com/ioos/ioos-us/tree/master/views where it is currently, a submodule link will work. That's the part I don't know about this site framework. If you can look into that, great!

Then, a submodule/branch could be added similar to the theme folder we use forJekyll content here: https://github.com/ioos/ioos-documentation-jekyll-skeleton.

As far as hosting platform, investigating GH Pages seems like a great idea, especially if the site build step could be written as a GitHub Action to run automatically on commit.

If not, then S3 sounds like a good alternative too.

Before deciding on a hosting change though, we'll need to discuss internally here.

The submodule approach I think is worth looking into whenever you have a chance though.

[sherischwartz]

Notes from Micah:

"One problem we realized after creating that issue that we'll need to bear in mind is that any automated GHA will stop running on a repo if there are no commits made to it after 90 days (I believe).
So, the approach we were thinking to base this off of - pulling in an updated submodule with header configuration via a GHA cron configuration (as in the ioos-documentation-jekyll-skeleton repo here: https://github.com/ioos/ioos-documentation-jekyll-skeleton/blob/gh-pages/.github/workflows/sync_theme.yml) may not be fully maintainable. You can re-enable GHA again, but it takes navigating to the repo with the submodule and GHA configured and manually re-enabling the action.
Here's an example of one of our documentation repos where the daily GHA stopped running back in Nov because of this timeout: https://github.com/ioos/ioos-metadata/actions.
Somewhere we have a GH issue of links to individual documentation repos so we know which ones to manually re-enable (since there are 15 or so), but at the moment I can't find it, so obviously not ideal in terms of maintenance.

I'm wondering if instead we should consider other ways to redesign ioos.us, but I don't know what alternatives might be (one big ioos.us site/repo using the same framework as the existing sites with a single nav bar, Wordpress, etc). "

[MathewBiddle]

@ocefpaf mentioned these required workflows and configuration variables just this week https://github.blog/2023-01-10-introducing-required-workflows-and-configuration-variables-to-github-actions/

It could be worth investigating how those could make our lives easier for these common GHA across repos.

[ocefpaf]

@MathewBiddle it seems that I waited way too long to send you that page:

https://docs.github.com/en/enterprise-server@3.11/actions/using-workflows/required-workflows

GH is sun-setting that feature and is supporting only those who got into the beta phase.

Sadly this is no longer an option. Sorry for the noise. However, this one may be a viable option to simplify the current setup: https://docs.github.com/en/actions/using-workflows/reusing-workflows#access-to-reusable-workflows

[nguyandy]

Hey @mwengren , We've got a bunch of ioos sites built on different tech stacks, and I’m thinking the slickest way to sync them up is by using a web component for both the header and footer.

By using a web component, specifically the use of shadow DOM, we can ensure that styles and scripts used by the header/footer are encapsulated. This means they won’t mess with whatever else is running on the site—giving us the best mix of compatibility and portability.

We’ll host the web component's JS file on S3 and serve it through Cloudfront. Any project that needs the new header or footer can just load the component from our CDN. This way, we skip needing to set up separate deployment pipelines for every project using these components.

Adding them to a site is a breeze: Just include the script in the HTML:

<script src=“some-url/ioos-layout.js"></script>

And then drop in the custom tags wherever you need them:

<ioos-header></ioos-header>
<ioos-footer></ioos-footer>

We can also toss in some attributes to tweak settings or add customization options down the road:

<ioos-header theme=“dark”></ioos-header>

Workflow for updating the header/footer

1. Make changes to the ioos-layout web component
2. Build and push the component to S3
3. Any site using the component will automatically snag the latest version thanks to Cloudfront.

Cloudfront lets us set HTTP cache headers and even invalidate old cached versions when we push updates. That means we can keep things fresh without any fuss.

I’ve got a POC already up and running on S3 here: https://ioos-cloud-www.s3.amazonaws.com/ioos-layout.min.js

And I've slapped this header into a bunch of different projects to see it in action—it’s working like a charm.

I think this setup will smooth out a lot of our headaches, but please let me know if you see any potential snags or have questions.

[mwengren]

Hi @nguyandy, sorry for the slow response!

This sounds promising to me. I had a couple of reactions/questions:

1. Where would the source for the web component be stored? On GitHub in this ioos-us repo? I see the link you included above and I see that's been minified, but would the source be something that would be preview-able if it were hosted on GitHub? How would someone go about proposing a change?

2. The web component would be able to be loaded by all of the current .ioos.us sites? Since not all of our .ioos.us sub-sites are hosted on GitHub (I'm thinking mainly of the Axiom ones like sensors.ioos.us, mbon.ioos.us, and videos.ioos.us - atn.ioos.us is in one of our GitHub repos), it would be up to Axiom to integrate the component with their code/servers, so we'd need to ensure it will work for their system.

3. Ideally, we should aim to have all of the *.ioos.us sites use this web component and display a uniform header and footer across the board, however this might require some changes to existing sites menus (beyond just integrating the shared web component). I'm thinking of the IOOS Catalog (data.ioos.us), for one, which already has its own header bar with similar menu items. It wouldn't make sense to have a 'Data' menu on the web component menu, and then another 'Datasets' menu item on the Catalog page menu. So, we'd have to make some changes there. This might affect other sub-sites as well, I haven't looked at them all. We'd also probably need to decide on what goes in the web component menu itself - the current menu probably needs review/update.

4. At the same time we implement this, could we look into whether we might be able to consolidate some of the GitHub repos backing each of the static *.ioos.us subdomain sites? It would be nice to not have a separate repo for comt.ioos.us, hfradar.ioos.us, and osmc.ioos.us, and on and on. I expect mapping the different domains to different source pages/content might be an issue (i.e. when someone goes to comt.ioos.us, what static page is returned if it's served from the same repo as hfradar.ioos.us)? Any chance this can be done?

So, especially regarding the Items 3 and 4, this might turn into its own project to implement these changes across all of our ioos.us sites.

I'm thinking we might want to assign this implementation project to a new staff member we have coming on board soon. Will discuss with our team and figure out a plan.

In the meantime, please let me know any thoughts you have on the above. If we can get some agreement on Items 1 and 2, I think we can move forward with this approach. Thanks!

[MathewBiddle]

@nguyandy this looks promising! Thanks for digging into it and putting this proposal out here.

My first reaction is in line with @mwengren's first comment above. Where will the source be stored and how can we contribute to it? It would be nice to have it in a GitHub repo.

Otherwise, I'm on board. But, addressing Micah's other notes is essential too.

[nguyandy]

Hey @mwengren, @MathewBiddle, sorry I thought I replied

Where would the source for the web component be stored? On GitHub in this ioos-us repo? I see the link you included above and I see that's been minified, but would the source be something that would be preview-able if it were hosted on GitHub? How would someone go about proposing a change?

Where ever you think makes since. It is a node project, the code gets minified during the build process. It should be relatively easy for anyone to make changes to the components.

If you have a repo/location in mind for this, let me know and I can start moving things over!

The web component would be able to be loaded by all of the current .ioos.us sites? Since not all of our .ioos.us sub-sites are hosted on GitHub (I'm thinking mainly of the Axiom ones like sensors.ioos.us, mbon.ioos.us, and videos.ioos.us - atn.ioos.us is in one of our GitHub repos), it would be up to Axiom to integrate the component with their code/servers, so we'd need to ensure it will work for their system.

Yes, the components themselves can be integrated easily into any of the websites by importing the component src file like in the example from my previous post. The styling is encapsulated as well so it shouldn't affect any site design.

Ideally, we should aim to have all of the *.ioos.us sites use this web component and display a uniform header and footer across the board, however this might require some changes to existing sites menus (beyond just integrating the shared web component). I'm thinking of the IOOS Catalog (data.ioos.us), for one, which already has its own header bar with similar menu items. It wouldn't make sense to have a 'Data' menu on the web component menu, and then another 'Datasets' menu item on the Catalog page menu. So, we'd have to make some changes there. This might affect other sub-sites as well, I haven't looked at them all. We'd also probably need to decide on what goes in the web component menu itself - the current menu probably needs review/update.

In the node project directory, we can have a separate .json config file that is used to build the component menu items. For projects that wish to use their own components, but still wish to have the menu items synchronized to the main ioos.us specs, they can reference the .json config file either during their build process, or referenced through the file which will be available through the Github repo or Cloudfront.

This is the current config I through together for the example above, but can be refined to fit our needs.

const menus = [
    {
        "id": "drop1",
        "title": "Data",
        "items": [
            { "title": "Metrics", "url": "https://ioos.noaa.gov/about/ioos-by-the-numbers/", "target": "_blank" },
            { "title": "Search", "url": "https://data.ioos.us/", "target": "_blank" },
            { "title": "Compliance Checker", "url": "https://compliance.ioos.us/index.html", "target": "_blank" },
            { "title": "IOOS Archive Data Portal", "url": "https://www.nodc.noaa.gov/ioos/", "target": "_blank" },
            { "title": "Coastal and Ocean Modeling Testbed", "url": "https://comt.ioos.us", "target": "_blank" }
        ]
    },
    {
        "id": "drop2",
        "title": "Viewers",
        "items": [
            { "title": "Coastal and Ocean Modeling Testbed", "url": "https://oceansmap.com/comt/", "target": "_blank" },
            { "title": "Sensor Map", "url": "https://sensors.ioos.us/", "target": "_blank" },
            { "title": "IOOS Model Viewer", "url": "https://eds.ioos.us/", "target": "_blank" },
            { "title": "Profiling Gliders", "url": "https://gliders.ioos.us/", "target": "_blank" }
        ]
    },
    {
        "id": "drop3",
        "title": "DACs",
        "items": [
            { "title": "Glider DAC", "url": "https://gliders.ioos.us/", "target": "_blank" },
            { "title": "HF Radar DAC", "url": "https://hfradar.ioos.us/", "target": "_blank" },
            { "title": "ATN DAC", "url": "https://atn.ioos.us/", "target": "_blank" }
        ]
    },
    {
        "id": "drop5",
        "title": "Regional Associations",
        "items": [
            { "title": "IOOS Region Map", "url": "https://ioos.us/regions", "target": "_blank" },
            { "title": "Alaska - AOOS", "url": "https://portal.aoos.org/", "target": "_blank" },
            { "title": "Caribbean - CariCOOS", "url": "https://www.caricoos.org/drupal/", "target": "_blank" },
            { "title": "Central and Northern California - CeNCOOS", "url": "https://data.cencoos.org/", "target": "_blank" },
            { "title": "Great Lakes - GLOS", "url": "https://glos.us/", "target": "_blank" },
            { "title": "Gulf of Mexico - GCOOS", "url": "https://data.gcoos.org/", "target": "_blank" },
            { "title": "Mid-Atlantic - MARACOOS", "url": "https://maracoos.org/", "target": "_blank" },
            { "title": "Northeast Atlantic - NERACOOS", "url": "https://neracoos.org/realtime_map", "target": "_blank" },
            { "title": "Pacific Islands - PacIOOS", "url": "https://oos.soest.hawaii.edu/pacioos/voyager/", "target": "_blank" },
            { "title": "Pacific Northwest - NANOOS", "url": "https://nvs.nanoos.org/", "target": "_blank" },
            { "title": "Southeast Atlantic - SECOORA", "url": "https://portal.secoora.org/", "target": "_blank" },
            { "title": "Southern California - SCCOOS", "url": "https://www.sccoos.org/data/overview/", "target": "_blank" }
        ]
    },
    {
        "id": "drop4",
        "title": "About",
        "items": [
            { "title": "Contact Us", "url": "https://ioos.us/contact-us" }
        ]
    }
];

At the same time we implement this, could we look into whether we might be able to consolidate some of the GitHub repos backing each of the static *.ioos.us subdomain sites? It would be nice to not have a separate repo for comt.ioos.us, hfradar.ioos.us, and osmc.ioos.us, and on and on. I expect mapping the different domains to different source pages/content might be an issue (i.e. when someone goes to comt.ioos.us, what static page is returned if it's served from the same repo as hfradar.ioos.us)? Any chance this can be done?

I don't see why not! I imagine we would have a main template repo, where each of the subsites will have their own project directory that contains their landing page content.

You're right, we'd probably have to figure out how to handle the mapping/routing if we want to deploy it as a single website.

These static websites are pretty small in size though. To avoid overcomplicating things, we can continue to build the subsites individually through a CI/CD pipeline to start.

[mwengren]

Hi @nguyandy thanks, and we'll get back to you with more details soon-ish. I'm just pinging @sarinamann-noaa on this because she'll be taking on coordinating this from our side and reaching out to each of the maintainers of the various *.ioos.us sub-sites to coordinate their implementing the new menu framework.

IOOS will probably have an internal meeting to kick the project off first, and before anything is deployed, we (IOOS Ops Division/DMAC) need to decide how to update the menubar content itself (.json file example you posted above) since it's been awhile since we've changed it.