Dynamically pull repo information for HPC lessons

[ocaisa]

Fixes #24

[ocaisa]

See https://github.com/hpc-carpentry/hpc-carpentry.github.io/runs/2391477702 for where this change is checked

[ocaisa]

For this to work completely we will need to start using GitHub topics consistently on our repos. In particular things need to be at least tagged with lesson and hpc-carpentry before they will appear. The nice thing is that they will still show up even if they get migrated to the Carpentries incubator.

[tkphd]

If those "tags" are Labels, we can configure Issue Templates for each repo to apply them automagically going forward. Then we only have to manually intervene with... all of history.

[ocaisa]

If those "tags" are Labels, we can configure Issue Templates for each repo to apply them automagically going forward. Then we only have to manually intervene with... all of history.

No, not labels (that's what I thought too first). The labels are used for the help wanted data, but the topics are set globally for the repo in the "About" section of the repo homepage (see https://github.com/hpc-carpentry/hpc-shell where I set them for testing this)

[tkphd]

Oh, cool!! That's even easier!

[ocaisa]

At the moment we don't actually have a "Help Wanted" page (I think) but with this we can add one based on the Carpentries.

It also opens the door to being able to include lessons from other providers under separate organisations (and we can group them accordingly)

[tkphd]

Yes, this is fantastic. I introduced a "Help Wanted" page in #22, but hybridizing this feed with The Carpentries' code is a much better solution.

[tkphd]

Applied the lesson and hpc-carpentry tags to all our lesson repositories, FYI. Also english, alpha, and (for Chapel) pre-alpha.

[ocaisa]

Applied the lesson and hpc-carpentry tags to all our lesson repositories, FYI. Also english, alpha, and (for Chapel) pre-alpha.

I tested this locally and they all appear fine

[ocaisa]

For the record, here are the changes required to the upstream code at https://github.com/carpentries/feeds.carpentries.org/tree/main/R

diff --git a/home/alanc/feeds.carpentries.org/R/community_lessons.R b/R/hpc-carpentry_lessons.R
index b0d33b8a..a8ead1c5 100644
--- a/home/alanc/feeds.carpentries.org/R/community_lessons.R
+++ b/feeds/hpc-carpentry_lessons.R
@@ -1,6 +1,7 @@
 source("R/utils.R")

 LIFE_CYCLE_TAGS <- c("pre-alpha", "alpha", "beta", "stable")
+# The tags below will be filtered out in the json
 COMMON_TAGS <- c(
   "carpentries",
   "carpentries-incubator",
@@ -43,10 +44,11 @@ check_repo_info <- function(.d, fields) {
 make_community_lessons_feed <- function(path, ...) {

   carp_inc <- get_org_topics("carpentries-incubator")
-  carp_lab <- get_org_topics("carpentrieslab")
+  hpc_carp <- get_org_topics("hpc-carpentry")

-  res <- dplyr::bind_rows(carp_inc, carp_lab) %>%
+  res <- dplyr::bind_rows(carp_inc, hpc_carp) %>%
     dplyr::select(-private) %>%
+    dplyr::filter(grepl("hpc-carpentry", github_topics)) %>%
     dplyr::filter(grepl("lesson", github_topics)) %>%
     extract_tag(
       life_cycle_tag,
@@ -71,4 +73,4 @@ make_community_lessons_feed <- function(path, ...) {

 }

-make_community_lessons_feed("_data/community_lessons.json")
+make_community_lessons_feed("_data/hpc_lessons.json")

diff --git a/home/alanc/feeds.carpentries.org/R/help_wanted_issues.R b/feeds/help_wanted_issues.R
index d0f79ddd..cb154f6e 100644
--- a/home/alanc/feeds.carpentries.org/R/help_wanted_issues.R
+++ b/feeds/help_wanted_issues.R
@@ -1,4 +1,4 @@
-source("R/utils.R")
+source("feeds/utils.R")

 new_tbl_github_issues <- function(url = character(0),
                                   title = character(0),
@@ -97,40 +97,17 @@ get_gh_issues <- function(owner, repo, labels) {
     )
 }

-keep_opted_in <- function(orgs) {
-
-  at_opted_in <- airtabler::airtable(
-    base = "appeZJGnGremE1MYm",
-    tables = "Repositories"
+keep_hpc_carpentry_repos <- function(orgs) {
+  dplyr::filter(
+    orgs, carpentries_org == "hpc-carpentry"
   )
-
-  opted_in <- at_opted_in$Repositories$select_all() %>%
-    dplyr::mutate(lesson_program = tolower(lesson_program))
-
-  dplyr::left_join(
-    orgs, opted_in,
-    by = c(
-      "carpentries_org" = "lesson_program",
-      "repo" = "repository"
-    )
-  ) %>%
-    dplyr::filter(display_help_wanted)
-
 }

 keep_other_repos <- function(orgs) {
+  # Repos that wish to opt in should be manually added here
   other_repos <- tibble::tribble(
     ~carpentries_org, ~repo,
-    "carpentries", "glosario",
-    "carpentries", "glosario-r",
-    "carpentries", "glosario-py",
-    ##    "carpentries", "glosario-js",
-    "carpentries", "carpentries.org",
-    "carpentries", "docs.carpentries.org",
-    "datacarpentry", "datacarpentry.github.io",
-    "librarycarpentry", "librarycarpentry.github.io",
-    "carpentryconnect", "carpentryconnect.org",
-    "swcarpentry", "website"
+    "carpentries-incubator", "hpc-intro",
   )

   dplyr::inner_join(
@@ -141,13 +118,8 @@ keep_other_repos <- function(orgs) {

 list_organizations <- c(
-  "Data Carpentry" = "datacarpentry",
-  "Software Carpentry" = "swcarpentry",
-  "Library Carpentry" = "librarycarpentry",
-  "CarpentriesLab" = "carpentrieslab",
-  "The Carpentries Incubator" = "carpentries-incubator",
-  "The Carpentries" = "carpentries",
-  "CarpentryConnect" = "carpentryconnect"
+  "HPC Carpentry" = "hpc-carpentry",
+  "The Carpentries Incubator" = "carpentries-incubator"
 )

 list_help_wanted <- purrr::imap_dfr(
@@ -158,14 +130,14 @@ list_help_wanted <- purrr::imap_dfr(
       ignore_pattern = "^\\d{4}-\\d{2}-\\d{2}"
     )

-    lessons <- orgs %>%
-      keep_opted_in()
+    hpc_carpentry_repos <- orgs %>%
+      keep_hpc_carpentry_repos()

     other_repos <- orgs %>%
       keep_other_repos()

     dplyr::bind_rows(
-      lessons,
+      hpc_carpentry_repos,
       other_repos
     )  %>%
       dplyr::distinct(carpentries_org, repo, .keep_all = TRUE) %>%

[tkphd]

@ocaisa the EESSI trick is neat for getting an R environment. Is there a way to run it locally, so I can make serve and see the updated list of lessons? Can we document that step?

[ocaisa]

@ocaisa the EESSI trick is neat for getting an R environment. Is there a way to run it locally, so I can make serve and see the updated list of lessons? Can we document that step?

@tkphd There are a few example installation scripts at https://github.com/EESSI/eessi-demo/tree/main/scripts but they are slightly outdated (but should still work). You can now use https://github.com/EESSI/filesystem-layer/releases/download/latest/cvmfs-config-eessi_latest_all.deb as the configuration file.

After that you need to source /cvmfs/pilot.eessi-hpc.org/latest/init/bash and that will give you access to the stack for your CPU. You can then follow the same steps as the CI:

GITHUB_PAT=XXXXX make data
make serve