search

weecology / updating-data

Hugo website for instructions on how to make a regularly updating data pipeline
https://www.updatingdata.org/
MIT License
0 stars 2 forks source link

Add support for GitHub Actions #7

Closed rudeboybert closed 3 years ago

rudeboybert commented 3 years ago

Hello, I previously had a chat with @ethanwhite about adding support for GitHub Actions (GA) given that Travis CI is slowly shifting to a business model that is slightly less open-source friendly. I've got a version working for tree growth data

Here are the immediate changes that come to mind, assuming we have support for both Travis and GA interspersed on the same pages:

  1. Page 4 "Connect to Travis" would become "Connect to Continuous Integration" and have Travis and GA subsections. The GA subsection would say something to the effect of "Nothing to do here since there is no need to link to different services"
  2. Page 5 "Allow for Automated Updating" would be split into "Generate Personal Access", "Give Travis Access to PAT", and "Give GitHub Actions Access to PAT" sections. Also FWIW we could also use usethis::create_github_token() to generate the token instead of going to GitHub.com settings
  3. Page 6 "Add data": Show similar snapshots of what it looks like on GA
  4. Page 7 "Add Data Code": Packages can be installed via a dummy DESCRIPTION file for the GitHub actions arm instead of using pacman (this file tricks GA into thinking this is an R package). See my example
  5. Page 8: "Add data checks" Show similar snapshots to what it looks like on GA
  6. Page 9: "Update data": Show similar snapshots to what it looks like on GA

Another approach instead of interspersing the Travis and GA approaches is to add a link to a completely separate page of equivalent 10 steps for GA.

Lastly, the following files would be different in the template repository weecology/livedat.git:

  1. The dummy DESCRIPTION I mention above
  2. The following .yaml file in github/workflows/
  3. archive.R which links the repo's contents to Zenodo. I don't think I could do this without a lot more investigating of GA at this point however.

Let me know if you'd be interested in this; I'm happy to make a PR.

ethanwhite commented 3 years ago

I think this would be a great addition and would be happy to support it in either format. Separate sections of pages is easiest for us. Mirrored sets of instructions is probably easiest for users. Given that my initial thought is mirrored instructions. I'm pretty sure (though it's been a while) that we can next things, so we could basically have a link in the sidebar for each infrastructure choice and then have it expand when you choose an infrastructure with the step by step for that infrastructure.

@rudeboybert if that sounds like the right choice to you I'll put it on my To Do list to get the basic infrastructure change made an copy the current instructions into a GitHub actions folder and then let you work on the necessary changes from there.

rudeboybert commented 3 years ago

This all sounds good. Your idea of laying down a scaffolding and then me filling in the blanks will likely make this go the smoothest possible.

ethanwhite commented 3 years ago

OK, you should be all set to go. The files for GitHub Actions are in /content/githubactions and are currently just a copy of the files from Travis CI, but with a modified _index.en.md file. Let me know if you have questions as you go. @gmyenni, who just finished shifting our Portal Data work over the GitHub actions, is also around to help as well.

rudeboybert commented 3 years ago

Great thanks, I should have time next week to do this.

rudeboybert commented 3 years ago

Hey @ethanwhite & @gmyenni, I forked a copy of this repo but I'm not able to locally preview the site. I was wondering if there is something obvious I'm missing. I've only used hugo in passing, so I'm only slightly above being a noob.

  1. I tried deploying the site locally by running hugo server. The resulting http://localhost:1313/ is coming out blank.
  2. I tried deploying the site on netlify by connecting my forked copy. I'm getting a 404 html error
  3. I noticed the themes/ folder only has a pointer to the learn @ 0c0b906 Hugo theme. So I cloned it into themes/, updated config.toml: theme from learn to hugo-theme-learn but am getting the following errors:
alberts-imac:updating-data rudeboybert$ hugo server
ERROR 2021/03/29 11:52:09 Failed to add template "theme/partials/header.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/hugo-theme-learn/layouts/partials/header.html": template: theme/partials/header.html:6: function "hugo" not defined
ERROR 2021/03/29 11:52:09 Failed to add template "theme/partials/logo.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/hugo-theme-learn/layouts/partials/logo.html": template: theme/partials/logo.html:1: function "cond" not defined
ERROR 2021/03/29 11:52:09 Failed to add template "theme/partials/menu.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/hugo-theme-learn/layouts/partials/menu.html": template: theme/partials/menu.html:17: function "cond" not defined
ERROR 2021/03/29 11:52:09 Failed to add template "theme/shortcodes/tab.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/hugo-theme-learn/layouts/shortcodes/tab.html": template: theme/shortcodes/tab.html:9: function "errorf" not defined
ERROR 2021/03/29 11:52:09 Failed to add template "theme/shortcodes/tabs.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/hugo-theme-learn/layouts/shortcodes/tabs.html": template: theme/shortcodes/tabs.html:9: function "cond" not defined
ERROR 2021/03/29 11:52:09 theme/partials/header.html : template: theme/partials/header.html:6: function "hugo" not defined
ERROR 2021/03/29 11:52:09 theme/partials/logo.html : template: theme/partials/logo.html:1: function "cond" not defined
ERROR 2021/03/29 11:52:09 theme/partials/menu.html : template: theme/partials/menu.html:17: function "cond" not defined
ERROR 2021/03/29 11:52:09 theme/shortcodes/tab.html : template: theme/shortcodes/tab.html:9: function "errorf" not defined
ERROR 2021/03/29 11:52:09 theme/shortcodes/tabs.html : template: theme/shortcodes/tabs.html:9: function "cond" not defined
Started building sites ...
ERROR 2021/03/29 11:52:09 Error while rendering "section": template: theme/_default/list.html:1:3: executing "theme/_default/list.html" at <partial "header.html...>: error calling partial: template: "theme/partials/header.html" is an incomplete or empty template
ERROR 2021/03/29 11:52:09 Error while rendering "home": template: theme/index.html:1:3: executing "theme/index.html" at <partial "header.html...>: error calling partial: template: "theme/partials/header.html" is an incomplete or empty template
ethanwhite commented 3 years ago

Sorry about that @rudeboybert - totally undocumented. You've got the right basic idea, but I'm guessing some detail just didn't work quite right. Can you try cloning again with the following command and see if that fixes things?

git clone --recurse-submodules https://github.com/weecology/updating-data.git
rudeboybert commented 3 years ago

Hey @ethanwhite, I tried cloning the repo using the command you provided, but when running hugo server, I'm getting the same error:

rudeboybert@alberts-imac updating-data % hugo server
ERROR 2021/04/02 09:25:45 Failed to add template "theme/partials/header.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/learn/layouts/partials/header.html": template: theme/partials/header.html:6: function "hugo" not defined
ERROR 2021/04/02 09:25:45 Failed to add template "theme/partials/logo.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/learn/layouts/partials/logo.html": template: theme/partials/logo.html:1: function "cond" not defined
ERROR 2021/04/02 09:25:45 Failed to add template "theme/partials/menu.html" in path "/Users/rudeboybert/Documents/Sabbatical/updating-data/themes/learn/layouts/partials/menu.html": template: theme/partials/menu.html:17: function "cond" not defined
ERROR 2021/04/02 09:25:45 theme/partials/header.html : template: theme/partials/header.html:6: function "hugo" not defined
ERROR 2021/04/02 09:25:45 theme/partials/logo.html : template: theme/partials/logo.html:1: function "cond" not defined
ERROR 2021/04/02 09:25:45 theme/partials/menu.html : template: theme/partials/menu.html:17: function "cond" not defined
Started building sites ...
ERROR 2021/04/02 09:25:45 Error while rendering "section": template: theme/_default/list.html:1:3: executing "theme/_default/list.html" at <partial "header.html...>: error calling partial: template: "theme/partials/header.html" is an incomplete or empty template
ERROR 2021/04/02 09:25:45 Error while rendering "home": template: theme/index.html:1:3: executing "theme/index.html" at <partial "header.html...>: error calling partial: template: "theme/partials/header.html" is an incomplete or empty template

Another approach would be for you to give me push access to this repo for the duration of time it takes to get the GitHub Actions part of the page up and running, if you're comfortable with that.

ethanwhite commented 3 years ago

I've given you push access to the repo. It will still be slower than developing locally but might be worth the hassle of not needing to get a local install up and running.

I'm guessing that this is a hugo version mismatch or a missing dependency. What is the output of hugo version?

rudeboybert commented 3 years ago

hugo v0.82.0+extended darwin/amd64 BuildDate=unknown FWIW I'm able to locally deploy a different Hugo website I have.

Also I just successfully made a trivial push to the website. This approach is probably as you say the path of least resistance for now. thanks

ethanwhite commented 3 years ago

Weird - I can't replicate locally and can't find anything useful for the error message with a quick search. Let me know if this starts getting in your way and I can start digging around.

rudeboybert commented 3 years ago

Hey @ethanwhite, some updates:

  1. I just pushed/deployed a minimally viable product of the GitHub Actions arm of the webpage. I still need to polish it and have a 3rd party test it, but the general outline is in place.
  2. I don't think I'll be able to figure out how to automate the Zenodo publishing, so I added a "Coming Soon" disclaimer to the Zenodo page.
  3. Note I did manage to get the webpage to deploy locally; I added instructions on how to do so in the README
  4. We're going to need a separate GitHub Actions version of the example repo weecology/livedat for users to import. At first I thought we could create a branch called github-actions in weecology/livedat, but I don't think there is a way to get a URL of a specific branch to be used for importing purposes. So could you create a weecology/livedat-github-actions repo and give me write privileges? I'll copy over the contents of the original weecology/livedat and make the necessary changes

thanks

ethanwhite commented 3 years ago

Apologies for losing track of this. I just created the repo and invited you as an admin.

ethanwhite commented 3 years ago

Nice work getting this going. The first draft looks great! Glad you got the local install working and thanks for adding instructions!

We have Zenodo publishing up and running here: https://github.com/weecology/PortalData/blob/main/.github/workflows/main.yml Take a look and see if that helps and ping @gmyenni if you need any help getting it working more generally.