+ GH Action to render news articles

bbest commented 1 year ago

[x] Move the news_code into _scripts
[x] https://github.com/quarto-dev/quarto-actions

bbest commented 1 year ago

Getting error installing integral R package in GH Action log fix bug · cstories-app/beta@feaeab5:

Retrieving 'https://cran.rstudio.com/src/contrib/Archive/integral/integral_1.0.1.9001.tar.gz' ...
Warning: Warning: failed to find source for 'integral 1.0.1.9001' in package repositories
Warning: Warning: failed to retrieve 'https://cran.rstudio.com/src/contrib/Archive/integral/integral_1.0.1.9001.tar.gz' [error code 22]
Error: Error: failed to retrieve package 'integral@1.0.1.9001'
In addition: Warning message:
curl: (22) The requested URL returned error: 404 
Traceback (most recent calls last):
9: renv::restore()
8: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
7: retrieve(packages)
6: handler(package, renv_retrieve_impl(package))
5: renv_retrieve_impl(package)
4: renv_retrieve_unknown_source(record)
3: renv_retrieve_repos(record)
2: stopf("failed to retrieve package '%s'", renv_record_format_remote(record))
1: stop(sprintf(fmt, ...), call. = call.)
Execution halted
Error: Process completed with exit code 1.

Ran the following locally in RStudio after opening beta.Rproj:

renv::install("IntegralEnvision/integral")
renv::snapshot()

Here was the output alongside input:

> renv::install("IntegralEnvision/integral")
Retrieving 'https://api.github.com/repos/IntegralEnvision/integral/tarball/2fd02dc3b79ce224f68fa35aed4c03b57d28170d' ...
    OK [downloaded 23.7 Mb in 7.2 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.2/rlang_1.0.6.tgz' ...
    OK [downloaded 1.7 Mb in 0.4 secs]
Retrieving 'https://cran.rstudio.com/bin/macosx/big-sur-arm64/contrib/4.2/htmltools_0.5.4.tgz' ...
    OK [downloaded 339.7 Kb in 0.2 secs]
Installing rlang [1.0.6] ...
    OK [installed binary]
Moving rlang [1.0.6] into the cache ...
    OK [moved to cache in 0.63 milliseconds]
Installing htmltools [0.5.4] ...
    OK [installed binary]
Moving htmltools [0.5.4] into the cache ...
    OK [moved to cache in 0.47 milliseconds]
Installing integral [1.0.1.9001] ...
    OK [built from source]
Moving integral [1.0.1.9001] into the cache ...
    OK [moved to cache in 0.55 milliseconds]
> renv::snapshot()
The following package(s) will be updated in the lockfile:

# CRAN ===============================
- htmltools   [0.5.3 -> 0.5.4]
- rlang       [1.0.5 -> 1.0.6]

# GitHub =============================
- integral    [1.0.1.9001 -> IntegralEnvision/integral@main]

Do you want to proceed? [y/N]: y
* Lockfile written to '~/Github/cstories-app/beta/renv.lock'.

bbest commented 1 year ago

Next R package problem with curl in GH Action log Merge branch 'main' of https://github.com/cstories-app/beta · cstories-app/beta@7d6e8fd:

Installing curl [4.3.3] ...
Error: Error: install of package 'curl' failed [error code 1]
    FAILED
Traceback (most recent calls last):
12: renv::restore()
11: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
10: renv_install_impl(records)
 9: renv_install_staged(records)
 8: renv_install_default(records)
 7: handler(package, renv_install_package(record))
 6: renv_install_package(record)
 5: withCallingHandlers(renv_install_package_impl(record), error = function(e) {
        vwritef("\tFAILED")
        writef(e$output)
    })
 4: renv_install_package_impl(record)
 3: r_cmd_install(package, path)
 2: r_exec_error(package, output, "install", status)
 1: stop(error)
Error installing package 'curl':
================================

* installing *source* package ‘curl’ ...
** package ‘curl’ successfully unpacked and MD5 sums checked
** using staged installation
Package libcurl was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcurl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libcurl' found
Package libcurl was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcurl.pc'
to the PKG_CONFIG_PATH environment variable
No package 'libcurl' found
Using PKG_CFLAGS=
Using PKG_LIBS=-lcurl
------------------------- ANTICONF ERROR ---------------------------
Configuration failed because libcurl was not found. Try installing:
 * deb: libcurl4-openssl-dev (Debian, Ubuntu, etc)
 * rpm: libcurl-devel (Fedora, CentOS, RHEL)
If libcurl is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a libcurl.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
--------------------------------------------------------------------
ERROR: configuration failed for package ‘curl’
* removing ‘/home/runner/work/beta/beta/renv/staging/1/curl’
Execution halted
Error: Process completed with exit code 1.

bbest commented 1 year ago

fixed with commit:

Install system dependencies (for R package curl) · cstories-app/beta@f8597a0

per:

libcurl not included in ubuntu-latest software · Issue #37 · actions/runner-images

bbest commented 1 year ago

Next, probably need to add news API token as Github repo secret

bbest commented 1 year ago

References:

bbest commented 1 year ago

Next R package textshaping dependency error in log Install system dependencies (for R package curl) · cstories-app/beta@f8597a0

Installing textshaping [0.3.6] ...
    FAILED
Error: Error: install of package 'textshaping' failed [error code 1]
Traceback (most recent calls last):
12: renv::restore()
11: renv_restore_run_actions(project, diff, current, lockfile, rebuild)
10: renv_install_impl(records)
 9: renv_install_staged(records)
 8: renv_install_default(records)
 7: handler(package, renv_install_package(record))
 6: renv_install_package(record)
 5: withCallingHandlers(renv_install_package_impl(record), error = function(e) {
        vwritef("\tFAILED")
        writef(e$output)
    })
 4: renv_install_package_impl(record)
 3: r_cmd_install(package, path)
 2: r_exec_error(package, output, "install", status)
 1: stop(error)
Error installing package 'textshaping':
=======================================

* installing *source* package ‘textshaping’ ...
** package ‘textshaping’ successfully unpacked and MD5 sums checked
** using staged installation
Package harfbuzz was not found in the pkg-config search path.
Perhaps you should add the directory containing `harfbuzz.pc'
to the PKG_CONFIG_PATH environment variable
No package 'harfbuzz' found
Package fribidi was not found in the pkg-config search path.
Perhaps you should add the directory containing `fribidi.pc'
to the PKG_CONFIG_PATH environment variable
No package 'fribidi' found
Using PKG_CFLAGS=
Using PKG_LIBS=-lfreetype -lharfbuzz -lfribidi -lpng
--------------------------- [ANTICONF] --------------------------------
Configuration failed to find the harfbuzz freetype2 fribidi library. Try installing:
 * deb: libharfbuzz-dev libfribidi-dev (Debian, Ubuntu, etc)
 * rpm: harfbuzz-devel fribidi-devel (Fedora, EPEL)
 * csw: libharfbuzz_dev libfribidi_dev (Solaris)
 * brew: harfbuzz fribidi (OSX)
If harfbuzz freetype2 fribidi is already installed, check that 'pkg-config' is in your
PATH and PKG_CONFIG_PATH contains a harfbuzz freetype2 fribidi.pc file. If pkg-config
is unavailable you can set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
-------------------------- [ERROR MESSAGE] ---------------------------
<stdin>:1:10: fatal error: hb-ft.h: No such file or directory
compilation terminated.
--------------------------------------------------------------------
ERROR: configuration failed for package ‘textshaping’
* removing ‘/home/runner/work/beta/beta/renv/staging/1/textshaping’
Execution halted
Error: Process completed with exit code 1.

bbest commented 1 year ago

Yay, got past dependency hell. Now error running R scripts in log Update update_news.yml · cstories-app/beta@f71856c:

Run Rscript -e news_code/news_apis.R
  Rscript -e news_code/news_apis.R
  Rscript -e news_code/news_rss.R
  shell: /usr/bin/bash -e {0}
  env:
    BUNDLE_EXT: linux-amd6[4](https://github.com/cstories-app/beta/actions/runs/3954460846/jobs/6771834947#step:8:4).deb
    R_LIBS_USER: /home/runner/work/_temp/Library
    TZ: UTC
    _R_CHECK_SYSTEM_CLOCK_: FALSE
    NOT_CRAN: true
    RENV_PATHS_ROOT: /home/runner/work/_temp/renv
    MSYS: winsymlinks:nativestrict
Error: Error: object 'news_code' not found
Execution halted
Error: Process completed with exit code 1.

bbest commented 1 year ago

Next problem with Run source("news_code/news_apis.R") in log fix Rscript syntax #4 · cstories-app/beta@5512824:

ℹ Searching "Humboldt" AND ("offshore wind" )...
Error in `map()`:
ℹ In index: 1.
Caused by error in `resp_abort()`:
! HTTP 401 Unauthorized.
Backtrace:
     ▆
  1. ├─base::source("news_code/news_apis.R")
  2. │ ├─base::withVisible(eval(ei, envir))
  3. │ └─base::eval(ei, envir)
  4. │   └─base::eval(ei, envir)
  5. ├─global update_newsapi_data(terms)
  6. │ └─... %>% bind_rows()
  7. ├─dplyr::bind_rows(.)
  8. │ └─rlang::list2(...)
  9. ├─purrr::map(...)
 10. │ └─purrr:::map_("list", .x, .f, ..., .progress = .progress)
 11. │   ├─purrr:::with_indexed_errors(...)
 12. │   │ └─base::withCallingHandlers(...)
 13. │   └─.f(.x[[i]], ...)
 14. │     └─... %>% resp_body_json()
 15. ├─httr2::resp_body_json(.)
 16. │ └─httr2:::check_response(resp)
 17. │   └─httr2:::is_response(req)
 18. └─httr2::req_perform(.)
 19.   └─httr2:::resp_abort(resp, error_body(req, resp))
 20.     └─rlang::abort(...)
Execution halted
Error: Process completed with exit code 1.

jzadra commented 1 year ago

I think that this is because it isn't getting the NEWS_API_KEY from the environment variable set in github secrets.

Also today I added some code that should speed up the renv loading by caching all the libraries, however I'm still running into the 401 error above so I haven't been able to complete the run and fully test it.

caching info: https://github.com/marketplace/actions/cache https://rstudio.github.io/renv/articles/ci.html

encrypted secretes in workflow: https://docs.github.com/en/actions/security-guides/encrypted-secrets

I'm out Friday, so won't get to this until Monday

bbest commented 1 year ago

Yay, adding the secret worked.

There's a duplicate of NEWSAPI_KEY in settings/secrets/actions:

Keeping Repository secrets, removing Environment secrets -- that "environment" is more like different profiles (per Using environments for deployment - GitHub Docs versus the operating system's environmental parameters set with env in the yaml:

https://github.com/cstories-app/beta/blob/c9df29ab0d50e61c761a97dc3b01e44164b8d2b6/.github/workflows/update_news.yml#L55-L59

bbest commented 1 year ago

Next (and hopefully last) error with the publishing step in log + NEWSAPI_KEY: ${{ secrets.NEWSAPI_KEY }} #4 · cstories-app/beta@c9df29a:

ERROR: No _publish.yml file available (_publish.yml specifying a destination required for non-interactive publish)

bbest commented 1 year ago

Folllowing instructions at Quarto - GitHub Pages to switch from branch main:/_site to gh-pages:/...

git checkout --orphan gh-pages
git reset --hard
git commit --allow-empty -m "Initialising gh-pages branch"
git push origin gh-pages

quarto publish gh-pages

? Update site at https://cstories.app? (Y/n) › Yes
From https://github.com/cstories-app/beta
 * branch            gh-pages   -> FETCH_HEAD
Rendering for publish:

[  1/124] chat/bot.qmd
[  2/124] chat/index.qmd
[  3/124] news.qmd
[  4/124] infographics/cetaceans.qmd
[  5/124] infographics/viewshed.qmd
Error in loadNamespace(x) : there is no package called ‘jsonlite’
Calls: .main ... loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted

R installation:
  Version: 4.2.1
  Path: /Library/Frameworks/R.framework/Resources
  LibPaths:
    - /Users/bbest/Github/cstories-app/beta/renv/library/R-4.2/aarch64-apple-darwin20
    - /Users/bbest/Github/cstories-app/beta/renv/sandbox/R-4.2/aarch64-apple-darwin20/84ba8b13
  rmarkdown: (None)

The rmarkdown package is not available in this R installation.
Install with install.packages("rmarkdown")

renv::install("jsonlite")

...
[  5/124] infographics/viewshed.qmd

processing file: viewshed.qmd
  |                                                                        |                                                                        |                                                                        |..........                                                            |  14%
  ordinary text without R code

  |                                                                        |....................                                                  |  29%
label: unnamed-chunk-1
Quitting from lines 76-91 (viewshed.qmd) 
Error in loadNamespace(x) : there is no package called 'librarian'
Calls: .main ... loadNamespace -> withRestarts -> withOneRestart -> doWithOneRestart
Execution halted

renv::install("librarian")

And proactively installing dependencies for news R scripts so can use the renv cache in GH Action:

renv::install("IntegralEnvision/integral")
renv::install(c(
  "cli", "crayon", "fs", "janitor", "httr2", 
  "lubridate", "rrapply", "stringr", "textclean", "tidyRSS", "tidyverse"))

...
create mode 100644 images/infographic/ship-sounds.png
 create mode 100644 images/infographic/turbine-sounds.porigin   https://github.com/cstories-app/beta.git (fetch)
origin  https://github.com/cstories-app/beta.git (push)
To https://github.com/cstories-app/beta.git
   06805d0..c6b7ea5  HEAD -> gh-pages

NOTE: GitHub Pages sites use caching so you might need to click the refresh
button within your web browser to see changes after deployment.

(|) Deploying gh-pages branch to website (this may take a few minutes)

Add packages from infographic/viewshed.qmd:

renv::install(c(
 "dplyr", "glue", "here", "leaflet", "readr"))
renv::snapshot()

bbest commented 1 year ago

New error with terra R package install, a dependency of leaflet for infographics/viewshed.qmd in log * renv::install(...), renv::snapshot() #4 · cstories-app/beta@4e42102:

* installing *source* package ‘terra’ ...
** package ‘terra’ successfully unpacked and MD5 sums checked
** using staged installation
configure: CC: gcc
configure: CXX: g++ -std=gnu++11
checking for gdal-config... no
no
configure: error: gdal-config not found or not executable.
ERROR: configuration failed for package ‘terra’

So adding system libs needed for terra gdal, proj...

bbest commented 1 year ago

Hallelujah! Success:

Update News · cstories-app/beta@fc5f258

Hopefully it'll run much faster tonight at midnight (timezone?) by restoring from renv R packages cache.

cstories-app / beta

+ GH Action to render news articles #4