Open pbchase opened 1 year ago
Would this be an option? This is an example with render_report.R
run locally. Output attached.
docker run --rm --env-file .env \
-v $(pwd):/home/rocker/redcapcustodian \
redcapcustodian /bin/bash -c "Rscript report/render_report.R \
report/sample_report.Rmd > /home/rocker/redcapcustodian/$(date +%Y%m%d-%H%M%S)-sample_report.log 2>&1"
Separate goal of switching out docker run ...
crons to docker compose:
docker-compose.yaml
version: '3.8'
services:
redcapcustodian:
container_name: redcapcustodian
env_file:
- /path/to/default.env
environment:
- FOO=${FOO}
image: redcapcustodian
restart: "no"
volumes:
- /path/to/host/dir:/mnt/homedir
new cron format:
* * * * * root docker compose -f /path/to/redcapcustodian/docker-compose.yaml --env-file /path/to/override.env run --rm redcapcustodian Rscript report/sample_report.Rmd
Note that override.env
does not seem to actually end up replacing variables sourced from Sys.getenv
if they aren't also specified in the "environment" section; however, using -e KEY=var
in the docker compose string after run
does work (--env-file
cannot be placed after run
). This is not the case for docker compose up
.
Is this proceeding with the docker compose solution?
On 2024-04-30 @ljwoodley, @pbc, and @saipavan10-git discussed using https://airflow.apache.org/docs/apache-airflow/stable/index.html to manage our redcapcustodian tasks. @ljwoodley will do a proof of concept to run one workflow under airflow and produce a log file.
We want to dump all of the R log to test files to ensure a complete run/crash log. We will do these things:
SINK_DIR
from env file that sets the root folder of the logged files.SINK_DIR
in the example env file setting the value to~/rcc_sink_log
sink()
toinit_etl()
so that existing calls to init_etl will get the new functionality without modification~/rcc_sink_log
for the sink logging hierarchy. This will place folders named 2023, 2024, etc. in~/rcc_sink_log
.$SINK_DIR/$YEAR/filename
. An example of this with full defaults for the scriptfoo_hotness
run on 2023-03-14 would be~/rcc_sink_log/2023/20230314-123456-foo_hotness.log
.Note, the discussion about this feature has underscored the value of using docker-compose and yaml files to run these containers. That will be addressed in another issue.