Write errors to separate log file with suffix .err. The .log file shown in the UI was getting merged with stuff airflow was logging and the errors were getting truncated
Add script to trigger whole pipeline including steps that require whole session, container to run
fix bug in classifier thumbnail generation. It was using file produced by segmentation for ids, needs to use database values
Make sure datetimes are in UTC, since we need to compare timestamps in UTC with timestamps in the same timezone. On-prem records timestamps in local time
add support for custom dockerhub repo/python interpreter (needed for event detection)
This PR changes
is_debug
to work with a truncated ophys movie. I then ran the pipeline and fixed issues.is_debug
will truncate an ophys movie to 200 frames and write to disk, this is then used for all subsequent steps.err
. The.log
file shown in the UI was getting merged with stuff airflow was logging and the errors were getting truncatedrun_id
so it's easier to link up with the trigger run https://alleninstitute-brainscience.atlassian.net/jira/software/projects/PSB/boards/48/backlog?selectedIssue=PSB-139submit_job
ifwait_for_job_to_finish
detects that it failed