The behavior of jenkins (running containers as a non-root user, and therefore being unable to modify the contents of the /app directory) is not well documented and it's not clear to developers that bqetl logic has this as a potential runtime constraint. The purpose of --sql-dir to specify an alternate writable location I think is mostly to support the Jenkins use case. There is a comment in https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/Jenkinsfile.bigquery.prod#L135 that explains the cp -R sql /tmp/sql line but it might make sense to document or otherwise modify bqetl CI to catch this in CI.
kik-kik
commented
1 year ago
There have been a couple of PRs that have broken automated views deploys due to permissions errors, most recently:
https://github.com/mozilla/bigquery-etl/issues/2212 https://github.com/mozilla/bigquery-etl/issues/2214 https://github.com/mozilla/bigquery-etl/pull/2238
The behavior of jenkins (running containers as a non-root user, and therefore being unable to modify the contents of the
/appdirectory) is not well documented and it's not clear to developers that bqetl logic has this as a potential runtime constraint. The purpose of--sql-dirto specify an alternate writable location I think is mostly to support the Jenkins use case. There is a comment in https://github.com/mozilla-services/cloudops-infra/blob/master/projects/data-shared/Jenkinsfile.bigquery.prod#L135 that explains thecp -R sql /tmp/sqlline but it might make sense to document or otherwise modify bqetl CI to catch this in CI.