Closed td928 closed 1 year ago
Hi Te! Thanks for flagging.
A couple notes for fixing
docker
command with published data library image to invoke library archive
commandtry...except
wrapper - which just wraps the "decode" call at the end of the subprocess
call as written. If this is fixed, jobs will succeed (albeit with maybe incorrect execution details). See fix in #403 docker
command, not the published github action. Inside docker container in CI, both CI
variable and git
command are unavailable, even if they're available in the environment where it's being called. Need to think a bit about use cases of someone calling data library via docker
command line call and how this should be handled. These failing jobs for devdb ideally are identified as CI jobs with links to github, but then they need to be explicitly passed in to docker somehow. Blanking this morning, off the top of my head, whether the published action could just be used at the end of a job rather than this docker command. Really, passing CI details is the main one - git is used as a relatively simple way to get a local username, but if manual vs CI is the more useful thing to knowAre you also specifically having issues using the published action? At least in the data library repo, those seem to be working fine on our end. Either way, if we fix the try...except issue
you'll be able to run. Other than that, if you're using docker from the command line you could pass the CI
variable in with your call to docker to run it right now I believe
@fvankrieken Thanks for the explanation that makes sense to me.Upon closer look it is true devdb is using it as a docker image not the CI. But in my case, I should be able to use your suggestions to use it as an action and sidestep this now. I will keep my eyes on the PR and if it made it into the release (docker and CI) and can test to let you know.
Gonna close this. Docker calls do work now even if execution details might be inaccurate if env variables aren't passed. But if you invoke with -e CI=$CI
the execution details will be accurate at least in that regard. Longer term should maybe move away from using git cli to get a username (or let users specify somehow), but it made sense in our dev environment with dev containers (and "vscode" user rather than the actual user's username).
Hey Data Engineering!
I've been trying to set up my version of data library on AWS s3 for a project I am working on. I see there are some new changes to data library and one of them is causing archive function when called from the latest release in Actions.
When something like this below run for development database for example:
It throws an error like below. This is after I added git installation to the Actions runner but still the error persists. I wonder if the new module added was tested with some of the existing data products and what adjustments if any you have to make to work with the new improvements.
I also did find at least one instance of the scheduled run in the now archived DevDB main branch see here. If you go to the geocode step and then to the
Archive to Data Library
step you would see the same error there and the geocoded data probably failed to push to s3 there.Just want to flag this for your attention and will let you know if I somehow find the answer for this.
Thanks!
All the best.
Te