`get_execution_details` fails in Github Actions

[td928]

Hey Data Engineering!

I've been trying to set up my version of data library on AWS s3 for a project I am working on. I see there are some new changes to data library and one of them is causing archive function when called from the latest release in Actions.

When something like this below run for development database for example:

./devdb.sh library_archive_version dob_geocode_results 20230709

It throws an error like below. This is after I added git installation to the Actions runner but still the error persists. I wonder if the new module added was tested with some of the existing data products and what adjustments if any you have to make to work with the new improvements.

❱ 73 │   │   git_user = try_func(subprocess.run(["git", "config", "user.name │
│   74 │   │   return {                                                        │
│   75 │   │   │   "type": "manual",                                           │
│   76 │   │   │   "user": git_user,                                           │
│                                                                              │
│ /usr/lib/python3.10/subprocess.py:501 in run                                 │
│                                                                              │
│    498 │   │   kwargs['stdout'] = PIPE                                       │
│    499 │   │   kwargs['stderr'] = PIPE                                       │
│    500 │                                                                     │
│ ❱  501 │   with Popen(*popenargs, **kwargs) as process:                      │
│    502 │   │   try:                                                          │
│    503 │   │   │   stdout, stderr = process.communicate(input, timeout=timeo │
│    504 │   │   except TimeoutExpired as exc:                                 │
│                                                                              │
│ /usr/lib/python3.10/subprocess.py:969 in __init__                            │
│                                                                              │
│    966 │   │   │   │   │   self.stderr = io.TextIOWrapper(self.stderr,       │
│    967 │   │   │   │   │   │   │   encoding=encoding, errors=errors)         │
│    968 │   │   │                                                             │
│ ❱  969 │   │   │   self._execute_child(args, executable, preexec_fn, close_f │
│    970 │   │   │   │   │   │   │   │   pass_fds, cwd, env,                   │
│    971 │   │   │   │   │   │   │   │   startupinfo, creationflags, shell,    │
│    972 │   │   │   │   │   │   │   │   p2cread, p2cwrite,                    │
│                                                                              │
│ /usr/lib/python3.10/subprocess.py:1845 in _execute_child                     │
│                                                                              │
│   1842 │   │   │   │   │   │   err_filename = orig_executable                │
│   1843 │   │   │   │   │   if errno_num != 0:                                │
│   1844 │   │   │   │   │   │   err_msg = os.strerror(errno_num)              │
│ ❱ 1845 │   │   │   │   │   raise child_exception_type(errno_num, err_msg, er │
│   1846 │   │   │   │   raise child_exception_type(err_msg)                   │
╰──────────────────────────────────────────────────────────────────────────────╯
FileNotFoundError: [Errno 2] No such file or directory: 'git'

I also did find at least one instance of the scheduled run in the now archived DevDB main branch see here. If you go to the geocode step and then to the Archive to Data Library step you would see the same error there and the geocoded data probably failed to push to s3 there.

Just want to flag this for your attention and will let you know if I somehow find the answer for this.

Thanks!

All the best.

Te

[fvankrieken]

Hi Te! Thanks for flagging.

A couple notes for fixing

• this is specifically for using docker command with published data library image to invoke library archive command
• biggest issue is the misuse (my misuse) of a try...except wrapper - which just wraps the "decode" call at the end of the subprocess call as written. If this is fixed, jobs will succeed (albeit with maybe incorrect execution details). See fix in #403
• as mentioned above, this is specifically when using docker command, not the published github action. Inside docker container in CI, both CI variable and git command are unavailable, even if they're available in the environment where it's being called. Need to think a bit about use cases of someone calling data library via docker command line call and how this should be handled. These failing jobs for devdb ideally are identified as CI jobs with links to github, but then they need to be explicitly passed in to docker somehow. Blanking this morning, off the top of my head, whether the published action could just be used at the end of a job rather than this docker command. Really, passing CI details is the main one - git is used as a relatively simple way to get a local username, but if manual vs CI is the more useful thing to know

[fvankrieken]

Are you also specifically having issues using the published action? At least in the data library repo, those seem to be working fine on our end. Either way, if we fix the try...except issue you'll be able to run. Other than that, if you're using docker from the command line you could pass the CI variable in with your call to docker to run it right now I believe

[td928]

@fvankrieken Thanks for the explanation that makes sense to me.Upon closer look it is true devdb is using it as a docker image not the CI. But in my case, I should be able to use your suggestions to use it as an action and sidestep this now. I will keep my eyes on the PR and if it made it into the release (docker and CI) and can test to let you know.

[fvankrieken]

Gonna close this. Docker calls do work now even if execution details might be inaccurate if env variables aren't passed. But if you invoke with -e CI=$CI the execution details will be accurate at least in that regard. Longer term should maybe move away from using git cli to get a username (or let users specify somehow), but it made sense in our dev environment with dev containers (and "vscode" user rather than the actual user's username).