Closed tloubrieu-jpl closed 1 year ago
@tloubrieu-jpl please accept my apologies. I read the title:
CICD: build does not work because of dependency conflicts
and fixed that. But I didn't notice the acceptance criterion:
I expected the github actions to run successfully in less than 30 minutes.
I shall get to work on that.
Okay, a lot to unpack here. First, even if "the project builts properly in a new virtual environment", that's no guarantee it'll work in GitHub Actions. The two environments are vastly different: different versions of Python 3.9 and package dependencies, different operating systems (macOS vs Linux vs Windows), and now thanks to the rise of Apple Silicon, even different CPUs.
What we strive to do is compromise between the CI/CD environment and developers' desktops. That's what github-actions-base
really encapsulates: a compromise between the desired development context and what GitHub Actions can reasonably provide for free.
So, executive summary: this ticket cannot be resolved.
Read on for the gory details.
The doi-service
has three problematic dependencies:
lxml >= 4.5
numpy >= 1.18
pandas ~= 1.4.4
These all slow down the build because these packages in particular take a long time to construct. By changing these dependencies to:
lxml == 4.6.3
numpy == 1.21.2
we satisfy the the previous setup.cfg
requirements while taking advantage of these packages that are present in github-actions-base
.
It also has two other speed-bumps:
tox
without a version pinvirtualenv
as an implied dependency but also without a version pinAdding these pins:
tox == 3.28.0
virtualenv == 20.8.1
we can further speed up the build.
That brings us to pandas
. According to this issue, the doi-service
needs pandas
in the 1.4.4+ version range. The github-actions-base
provides pandas == 1.3.4
. We've got two ways to speed this up:
github-actions-base
to pandas == 1.4.4
(the latest in the 1.4.x range). However, this would require us to check every other Python package within the PDS-EN that's using Pandas to see if this version is compatible. Maybe this is okay, though? I haven't looked extensively, but it seems maybe only doi-service
is using pandas
right now.doi-service
to pandas == 1.3.4
. I ran pytest
using both pandas-1.4.4
and pandas-1.3.4
and got the same number of failures, errors, warnings, and passes, so maybe this is okay.I'd like @jimmie to chime in as he originally submitted #369 and @alexdunnjpl who provided the fix to it. What's important about 1.4? Is 1.3 okay?
Once we address the above, there are lingering issues in the main
branch even today. I guess this is what we get for not having a routine Roundup running here!
Specifically, there were lint errors in these files: update.py
, input_util.py
, config_parser.py
, contributors_util_test.py
, doi_validator_tes.py
.
And in util.py
there was
from collections import Iterable
which doesn't exist; it should be
from typing import Iterable
After fixing all these, we can finally turn it over to the Roundup.
With all this in place, the free GitHub Actions runner can complete the Roundup in about 45 minutes, which exceeds the acceptance criterion for this ticket.
We could put even more of doi-service
dependencies into the github-actions-base
to try and speed things up. Aside from lxml
, numpy
, and pandas
, doi-service
has 32 other dependencies, from flask==1.1.2
to jinja2==3.0.1
(a troublemaker in the past) to requests>=2.23
and more. Putting those 32 dependencies into github-actions-base
would speed up the build by perhaps 7β9 minutes, which puts the total build time to 36β38 minutes, which still doesn't satisfy the acceptance criterion.
However, doing so means it freezes the versions of 36 packages for the entire NASA-PDS organization. Other packages could still supply newer versions, such as requests~=2.26
, and that's okayβthey'd lose the speed bonus of having a pre-cached package. Yet I'm still hesitant to put so many more sundry things into github-actions-base
. Something just feels off about it.
The other time sink is the sheer size of the doi-service
: it's 8600+ lines of Python and has an enormous test suite. Tox'ing, lint'ing, mypy-type-checking, flake8'ing, white-space-checking, shebanging, debug statement detection, black'ifying, etc., all takes a lot of timeβand that's before we even get to the unit tests.
The only way I've found so far is to use a self-hosted GitHub Actions runner. Using such a runner I believe I can get 10 minutes or less.
Sorry, @tloubrieu-jpl: cannot fix without self-hosted runner.
Let's downgrade pandas to 1.3.4
Checked for duplicates
Yes - I've already checked
π Describe the bug
For a few week the github action do not work anymore because of dependency conflict issues.
However, the project builts properly in a new virtual environment.
π΅οΈ Expected behavior
I expected the github actions to run successfully in less than 30 minutes.
π To Reproduce
No response
π₯ Environment Info
No response
π Version of Software Used
No response
π©Ί Test Data / Additional context
No response
π¦ Related requirements
No response
βοΈ Engineering Details
This ticket and https://github.com/NASA-PDS/roundup-action/issues/109 are both a result of not using a new virtual environment anymore in the github actions.