search

compSPI / ioSPI

I/O and Data Visualization
MIT License
4 stars 7 forks source link

GH "Linting and Testing" workflow is slow #4

Closed fredericpoitevin closed 2 years ago

fredericpoitevin commented 3 years ago

The "install dependencies" step takes a long time (>30mn).

Several non-necessarily mutually exclusive options need to be explored:

fredericpoitevin commented 3 years ago

a possible fix: https://github.com/actions/cache (see https://docs.github.com/en/actions/guides/caching-dependencies-to-speed-up-workflows)

fredericpoitevin commented 3 years ago

a possible example for conda: https://github.com/actions/cache/issues/355

ninamiolane commented 3 years ago

Did the github action cache solve it?

Putting compSPI on conda and then downloading this might not fix it (since compSPI would have to install its dependencies anyway?).

Container approach could also be an option indeed.

fredericpoitevin commented 3 years ago

Re: github action cache. At the moment we hit the following issue: https://github.com/actions/cache/issues/144 I'll look into it tomorrow.

And... container it is! (will be soon)

fredericpoitevin commented 3 years ago

A possible hint at what is going wrong.

The first of the tree workflow jobs gives this error:

Post job cleanup. /bin/tar -cz -f /home/runner/work/_temp/fa635f5d-cae1-4b20-b60e-ab45451efb83/cache.tgz -C /home/runner/work/ioSPI/ioSPI/$CONDA . /bin/tar: /home/runner/work/ioSPI/ioSPI/$CONDA: Cannot open: No such file or directory /bin/tar: Error is not recoverable: exiting now Warning: Tar failed with error: The process '/bin/tar' failed with exit code 2

The other two give this error:

Post job cleanup. Warning: Cache already exists. Scope: refs/heads/master, Key: Linux-conda-8f071096bd72a27b0cde7ff4ecc966e886d95c2fef88c93ea7f6c5c6b34e151f, Version: (null)

ninamiolane commented 3 years ago

Ah, this error looks nasty :(

But why don't we get this error on the build.yml of reduceSPI? We also have the conda cache there: https://github.com/compSPI/reduceSPI/blob/130a274d9bfe25186bf4ded06def1d7fb7596ee5/.github/workflows/build.yml#L35

fredericpoitevin commented 3 years ago

https://github.com/compSPI/reduceSPI/blob/130a274d9bfe25186bf4ded06def1d7fb7596ee5/.github/workflows/build.yml#L35

I do see the same error for reduceSPI, although it appears on the second job build (ubuntu-18.04, 3.8, tests.tests_reduceSPI) this time: https://github.com/compSPI/reduceSPI/runs/2920366016?check_suite_focus=true

I'll give it a quick look again today, but we might want to drop conda cache eventually.

geoffwoollard commented 2 years ago

Closed because out of date. Many changes since June 2021