search

qurator-spk / dinglehopper

An OCR evaluation tool
Apache License 2.0
59 stars 13 forks source link

Add travis pip caching #42

Closed b2m closed 3 years ago

b2m commented 3 years ago

The main reason for the long build time on Travis is the compilation of dependencies for Python 3.5 and Python 3.9.

By using pip caching we reduce the runtime for subsequent builds per branch by about 40 minutes.

We could speed this up further by splitting unit and integration tests and/or using anaconda but IMHO this is not yet necessary.

mikegerber commented 3 years ago

The main reason for the long build time on Travis is the compilation of dependencies for Python 3.5 and Python 3.9. By using pip caching we reduce the runtime for subsequent builds per branch by about 40 minutes.

Great! I somehow overlooked this possibility because I do a lot in Docker in other projects nowadays where this is not so easily available.

We could speed this up further by splitting unit and integration tests and/or using anaconda but IMHO this is not yet necessary.

I'm not sure if I understand this correctly, but: it's also important that we use PyPi, not anaconda packaging.

b2m commented 3 years ago

I'm not sure if I understand this correctly, but: it's also important that we use PyPi, not anaconda packaging.

Reduced and simplified to the use case in CI-Pipelines: conda installs binaries whereas pip compiles dependencies from source. This also means that pip needs the corresponding build environment whereas conda packages can be used as is.

mikegerber commented 3 years ago

I want to test against the environment the users use, which is PyPi.

pip also uses binary wheels from PyPi - when they are available. For example the build for Python 3.8 installs the wheel for OpenCV: https://travis-ci.org/github/qurator-spk/dinglehopper/jobs/740164235. There's no wheel (yet) for OpenCV on Python 3.9 yet: https://travis-ci.org/github/qurator-spk/dinglehopper/jobs/740164236. → Longer build time for Python 3.9. Python 3.5 is another beast, because it is EOL - I just like to still test against it because people like @kba are stuck with it still.

b2m commented 3 years ago

conda installs binaries whereas pip compiles dependencies from source.

pip also uses binary wheels from PyPi - when they are available

Yes I simplified there is more to it... like compiling other language dependencies, availability of binary wheels... But I guess the build time is not an issue in this project and caching already helps for branches under development.