DerwenAI / kglab

Graph Data Science: an abstraction layer in Python for building knowledge graphs, integrated with popular graph libraries – atop Pandas, NetworkX, RAPIDS, RDFlib, pySHACL, PyVis, morph-kgc, pslpython, pyarrow, etc.

https://derwen.ai/docs/kgl/

MIT License

581 stars 66 forks source link

CI improvements #189

Closed louisguitton closed 3 years ago

louisguitton commented 3 years ago

What

[x] Fix CI: the input device is not a TTY
[x] speed up docker build using github action docker layer caching
[x] fix CI for notebook 4

Description

The Github Actions chicken and egg problem means I didn't see this issue until later. For example this run https://github.com/DerwenAI/kglab/runs/2540865279 failed with

louisguitton commented 3 years ago

Good news: CI now works as expected 🎉

Bad news:

CI breaks for notebook 4 (I will fix this in this PR) (although I can't reproduce locally ... 🤔 )
the docker build is slow (2m56s on the clock for the last run) because the kglab image is pretty complex. We can speed this up using caching across different github action runs. I will add this in this PR as well.

Related feature idea: we can upload the kglab docker image straight to the github docker registry called packages as part of github actions (docs). This would mean that users that want to go through the tutorial don't even need to build the image locally etc... they would just need to do

docker pull ghcr.io/DerwenAI/kglab:latest

The docker-compose file would then point to that image instead of the local one 👀

louisguitton commented 3 years ago

the issue with the notebooks currently failing in CI comes from:

the fact that the notebooks are not self-contained and that they rely on each-other's output like tmp.ttl
the fact that the treon library is using a random order to run notebooks in 🤦

Although there are arguments for writing self-contained notebooks, given the fact that they are clearly names in sequential order, I should be able to find a tool to test them in that order ! so I will keep looking.

louisguitton commented 3 years ago

treon relies on the order from glob.glob which is arbitrary (see https://docs.python.org/3/library/glob.html#:~:text=results%20are%20returned%20in%20arbitrary%20order) I've switched to nbmake and dropped treon because there was no current option to sort results.

ceteri commented 3 years ago

If it ever helps, we could place copies of the files generated by earlier notebooks in the tutorial (e.g., tmp.ttl) into a public bucket somewhere?