Closed robinnewhouse closed 2 years ago
Thanks for the Issue @robinnewhouse. Can you also give us the Python runtime version that you're using in your virtual environment? So the output of
$ python --version --version
I guess I very rarely use my local python, and I admit, I typically use conda instead of venv. In this instance, I'm using neither and my python version is Python 2.7.16
. I'm upgrading to 3 right away ;)
Hm, I'm assuming from the diff between v0.1.0...v0.1.1
that this is related to a packtivity
change given
$ git diff v0.1.0 v0.1.1 -- setup.py
diff --git a/setup.py b/setup.py
index 0e24dc2..7e18883 100644
--- a/setup.py
+++ b/setup.py
@@ -10,7 +10,7 @@ with open(path.join(this_directory, 'README.md'), encoding='utf-8') as readme_md
setup(
name = 'recast-atlas',
- version = '0.1.0',
+ version = '0.1.1',
description = 'RECAST for ATLAS at the LHC',
url = '',
author = 'Lukas Heinrich',
@@ -33,7 +33,7 @@ setup(
'pydotplus==2.0.2',
'adage==0.10.1',
'yadage-schemas==0.10.6',
- 'packtivity==0.14.21',
+ 'packtivity==0.14.23',
'yadage[viz]==0.20.1',
],
'kubernetes': [
But I haven't looked at this really at all and I'm not a packtivity
expert (or even novice). So maybe @lukasheinrich can say if that's a reasonable thing at all or not.
In this instance, I'm using neither and my python version is
Python 2.7.16
.
Okay thanks. I ask only so that I can try to replicate your working environment when debugging with you.
I guess I very rarely use my local python, and I admit, I typically use conda instead of venv
That's fine. I'm using "virtual environment" here to mean it in the most broad sense (so a venv env and a Conda env both count)
I'm upgrading to 3 right away ;)
:snake: recast-atlas
has CI covering Python 3.6 - Python 3.8 (and should have Python 3.9 and Python 3.10 support soon) so I think you should be good to go there as long as you're in the currently supported range.
I can reproduce this in Python 3.8.11 environments, but given that I installed recast-atlas
from PyPI in two different virtual environments (one recast-atlas
v0.1.0
and one recast-atlas
v0.1.1
) and
$ diff freeze-recast-v0.1.0.txt freeze-recast-v0.1.1.txt
10c10
< recast-atlas==0.1.0
---
> recast-atlas==0.1.1
I'm pretty sure the difference is coming from the Docker images (as the default backend is the Docker backend). There's very minor changes in the Dockerfile itself
$ git diff v0.1.0 v0.1.1 -- docker/Dockerfile
diff --git a/docker/Dockerfile b/docker/Dockerfile
index b11a1ad..dc69998 100644
--- a/docker/Dockerfile
+++ b/docker/Dockerfile
@@ -1,12 +1,13 @@
-FROM docker:19.03.0
+FROM docker:20.10.5
RUN apk add py-pip automake autoconf libtool \
- python-dev musl-dev libffi-dev tree \
- python-dev musl-dev libffi-dev gcc \
+ python3 python3-dev musl-dev libffi-dev tree \
+ python3-dev musl-dev libffi-dev gcc \
autoconf curl gcc ipset ipset-dev iptables iptables-dev libnfnetlink libnfnetlink-dev libnl3 libnl3-dev make musl-dev openssl openssl-dev \
- jq util-linux font-bitstream-type1 build-base graphviz-dev imagemagick graphviz
+ jq util-linux font-bitstream-type1 build-base graphviz-dev imagemagick graphviz gcc musl-dev python3-dev libffi-dev openssl-dev cargo
+
WORKDIR /code
COPY . /code
-RUN pip install -e .[local,kubernetes]
+RUN pip3 install -e .[local,kubernetes]
ENV PACKTIVITY_DOCKER_CMD_MOD "-u root"
ENV PACKTIVITY_CVMFS_LOCATION /shared-mounts/cvmfs
ENV PACKTIVITY_CVMFS_PROPAGATION rslave
but we see (that in addition to a Python 2.7 to Python 3.7 change) there's a lot of differences in Python packages.
$ docker run --rm recast/recastatlas:v0.1.0 /bin/ash -c 'python -m pip freeze' &> freeze_docker_v0.1.0.txt
$ docker run --rm recast/recastatlas:v0.1.1 /bin/ash -c 'python3 -m pip freeze' &> freeze_docker_v0.1.1.txt
$ diff -y freeze_docker_v0.1.0.txt freeze_docker_v0.1.1.txt | wc -l
66
The pip freeze
s are attached here:
and their diff
I think we're going to need @lukasheinrich's advice here as I'm not really sure what the graph dependencies are of yadage
that is actually doing the drawing here, though it apparently isn't the fault of networkx
or pydot2
$ diff freeze_docker_v0.1.0.txt freeze_docker_v0.1.1.txt | grep 'networkx\|pydot' # Returns nothing
hm as a suggestion can you run somethting simple through graphviz
e.g.
strict digraph {
A -> B;
}
dot -Tpdf test.pdf
? That might help debug it
mayb it's som libgraphviz
related issue
That might help debug it
Indeed.
$ docker pull recast/recastatlas:v0.1.8
$ docker run --rm -ti -v $PWD:/work recast/recastatlas:v0.1.8
/work # cat test.gv
strict digraph {
A -> B;
}
/work # dot -Tpdf test.gv -o test.pdf
/work # dot -Tpng test.gv -o test.png
produces this test.png
I have a branch (docker/update-dockerfile
) that is trying to get a python:3.8-slim-bullseye
based image working which is able to render this correctly
$ docker run --rm -ti -v $PWD:/work recast/recastatlas:local-debug
root@d5e1c4970e41:/work# cat /etc/os-release
PRETTY_NAME="Debian GNU/Linux 11 (bullseye)"
NAME="Debian GNU/Linux"
VERSION_ID="11"
VERSION="11 (bullseye)"
VERSION_CODENAME=bullseye
ID=debian
HOME_URL="https://www.debian.org/"
SUPPORT_URL="https://www.debian.org/support"
BUG_REPORT_URL="https://bugs.debian.org/"
root@d5e1c4970e41:/work# cat test.gv
strict digraph {
A -> B;
}
root@d5e1c4970e41:/work# dot -Tpdf test.gv -o test.pdf
root@d5e1c4970e41:/work# dot -Tpng test.gv -o test.png
root@d5e1c4970e41:/work#
but I need to figure out how to get docker
working in this image for it to be useful.
(Related: Issue #50)
this is what yadag runs, so if dot
works yadag viz should also work:
if
dot
works yadag viz should also work:
Cool. This is good to know so we only have 1 thing to debug.
Here's a minimal working Dockerfile
FROM docker:20.10.5
RUN apk add --update && \
apk add --no-cache \
graphviz \
ttf-freefont
WORKDIR /work
that when built
docker build -f Dockerfile -t graphviz:debug-local .
is able to produce the minimal example
$ docker run --rm -v $PWD:/work graphviz:debug-local /bin/ash -c 'dot -Tpng test.gv -o test.png'
So I think ttf-freefont
is the only thing missing from the current Dockerfile
.
Yeah, it is just ttf-freefont
. With
FROM recast/recastatlas:v0.1.8
RUN apk add --update && \
apk add --no-cache \
graphviz \
ttf-freefont
I can get
#!/bin/bash
docker pull recast/recastatlas:v0.1.8
docker tag recast/recastatlas:v0.1.8 recast/recastatlas:v0.1.8-build-base
docker build -f Dockerfile -t recast/recastatlas:v0.1.8 .
docker run --rm -v $PWD:/work recast/recastatlas:v0.1.8 /bin/ash -c 'dot -Tpng test.gv -o test.png'
to produce the file as expected and then, as this new images is tagged to match the release pattern recast
is looking for
$ recast run examples/rome --tag issue-67
produces the following recast-issue-67/_yadage/yadage_workflow_instance.png
:+1:
I'll open a PR to fix this, and we can backport this to other images too.
@robinnewhouse I'll ping you here once I have the change from PR #69 backported to the recast/recastatlas:v0.1.8
Docker image.
Wow, you guys are fast! Thanks for the fix :)
@robinnewhouse I've backported the fix to recast/recastatlas:v0.1.8
so if you repull you should be good to go. :+1:
$ docker pull recast/recastatlas:v0.1.8
$ python -m pip list | grep recast-atlas
recast-atlas 0.1.8
$ recast run examples/rome --tag rome-example
produces this recast-rome-example/_yadage/yadage_workflow_instance.png
Thanks for opening up a great Issue and if you come across other bugs please do so as well — we really appreciate it!
In versions
0.1.1
through0.1.8
(latest), when running recast workflows, the workflow visualization lacks titles for each of the steps and artefacts. In place, square placeholder glyphs are used. It appears that all workflow steps are in the correct places with the expected layout, but there are no titles. If I downgrade to recast 0.1.0, the titles in the workflow return to normal.This issue appears in the .gif, .png, and .pdf versions of the visualization.
I am running macOS Big Sur 11.6 (20G165)
This can be minimally reproduced with the following:
which produces
_yadage/yadage_workflow_instance.png
ofcompared to