Closed dm-tran closed 7 years ago
nchammas
commented
7 years ago Thanks for the detailed report with references to possible causes. This will make it much easier to investigate! I presume you don't see this with the pip-installed version of Flintrock, correct? (It's OK if you haven't tried that setup.)
I will look at pinning Cryptography to a newer version and send you a test build to try out.
(Note to self, #158 is also related to our PyInstaller packaging.)
nchammas
commented
7 years ago @dm-tran - If you dig in to the Flintrock standalone package can you tell exactly what version of Cryptography is in there? It would be good to know if the version is < 1.6 or not. I can check on this later as well.
dm-tran
commented
7 years ago @nchammas Thanks for your reactivity.
I haven't tried with the pip-installed version. Actually, I'm using a fork of Flintrock that is compatible with my company's VPC.
I'm not familiar with pyinstaller. Where can I see the version of cryptography in the standalone package ?
nchammas
commented
7 years ago Hmm, I'm not sure. Don't worry about it if you can't see where. (Part of the issue is that the Linux standalone package for Flintrock is built on Travis CI, so the version of Cryptography selected there may technically be different from the one on my Mac workstation since I haven't pinned it in setup.py.)
nchammas
commented
7 years ago @dm-tran - I created a standalone Flintrock package on Ubuntu with Cryptography 1.7.1.
Flintrock-0.8.0.dev0-standalone-Linux-x86_64.zip
Can you test it out and let me know if it resolves the issue you were seeing? I know you said you couldn't reproduce it consistently, but I'm assuming with enough tries you'll know if the issue has been addressed or not.
dm-tran
commented
7 years ago @nchammas: sorry for the late answer, I was on holidays. I will try this standalone package and will let you know if I can reproduce the issue.
FYI, I have tried downgrading paramiko from version 2.0.2 to version 1.15.4: I haven't run into the "ImportError" since this downgrade.
nchammas
commented
7 years ago How did you downgrade if you were using the standalone package?
dm-tran
commented
7 years ago I reverted commit 65160b4b8d1a603d7e9f7a9ca150be82f09fd792 and used a docker container based on image "python:3.5.2": docker run -t -i -v /absolutePath/to/flintrockRepository:/flintrock python:3.5.2 /bin/bash
I ran pip3 install -r requirements/maintainer.pip and python generate-standalone-package.py to generate a standalone package.
dm-tran
commented
7 years ago I have tested your stand alone package that was built with Cryptography 1.7.1, and I haven't reproduced the issue.
Here is how I tested it, from my Mac workstation :
docker run -t -i -v /absolutePathTo/Flintrock-0.8.0.dev0-standalone-Linux-x86_64:/flintrock debian:jessie /bin/bash#!/bin/bash
for i in {1..200}
do
echo "Iteration $i"
/flintrock/flintrock --config myConfig.yaml run-command my-cluster 'touch /tmp/test'
sleep 1
done
Perados
commented
6 years ago @nchammas Today I had the exact same error : ImportError: cannot import name 'backend'. What I found is that when I build using python generate-standalone-package.py and PyInstaller==3.2, I get missing module named cryptography.hazmat.backends.openssl.backend.backend - imported by cryptography.hazmat.backends.openssl.backend, cryptography.hazmat.primitives.ciphers.aead, cryptography.hazmat.primitives.asymmetric.x25519, cryptography.hazmat.backends.openssl, cryptography.hazmat.backends in dist/build/flintrock/warnflintrock.txt (I debugged thanks to this). I do not get this warning with PyInstaller==3.3.
Our fork has been branched from your v0.9.0 tag where PyInstaller == 3.2, do you have the same problem when building that standalone version ? Or maybe I am doing something wrong during the build, maybe the python version ? I am building inside a docker container with Python 3.5.2.
Thank you in advance !
nchammas
commented
6 years ago I am not sure why you are seeing the error you are when building at v0.9.0. There may be some clues in the intervening commits where I bump the PyInstaller version or make changes to setup.py.
What version of cryptography are you pulling when you try to build a package at the forked v0.9.0?
Perados
commented
6 years ago @nchammas There is no error while building, it is just a warning in dist/build/flintrock/warnflintrock.txt, but it explains the error at runtime. Here is my pip freeze for the build:
asn1crypto==0.24.0
attrs==17.4.0
boto3==1.4.4
botocore==1.5.10
certifi==2018.4.16
cffi==1.11.5
chardet==3.0.4
click==6.7
coverage==4.5.1
cryptography==2.2.2
docutils==0.14
flake8==3.3.0
-e git+git@github.com:adotmob/flintrock.git@59e84b60a636e03532103be67549ae38c0dd9df6#egg=Flintrock
idna==2.6
jmespath==0.9.3
mccabe==0.6.1
more-itertools==4.1.0
paramiko==2.1.1
pkginfo==1.4.2
pluggy==0.6.0
py==1.5.3
pyasn1==0.4.2
pycodestyle==2.3.1
pycparser==2.18
pyflakes==1.5.0
PyInstaller==3.2
pypandoc==1.4
pytest==3.5.1
pytest-cov==2.5.1
python-dateutil==2.7.2
PyYAML==3.12
requests==2.18.4
requests-toolbelt==0.8.0
s3transfer==0.1.13
six==1.11.0
tqdm==4.23.2
twine==1.11.0
urllib3==1.22How do you usually build the standalone package ? I do pip install -r requirements/maintainer.pip and then python generate-standalone-package.py, on a Python 3.5.2 clean Docker container.
I found the line that fixes this in PyInstaller 3.3: https://github.com/pyinstaller/pyinstaller/blob/v3.3/PyInstaller/hooks/hook-cryptography.py#L26
Which is not there for PyInstaller 3.2:
https://github.com/pyinstaller/pyinstaller/blob/v3.2/PyInstaller/hooks/hook-cryptography.py
If the official release for Flintrock v0.9.0 has been built using PyInstaller 3.2, then it might contain the bug.
What do you think?
nchammas
commented
6 years ago Try building again with cryptography at the minimum version allowed in setup.py.
Perados
commented
6 years ago @nchammas Thank you for your quick response! I still get some warnings with cryptography==1.7.2, but not the one that was causing the error at runtime. I guess the best way to go is to update PyInstaller to 3.3. But in any case, I am curious to know how you build the official releases, what is the workflow to avoid this kind of nasty bugs? Any plans on using pipenv and its awesome Pipfile.lock to produce deterministic builds?
nchammas
commented
6 years ago The packaging code is just for me to use to distribute Flintrock. It's not intended for general use.
That said, we would indeed be better off if we pinned all transitive dependencies to known good versions to avoid situations like this one, where a non-pinned transitive dependency causes issues depending on its precise version. I'm guessing when I built the standalone packages for Flintrock for v0.9.0, the version of Cryptography available then did not have any problems. Do you have issues using the standalone package published here?
Pipenv looks like a great tool, and I've been following its progress since it was first announced. The tool is still moving quickly though, and we have bigger fish to fry here in the meantime, so I don't have any plans at the moment to move to Pipenv.
Do you regularly create your own Flintrock packages? I'm curious what motivated you to do so. Perhaps there are others doing the same who would benefit from more deterministic builds.
Perados
commented
6 years ago Yes, pinned versions would be better for the standalone version build because the executable is isolated from any other installed python package.
I am sure there is no problem with the official release, otherwise somebody else would have already raised the issue. Here is how I fixed my problem (this is definitely not permanent, I just needed to make it work): https://github.com/adotmob/flintrock/commit/416b19003104b07252ca1790fae2e807d6ff6211
Yes, we do regularly create our own Flintrock packages. Basically, every time we want to upgrade Flintrock, we rebase on your master branch and build internally. I think it is mainly because we use Flintrock from a scheduler, which runs on Python 2, so pip install flintrock is not an option. And also, we use private VPCs. You can check the commits we add to make it possible here: https://github.com/adotmob/flintrock/commits/adotmob-version-0.9
nchammas
commented
6 years ago Thanks for sharing that background. I'm sorry I've been difficult about getting support for private VPCs in. 😅 The contributions I've gotten in this regard have been good, but always tended to miss something important that would keep me from merging them in.
Would you like to file a separate issue about moving to more deterministic builds, whether it's by fleshing out setup.py some more, or by using a tool like Pipenv or poetry?
Perados
commented
6 years ago
Using the standalone package, commands
run-commandandcopy-filesometimes raiseImportError: cannot import name 'backend'. I haven't been able to reproduce this error consistently.Here's a stacktrace for run-command:
And a stacktrace for copy-file:
This was done with a cluster of 34 slaves, we can see that both commands are partially executed.
This error may be due to https://github.com/nchammas/flintrock/pull/161, which migrates paramiko from version 1.15.4 (which use pycryto) to version 2.0.2 (which uses cryptography).
cryptography supports pyInstaller since version 1.6, according to https://github.com/pyca/cryptography/issues/2039. However, paramiko 2.0.2+ uses cryptography>=1.1