Closed OrKoN closed 6 years ago
I think I see what's going on here.
The example's Cargo.toml depends on the current checkout of rust-crowbar
. Building it in a docker container like that ends up losing the parent directory, and thus that dependency fails.
I'm specifically referring to the lines:
It's possible that editing the dependency to be crowbar = "0.1"
will be enough for the docker method of building to work.
It does make sense for the examples to be compiled against the local copy of rust-crowbar, especially for development purposes... but that leads to it not working with the builder and a confusing error.
Maybe the builder documentation should mention it doesn't work on the examples, or maybe there's some cleverness would allow a single Cargo.toml to work for both use-cases?
Let me know if the above makes sense!
@euank Thanks! You are right. This was causing the problem. Actually I have seen these lines in Cargo.toml but I have failed to correlate the error message with them. I think it might help if examples also have a readme file which explains how to build them.
Nevertheless, after I changed the cargo.toml like this:
[package]
name = "lambda-echo"
version = "0.2.0"
authors = ["Iliana Weller <ilianaw@buttslol.net>"]
publish = false
[lib]
name = "lambda"
crate-type = ["cdylib"]
[dependencies]
# Normally you'd write: crowbar = "0.2"
crowbar = "0.1"
cpython = { version = "0.1", default-features = false }
[features]
default = ["cpython/python3-sys"]
I get the following error:
error[E0428]: the name `PYTHONSYS_ENV_VAR` is defined multiple times
--> /root/.cargo/registry/src/github.com-1ecc6299db9ec823/cpython-0.1.0/build.rs:10:1
|
7 | const PYTHONSYS_ENV_VAR: &'static str = "DEP_PYTHON27_PYTHON_FLAGS";
| -------------------------------------------------------------------- previous definition of the value `PYTHONSYS_ENV_VAR` here
...
10 | const PYTHONSYS_ENV_VAR: &'static str = "DEP_PYTHON3_PYTHON_FLAGS";
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ `PYTHONSYS_ENV_VAR` redefined here
|
= note: `PYTHONSYS_ENV_VAR` must be defined only once in the value namespace of this module
error: aborting due to previous error
error: Could not compile `cpython`.
warning: build failed, waiting for other jobs to finish...
error: build failed
It seems to be building if I change python3-sys to default = ["cpython/python27-sys"]
in cargo.toml. But when I try to run produced lambda.zip on AWS I get an error: Fatal Python error: PyThreadState_Get: no current thread
Hi @OrKoN,
The Fatal Python error: PyThreadState_Get: no current thread
error message means that the module was compiled against an incompatible version of Python. In my testing on US East (N. Virginia)
and US West (N. California)
the python version for the python 2.7 runtime is 2.7.12 but @ilianaw's builder image compiles a 2.7.13 pyenv.
I was able to successfully build and deploy a lambda function using rust-crowbar with the following Dockerfile:
FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y python27-devel bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]
with the following modified build.sh:
#!/bin/bash
set -euo pipefail
export CARGO_TARGET_DIR=$(mktemp -d)
(
if [[ $# -gt 0 ]]; then
yum install -y "$@"
fi
. $HOME/.cargo/env
cargo build ${CARGO_FLAGS:-} --release
) 1>&2
cd $CARGO_TARGET_DIR/release
(
strip liblambda.so
zip lambda.zip liblambda.so
) 1>&2
exec cat lambda.zip
Thanks @ChristopherMacGown I didn't that even minor versions of Python are not compatible :-) that worked fine for me too.
I wasn't sure if that was the case either, so I've run a test with just the pyenv 2.7.13 and it fails the same way:
FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ENV PYENV_ROOT=/usr/local/pyenv PYTHON_CONFIGURE_OPTS="OPT=\"-fPIC\""
RUN git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT
RUN /usr/local/pyenv/bin/pyenv install 2.7.13
#RUN /usr/local/pyenv/bin/pyenv install 3.6.1
#RUN /usr/local/pyenv/bin/pyenv global 3.6.1 2.7.13
RUN /usr/local/pyenv/bin/pyenv global 2.7.13
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]
with the base build.sh results in:
docker run -it -v $(pwd):/code:ro amazonlinux:latest /bin/bash
bash-4.2# yum install -y unzip
# elided yum stuff...
bash-4.2# unzip code/lambda.zip
Archive: code/lambda.zip
inflating: liblambda.so
bash-4.2# python
Python 2.7.12 (default, Sep 1 2016, 22:14:00)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblambda
Fatal Python error: PyThreadState_Get: no current thread
Aborted
I used pyenv because at the time of writing that Amazon Linux didn't have a 3.6 build available yet.
I wonder if it works fine if we use 2.7.12 in pyenv...?
Not as far as my testing shows:
FROM amazonlinux:latest
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y --default-toolchain stable
RUN yum install -y bzip2-devel gcc gcc-c++ git openssl-devel readline-devel sqlite-devel zip && yum clean all
ENV PYENV_ROOT=/usr/local/pyenv PYTHON_CONFIGURE_OPTS="OPT=\"-fPIC\""
RUN git clone https://github.com/pyenv/pyenv.git $PYENV_ROOT
RUN /usr/local/pyenv/bin/pyenv install 2.7.12
#RUN /usr/local/pyenv/bin/pyenv install 3.6.1
#RUN /usr/local/pyenv/bin/pyenv global 3.6.1 2.7.13
RUN /usr/local/pyenv/bin/pyenv global 2.7.12
ADD build.sh /usr/local/bin/
VOLUME ["/code"]
WORKDIR /code
ENTRYPOINT ["/usr/local/bin/build.sh"]
-- ELIDED -- Successfully built 91d63bed5bc0
$ docker run --rm -v $(pwd):/code:ro 91d63bed5bc0 > lambda.zip
... ELIDED ...
Compiling lambda-test v0.1.0 (file:///code)
Finished release [optimized] target(s) in 32.5 secs
adding: liblambda.so (deflated 61%)
Installed:
unzip.x86_64 0:6.0-4.10.amzn1
Complete!
bash-4.2# unzip code/
.git/ .gitignore Cargo.lock Cargo.toml butts lambda-2.zip lambda.zip src/ target/
bash-4.2# unzip code/lambda.zip
Archive: code/lambda.zip
inflating: liblambda.so
bash-4.2# python
Python 2.7.12 (default, Sep 1 2016, 22:14:00)
[GCC 4.8.3 20140911 (Red Hat 4.8.3-9)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import liblambda
Fatal Python error: PyThreadState_Get: no current thread
Aborted
bash-4.2#
Well, sounds like pyenv is out for 2.7 then.
pyenv is also out for 3.6. It fails with Fatal Python error: take_gil: NULL tstate
, which is somehow even more obtuse than the 2.7 error :)
I'm prepping a commit which just uses python27-devel
and python36-devel
from Amazon Linux, now that 2017.09 is out and Python 3.6 is available in the repos...
For @OrKoN's original issue, you can run examples in the builder like this, from the root of the project:
docker run --rm -v $(pwd):/code:ro -w /code/examples/echo ilianaw/crowbar-builder > lambda.zip
as the builder script just execs cargo build
from the workdir.
Hi!
I have encountered a problem running the builder:
I tried to fix it but it looks like a glitch in cargo. Perhaps someone could help to solve this problem.
Best regards, Alex