search

clab / dynet

DyNet: The Dynamic Neural Network Toolkit
Apache License 2.0
3.42k stars 704 forks source link

Manual build fails on Docker #1241

Open jacobsenkaare opened 6 years ago

jacobsenkaare commented 6 years ago

Hi

My company is using Domino Datalab, and to use DyNet with this I need to build a Docker-image with DyNet.

It works with RUN pip install git+https://github.com/clab/dynet#egg=dynet, but I would also like to build images with MKL and GPU support. Because of this I would also like to be able to do a manual install. I have tried with the following:

# prereq for DyNet
RUN sudo apt-get update
RUN sudo apt-get -y install python-pip build-essential cmake mercurial
RUN pip install cython

# Install DyNet
RUN mkdir dynet-base && \
 cd dynet-base  && \
 git clone https://github.com/clab/dynet.git && \
 hg clone https://bitbucket.org/eigen/eigen -r 2355b22 && \
 cd dynet && \
 mkdir build && \
 cd build && \
 cmake .. -DEIGEN3_INCLUDE_DIR=/home/ubuntu/dynet-base/eigen -DPYTHON=`which python` && \
 make -j 2

But the install fails, the log is below. I hope one of you can help me with what to do next. Thank you Kåre Jacobsen

(The -symbols should be small left pointing arrows)

======================================================================
Building
======================================================================
Step 1 : FROM quay.io/domino/base:2016-12-07_1239_flat2
---> 842e90aa6dcc
Step 2 : RUN sudo apt-get update
---> Running in 21fbe8499fe2
Ign http://cran.rstudio.com trusty/ InRelease
Get:1 http://cran.rstudio.com trusty/ Release.gpg [473 B]
Get:2 http://cran.rstudio.com trusty/ Release [3066 B]
Ign http://archive.ubuntu.com trusty InRelease
Get:3 http://archive.ubuntu.com trusty-updates InRelease [65.9 kB]
Get:4 http://archive.ubuntu.com trusty-security InRelease [65.9 kB]
Get:5 http://archive.ubuntu.com trusty-proposed InRelease [65.9 kB]
Ign http://ppa.launchpad.net trusty InRelease
Get:6 http://ppa.launchpad.net trusty InRelease [15.4 kB]
Get:7 http://ppa.launchpad.net trusty InRelease [15.4 kB]
Hit http://ppa.launchpad.net trusty InRelease
Get:8 http://ppa.launchpad.net trusty InRelease [15.5 kB]
Get:9 http://ppa.launchpad.net trusty InRelease [15.5 kB]
Ign http://developer.download.nvidia.com InRelease
Ign http://downloads-distro.mongodb.org dist InRelease
Hit http://archive.ubuntu.com trusty Release.gpg
Hit http://ppa.launchpad.net trusty Release.gpg
Hit http://archive.ubuntu.com trusty Release
Get:10 http://developer.download.nvidia.com Release.gpg [801 B]
Hit http://ppa.launchpad.net trusty Release
Get:11 http://developer.download.nvidia.com Release [564 B]
Hit http://downloads-distro.mongodb.org dist Release.gpg
Get:12 http://cran.rstudio.com trusty/ Packages [250 kB]
Hit http://downloads-distro.mongodb.org dist Release
Get:13 http://archive.ubuntu.com trusty-updates/main Sources [506 kB]
Get:14 http://archive.ubuntu.com trusty-updates/restricted Sources [6449 B]
Get:15 http://archive.ubuntu.com trusty-updates/universe Sources [248 kB]
Get:16 http://archive.ubuntu.com trusty-updates/main amd64 Packages [1311 kB]
Get:17 http://archive.ubuntu.com trusty-updates/restricted amd64 Packages [21.4 kB]
Get:18 http://archive.ubuntu.com trusty-updates/universe amd64 Packages [570 kB]
Get:19 http://archive.ubuntu.com trusty-security/main Sources [190 kB]
Get:20 http://archive.ubuntu.com trusty-security/restricted Sources [5050 B]
Get:21 http://archive.ubuntu.com trusty-security/universe Sources [82.5 kB]
Get:22 http://archive.ubuntu.com trusty-security/main amd64 Packages [881 kB]
Get:23 http://archive.ubuntu.com trusty-security/restricted amd64 Packages [18.1 kB]
Get:24 http://archive.ubuntu.com trusty-security/universe amd64 Packages [258 kB]
Get:25 http://archive.ubuntu.com trusty-proposed/main amd64 Packages [117 kB]
Get:26 http://archive.ubuntu.com trusty-proposed/restricted amd64 Packages [681 B]
Get:27 http://ppa.launchpad.net trusty/main amd64 Packages [1090 B]
Get:28 http://ppa.launchpad.net trusty/main amd64 Packages [4473 B]
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Get:29 http://ppa.launchpad.net trusty/main amd64 Packages [8302 B]
Ign http://developer.download.nvidia.com Release
Hit http://archive.ubuntu.com trusty/main Sources
Get:30 http://ppa.launchpad.net trusty/main amd64 Packages [3146 B]
Get:31 http://developer.download.nvidia.com Packages [183 kB]
Hit http://archive.ubuntu.com trusty/restricted Sources
Hit http://ppa.launchpad.net trusty/main amd64 Packages
Hit http://archive.ubuntu.com trusty/universe Sources
Hit http://downloads-distro.mongodb.org dist/10gen amd64 Packages
Hit http://archive.ubuntu.com trusty/main amd64 Packages
Hit http://archive.ubuntu.com trusty/restricted amd64 Packages
Hit http://archive.ubuntu.com trusty/universe amd64 Packages
Fetched 4930 kB in 10s (472 kB/s)
Reading package lists...
W: Size of file /var/lib/apt/lists/archive.ubuntu.com_ubuntu_dists_trusty-proposed_main_binary-amd64_Packages.gz is not what the server reported 117109 119717
W: Size of file /var/lib/apt/lists/ppa.launchpad.net_cwchien_gradle_ubuntu_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported 1090 1876
W: Size of file /var/lib/apt/lists/ppa.launchpad.net_staticfloat_juliareleases_ubuntu_dists_trusty_main_binary-amd64_Packages.gz is not what the server reported 8302 8303

W: GPG error: http://developer.download.nvidia.com Release: The following signatures couldn't be verified because the public key is not available: NO_PUBKEY F60F4B3D7FA2AF80

---> cac9f470ff03
Error removing intermediate container 21fbe8499fe2: Driver overlay failed to remove root filesystem 21fbe8499fe2232be0a03281dc69fb0b2a675071be2b713e9ce1d36824826305: remove /var/lib/docker/overlay/beec5dda4ccfd02a450a28248b503b3a8d41d3652b454a75e1bd3f049b8948bf/merged/root: directory not empty
Step 3 : RUN sudo apt-get -y install python-pip build-essential cmake mercurial
---> Running in 171fa9d79765
Reading package lists...
Building dependency tree...

Reading state information...
build-essential is already the newest version.
cmake is already the newest version.
The following extra packages will be installed:
mercurial-common python-chardet-whl python-colorama-whl python-distlib-whl
python-html5lib-whl python-pip-whl python-requests-whl python-setuptools-whl
python-six-whl python-urllib3-whl python-wheel
Suggested packages:
qct kdiff3 kdiff3-qt kompare meld tkcvs mgdiff python-mysqldb
python-pygments
Recommended packages:
python-dev-all
The following NEW packages will be installed:
mercurial mercurial-common python-chardet-whl python-colorama-whl
python-distlib-whl python-html5lib-whl python-pip-whl python-requests-whl
python-setuptools-whl python-six-whl python-urllib3-whl python-wheel
The following packages will be upgraded:
python-pip
1 upgraded, 12 newly installed, 0 to remove and 765 not upgraded.
Need to get 2798 kB of archives.
After this operation, 10.2 MB of additional disk space will be used.
Get:1 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe mercurial-common all 2.8.2-1ubuntu1.3 [1520 kB]
Get:2 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe mercurial amd64 2.8.2-1ubuntu1.3 [41.1 kB]
Get:3 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe python-chardet-whl all 2.2.1-2~ubuntu1 [170 kB]
Get:4 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe python-colorama-whl all 0.2.5-0.1ubuntu2 [18.2 kB]
Get:5 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe python-distlib-whl all 0.1.8-1ubuntu1 [140 kB]
Get:6 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-html5lib-whl all 0.999-3~ubuntu1 [109 kB]
Get:7 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-six-whl all 1.5.2-1ubuntu1 [10.5 kB]
Get:8 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-urllib3-whl all 1.7.1-1ubuntu4 [64.0 kB]
Get:9 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-requests-whl all 2.2.1-1ubuntu0.3 [227 kB]
Get:10 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-setuptools-whl all 3.3-1ubuntu2 [244 kB]
Get:11 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip-whl all 1.5.4-1ubuntu4 [111 kB]
Get:12 http://archive.ubuntu.com/ubuntu/ trusty-updates/universe python-pip all 1.5.4-1ubuntu4 [97.3 kB]
Get:13 http://archive.ubuntu.com/ubuntu/ trusty-updates/main python-wheel all 0.24.0-1~ubuntu1 [44.7 kB]
debconf: unable to initialize frontend: Dialog

debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)

debconf: falling back to frontend: Readline

debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype

dpkg-preconfigure: unable to re-open stdin:

Fetched 2798 kB in 0s (9534 kB/s)
Selecting previously unselected package mercurial-common.
(Reading database ... 187518 files and directories currently installed.)
Preparing to unpack .../mercurial-common_2.8.2-1ubuntu1.3_all.deb ...
Unpacking mercurial-common (2.8.2-1ubuntu1.3) ...
Selecting previously unselected package mercurial.
Preparing to unpack .../mercurial_2.8.2-1ubuntu1.3_amd64.deb ...
Unpacking mercurial (2.8.2-1ubuntu1.3) ...
Selecting previously unselected package python-chardet-whl.
Preparing to unpack .../python-chardet-whl_2.2.1-2~ubuntu1_all.deb ...
Unpacking python-chardet-whl (2.2.1-2~ubuntu1) ...
Selecting previously unselected package python-colorama-whl.
Preparing to unpack .../python-colorama-whl_0.2.5-0.1ubuntu2_all.deb ...
Unpacking python-colorama-whl (0.2.5-0.1ubuntu2) ...
Selecting previously unselected package python-distlib-whl.
Preparing to unpack .../python-distlib-whl_0.1.8-1ubuntu1_all.deb ...
Unpacking python-distlib-whl (0.1.8-1ubuntu1) ...
Selecting previously unselected package python-html5lib-whl.
Preparing to unpack .../python-html5lib-whl_0.999-3~ubuntu1_all.deb ...
Unpacking python-html5lib-whl (0.999-3~ubuntu1) ...
Selecting previously unselected package python-six-whl.
Preparing to unpack .../python-six-whl_1.5.2-1ubuntu1_all.deb ...
Unpacking python-six-whl (1.5.2-1ubuntu1) ...
Selecting previously unselected package python-urllib3-whl.
Preparing to unpack .../python-urllib3-whl_1.7.1-1ubuntu4_all.deb ...
Unpacking python-urllib3-whl (1.7.1-1ubuntu4) ...
Selecting previously unselected package python-requests-whl.
Preparing to unpack .../python-requests-whl_2.2.1-1ubuntu0.3_all.deb ...
Unpacking python-requests-whl (2.2.1-1ubuntu0.3) ...
Selecting previously unselected package python-setuptools-whl.
Preparing to unpack .../python-setuptools-whl_3.3-1ubuntu2_all.deb ...
Unpacking python-setuptools-whl (3.3-1ubuntu2) ...
Selecting previously unselected package python-pip-whl.
Preparing to unpack .../python-pip-whl_1.5.4-1ubuntu4_all.deb ...
Unpacking python-pip-whl (1.5.4-1ubuntu4) ...
Preparing to unpack .../python-pip_1.5.4-1ubuntu4_all.deb ...
Unpacking python-pip (1.5.4-1ubuntu4) over (1.5.4-1) ...
Selecting previously unselected package python-wheel.
Preparing to unpack .../python-wheel_0.24.0-1~ubuntu1_all.deb ...
Unpacking python-wheel (0.24.0-1~ubuntu1) ...
Processing triggers for man-db (2.6.7.1-1ubuntu1) ...
Setting up mercurial-common (2.8.2-1ubuntu1.3) ...
Setting up mercurial (2.8.2-1ubuntu1.3) ...
debconf: unable to initialize frontend: Dialog
debconf: (Dialog frontend will not work on a dumb terminal, an emacs shell buffer, or without a controlling terminal.)
debconf: falling back to frontend: Readline

debconf: unable to initialize frontend: Readline
debconf: (This frontend requires a controlling tty.)
debconf: falling back to frontend: Teletype


Creating config file /etc/mercurial/hgrc.d/hgext.rc with new version

Setting up python-chardet-whl (2.2.1-2~ubuntu1) ...
Setting up python-colorama-whl (0.2.5-0.1ubuntu2) ...
Setting up python-distlib-whl (0.1.8-1ubuntu1) ...
Setting up python-html5lib-whl (0.999-3~ubuntu1) ...
Setting up python-six-whl (1.5.2-1ubuntu1) ...
Setting up python-urllib3-whl (1.7.1-1ubuntu4) ...
Setting up python-requests-whl (2.2.1-1ubuntu0.3) ...
Setting up python-setuptools-whl (3.3-1ubuntu2) ...
Setting up python-pip-whl (1.5.4-1ubuntu4) ...
Setting up python-pip (1.5.4-1ubuntu4) ...
Setting up python-wheel (0.24.0-1~ubuntu1) ...
---> 234cdc856760
Error removing intermediate container 21fbe8499fe2: No such container: 21fbe8499fe2232be0a03281dc69fb0b2a675071be2b713e9ce1d36824826305
Step 4 : RUN pip install cython
---> Running in 274acfb8c41a
The directory '/home/ubuntu/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.
The directory '/home/ubuntu/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.

Requirement already satisfied (use --upgrade to upgrade): cython in /usr/local/lib/python2.7/dist-packages
You are using pip version 8.1.2, however version 9.0.1 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.

---> acb416342504
Removing intermediate container 171fa9d79765
Error removing intermediate container 274acfb8c41a: Driver overlay failed to remove root filesystem 274acfb8c41a4a40862e817383ceef1f3ee230fb971c0e7613a3a69a90a83731: remove /var/lib/docker/overlay/cd9f80fb728e28763ca9950f9f17858140819cb052ec0c66b649d5b726a6e515/merged/root: directory not empty
Step 5 : RUN mkdir dynet-base && cd dynet-base && git clone https://github.com/clab/dynet.git && hg clone https://bitbucket.org/eigen/eigen -r 2355b22 && cd dynet && mkdir build && cd build && cmake .. -DEIGEN3_INCLUDE_DIR=/home/ubuntu/dynet-base/eigen -DPYTHON=`which python` && make -j 2
---> Running in 9c721e5a68c0
Cloning into 'dynet'...

destination directory: eigen
adding changesets
adding manifests
adding file changes
added 9491 changesets with 30804 changes to 2945 files
updating to branch default
1672 files updated, 0 files merged, 0 files removed, 0 files unresolved
-- The C compiler identification is GNU 4.8.2
-- The CXX compiler identification is GNU 4.8.2
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc -- works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ -- works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- BACKEND not specified, defaulting to eigen.

-- Eigen dir is /home/ubuntu/dynet-base/eigen

-- Looking for include file pthread.h
-- Looking for include file pthread.h - found
-- Looking for pthread_create
-- Looking for pthread_create - not found
-- Looking for pthread_create in pthreads
-- Looking for pthread_create in pthreads - not found
-- Looking for pthread_create in pthread
-- Looking for pthread_create in pthread - found
-- Found Threads: TRUE
-- Found Cython version 0.23.4

CMAKE_INSTALL_PREFIX="/usr/local"
PROJECT_SOURCE_DIR="/home/ubuntu/dynet-base/dynet"

PROJECT_BINARY_DIR="/home/ubuntu/dynet-base/dynet/build"

LIBS="-lpthread"
EIGEN3_INCLUDE_DIR="/home/ubuntu/dynet-base/eigen"

MKL_LINK_DIRS=""
WITH_CUDA_BACKEND=""

CUDA_RT_FILES=""

CUDA_RT_DIRS=""
CUDA_CUBLAS_FILES=""

CUDA_CUBLAS_DIRS=""

MSVC=""

-- Configuring done
-- Generating done
-- Build files have been written to: /home/ubuntu/dynet-base/dynet/build
Scanning dependencies of target dynet
[ 3%] [ 3%]
Building CXX object dynet/CMakeFiles/dynet.dir/aligned-mem-pool.cc.o
Building CXX object dynet/CMakeFiles/dynet.dir/cfsm-builder.cc.o
[ 4%]
Building CXX object dynet/CMakeFiles/dynet.dir/deep-lstm.cc.o
[ 6%]
Building CXX object dynet/CMakeFiles/dynet.dir/devices.cc.o
[ 7%]
Building CXX object dynet/CMakeFiles/dynet.dir/dict.cc.o
[ 9%]
Building CXX object dynet/CMakeFiles/dynet.dir/dim.cc.o
[ 10%]
Building CXX object dynet/CMakeFiles/dynet.dir/dynet.cc.o
[ 12%]
Building CXX object dynet/CMakeFiles/dynet.dir/exec.cc.o
[ 13%]
Building CXX object dynet/CMakeFiles/dynet.dir/expr.cc.o
[ 15%]
Building CXX object dynet/CMakeFiles/dynet.dir/fast-lstm.cc.o
[ 16%]
Building CXX object dynet/CMakeFiles/dynet.dir/globals.cc.o
[ 18%]
Building CXX object dynet/CMakeFiles/dynet.dir/grad-check.cc.o
[ 20%]
Building CXX object dynet/CMakeFiles/dynet.dir/graph.cc.o
[ 21%]
Building CXX object dynet/CMakeFiles/dynet.dir/gru.cc.o
[ 23%]
Building CXX object dynet/CMakeFiles/dynet.dir/hsm-builder.cc.o
[ 24%] Building CXX object dynet/CMakeFiles/dynet.dir/init.cc.o
[ 26%] Building CXX object dynet/CMakeFiles/dynet.dir/io.cc.o
[ 27%]
Building CXX object dynet/CMakeFiles/dynet.dir/lstm.cc.o
[ 29%]
Building CXX object dynet/CMakeFiles/dynet.dir/mem.cc.o
[ 30%]
Building CXX object dynet/CMakeFiles/dynet.dir/model.cc.o
[ 32%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-activations.cc.o
[ 33%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-affinetransform.cc.o
[ 35%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-argmax.cc.o
[ 36%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-const.cc.o
[ 38%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-cwise.cc.o
[ 40%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-sum.cc.o
[ 41%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-arith-unary.cc.o
[ 43%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-concat.cc.o
[ 44%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-const.cc.o
[ 46%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-contract.cc.o
[ 47%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-conv.cc.o
[ 49%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-conv2d.cc.o
[ 50%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-dropout.cc.o
[ 52%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-flow.cc.o
[ 53%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-hinge.cc.o
[ 55%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-linalg.cc.o
[ 56%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-logsumexp.cc.o
[ 58%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-losses.cc.o
[ 60%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-lstm.cc.o
[ 61%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-matrixmultiply.cc.o
[ 63%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-maxpooling2d.cc.o
[ 64%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-minmax.cc.o
[ 66%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-moments.cc.o
[ 67%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-cumulative.cc.o
[ 69%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-normalization.cc.o
[ 70%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-norms.cc.o
[ 72%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-pickneglogsoftmax.cc.o
[ 73%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-random.cc.o
[ 75%] Building CXX object dynet/CMakeFiles/dynet.dir/nodes-select.cc.o
In file included from /home/ubuntu/dynet-base/dynet/dynet/dim.h:18:0,
from /home/ubuntu/dynet-base/dynet/dynet/tensor.h:15,
from /home/ubuntu/dynet-base/dynet/dynet/tensor-eigen.h:8,
from /home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc:1:
/home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc: In member function 'virtual dynet::Dim dynet::StridedSelect::dim_forward(const std::vector<dynet::Dim>&) const':
/home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc:379:42: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DYNET_ARG_CHECK(from[d] < xs[0].d[d] && from[d] >= 0, "require 0 <= from < dim_size, was " << from[d]);
^
/home/ubuntu/dynet-base/dynet/dynet/except.h:42:11: note: in definition of macro 'DYNET_ARG_CHECK'
if (!(cond)) { \
^

/home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc:381:39: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DYNET_ARG_CHECK(from[d] < xs[0].bd && from[d] >= 0, "require 0 <= from < batch_size, was " << from[d]);
^
/home/ubuntu/dynet-base/dynet/dynet/except.h:42:11: note: in definition of macro 'DYNET_ARG_CHECK'
if (!(cond)) { \
^

/home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc:386:41: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DYNET_ARG_CHECK(to[d] <= xs[0].d[d] && to[d] > 0, "require 0 < to <= dim_size, was " << to[d]);
^
/home/ubuntu/dynet-base/dynet/dynet/except.h:42:11: note: in definition of macro 'DYNET_ARG_CHECK'
if (!(cond)) { \
^

/home/ubuntu/dynet-base/dynet/dynet/nodes-select.cc:388:38: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
DYNET_ARG_CHECK(to[d] <= xs[0].bd && to[d] > 0, "require 0 < to <= batch_size, was " << to[d]);
^
/home/ubuntu/dynet-base/dynet/dynet/except.h:42:11: note: in definition of macro 'DYNET_ARG_CHECK'
if (!(cond)) { \
^

[ 76%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-similarities.cc.o
[ 78%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-softmaxes.cc.o
[ 80%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-to-device.cc.o
/home/ubuntu/dynet-base/dynet/dynet/nodes-softmaxes.cc: In member function 'void dynet::ConstrainedSoftmax::forward_dev_impl(const MyDevice&, const std::vector<const dynet::Tensor*>&, dynet::Tensor&) const [with MyDevice = dynet::Device_CPU]':
/home/ubuntu/dynet-base/dynet/dynet/nodes-softmaxes.cc:486:7: warning: 'max' may be used uninitialized in this function [-Wmaybe-uninitialized]
if (k == 0 || xs[0]->v[k] > max) max = xs[0]->v[k];
^

[ 81%]
Building CXX object dynet/CMakeFiles/dynet.dir/nodes-trig.cc.o
[ 83%]
Building CXX object dynet/CMakeFiles/dynet.dir/param-init.cc.o
[ 84%]
Building CXX object dynet/CMakeFiles/dynet.dir/param-nodes.cc.o
[ 86%]
Building CXX object dynet/CMakeFiles/dynet.dir/pretrain.cc.o
[ 87%] Building CXX object dynet/CMakeFiles/dynet.dir/rnn-state-machine.cc.o
[ 89%]
Building CXX object dynet/CMakeFiles/dynet.dir/rnn.cc.o
[ 90%]
Building CXX object dynet/CMakeFiles/dynet.dir/saxe-init.cc.o
[ 92%]
Building CXX object dynet/CMakeFiles/dynet.dir/shadow-params.cc.o
[ 93%] Building CXX object dynet/CMakeFiles/dynet.dir/tensor.cc.o
[ 95%]
Building CXX object dynet/CMakeFiles/dynet.dir/training.cc.o
[ 96%]
Building CXX object dynet/CMakeFiles/dynet.dir/treelstm.cc.o
/home/ubuntu/dynet-base/dynet/dynet/treelstm.cc: In member function 'virtual dynet::Expression dynet::BidirectionalTreeLSTMBuilder::add_input(int, std::vector<int>, const dynet::Expression&)':
/home/ubuntu/dynet-base/dynet/dynet/treelstm.cc:375:19: warning: comparison between signed and unsigned integer expressions [-Wsign-compare]
if (h.size() == id)
^

[ 98%]
Building CXX object dynet/CMakeFiles/dynet.dir/weight-decay.cc.o
Linking CXX shared library libdynet.so
[ 98%]
Built target dynet
Scanning dependencies of target pydynet_precopy
[ 98%]
Built target pydynet_precopy
Scanning dependencies of target pydynet
[100%]
Generating timestamp
running build_ext
INFO:root:Building Cython extensions...
INFO:root:INCLUDE_DIRS='/home/ubuntu/dynet-base/dynet /home/ubuntu/dynet-base/eigen'
INFO:root:LIBRARIES='dynet'
INFO:root:LIBRARY_DIRS='/home/ubuntu/dynet-base/dynet/build/dynet/ .'
INFO:root:COMPILER_ARGS='-std=c++11 -Wno-unused-function'
INFO:root:EXTRA_LINK_ARGS="-Wl,-rpath='/usr/local/lib/',--no-as-needed"
INFO:root:RUNTIME_LIB_DIRS='/home/ubuntu/dynet-base/dynet/build/dynet/ /usr/local/lib/'
cythoning /home/ubuntu/dynet-base/dynet/build/python/_dynet.pyx to /home/ubuntu/dynet-base/dynet/build/python/_dynet.cpp

Error compiling Cython file:
------------------------------------------------------------
...

Returns:
dynet.Expression: The one hot argmax vector
"""
if gradient_mode == "zero_gradient":
return Expression.from_cexpr(x.cg_version, c_argmax(x.c(), c_ArgmaxGradient.zero_gradient))
^
------------------------------------------------------------

_dynet.pyx:2672:83: 'c_ArgmaxGradient' is not a constant, variable or function identifier


Error compiling Cython file:
------------------------------------------------------------
...
dynet.Expression: The one hot argmax vector
"""
if gradient_mode == "zero_gradient":
return Expression.from_cexpr(x.cg_version, c_argmax(x.c(), c_ArgmaxGradient.zero_gradient))
elif gradient_mode == "straight_through_gradient":
return Expression.from_cexpr(x.cg_version, c_argmax(x.c(), c_ArgmaxGradient.straight_through_gradient))
^
------------------------------------------------------------

_dynet.pyx:2674:83: 'c_ArgmaxGradient' is not a constant, variable or function identifier

building '_dynet' extension
creating build
creating build/temp.linux-x86_64-2.7
creating build/temp.linux-x86_64-2.7/home
creating build/temp.linux-x86_64-2.7/home/ubuntu
creating build/temp.linux-x86_64-2.7/home/ubuntu/dynet-base
creating build/temp.linux-x86_64-2.7/home/ubuntu/dynet-base/dynet
creating build/temp.linux-x86_64-2.7/home/ubuntu/dynet-base/dynet/build
creating build/temp.linux-x86_64-2.7/home/ubuntu/dynet-base/dynet/build/python
x86_64-linux-gnu-gcc -pthread -fno-strict-aliasing -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -fPIC -I/home/ubuntu/dynet-base/dynet -I/home/ubuntu/dynet-base/eigen -I/usr/include/python2.7 -c /home/ubuntu/dynet-base/dynet/build/python/_dynet.cpp -o build/temp.linux-x86_64-2.7/home/ubuntu/dynet-base/dynet/build/python/_dynet.o -std=c++11 -Wno-unused-function
cc1plus: warning: command line option '-Wstrict-prototypes' is valid for C/ObjC but not for C++ [enabled by default]

/home/ubuntu/dynet-base/dynet/build/python/_dynet.cpp:1:2: error: #error Do not use this file, it is the result of a failed Cython compilation.
#error Do not use this file, it is the result of a failed Cython compilation.
^

error: command 'x86_64-linux-gnu-gcc' failed with exit status 1


make[2]: *** [python/timestamp] Error 1

make[1]: *** [python/CMakeFiles/pydynet.dir/all] Error 2

make: *** [all] Error 2

The command '/bin/sh -c mkdir dynet-base && cd dynet-base && git clone https://github.com/clab/dynet.git && hg clone https://bitbucket.org/eigen/eigen -r 2355b22 && cd dynet && mkdir build && cd build && cmake .. -DEIGEN3_INCLUDE_DIR=/home/ubuntu/dynet-base/eigen -DPYTHON=`which python` && make -j 2' returned a non-zero code: 2
======================================================================
Failed
======================================================================
ERROR: ProgressMessage{id=null, status=null, stream=null, error=The command '/bin/sh -c mkdir dynet-base && cd dynet-base && git clone https://github.com/clab/dynet.git && hg clone https://bitbucket.org/eigen/eigen -r 2355b22 && cd dynet && mkdir build && cd build && cmake .. -DEIGEN3_INCLUDE_DIR=/home/ubuntu/dynet-base/eigen -DPYTHON=`which python` && make -j 2' returned a non-zero code: 2, progress=null, progressDetail=null}
neubig commented 6 years ago

Hmm, the functions here look like recently introduced ones. I suspect that either:

  1. You have two versions of DyNet, one slightly older than the other, on your system. Could you make sure that any version of DyNet on your system is the latest one.
  2. It's possible that the recent addition of argmax/straight-through might have broken something (CC: @pmichel31415)

First, could you take a look at 1.?

pmichel31415 commented 6 years ago

I suspect this is a cython version issue, can you try to pip install cython -U ?

jacobsenkaare commented 6 years ago

Thank both of you very much. pip install cython -U upgraded cython from 0.23.4 to 0.27.3 and that solved the problem.

pmichel31415 commented 6 years ago

Glad that helped. We should either up the required version of cython or change the faulty code so that this doesn't happen in the future