dockerization

[youngser]

@disa-mhembere: I just talked to the guy at JPL, who is in charge for the annotation schema. He suggested to put all the 11 primitives into a single docker{file/image/container?} whatever that means, and let the Python wrapper (or separate Python code, I didn't fully get this) to "communicate" with the docker image! Do you understand this?

For now, if it's not easy to do, that's ok, we will just use sgm for this dry run.

Meanwhile, here is a dockerfile using D3M base docker image and a shell script start.sh to call it. I don't even know if this is relevant or not. (NB: There is another one for Python 3.5.)

This fixed environment will be one of the Docker containers provided by JPL in this gitlab repository.

FROM registry.datadrivendiscovery.org/jpl/docker_images/jpl_base_python3.5
MAINTAINER D3M

RUN \
pip3 install punk

EXPOSE 8888

COPY start.sh /home/start.sh
RUN set -ex \ && chmod 755 /home/start.sh
WORKDIR /home
CMD ["./start.sh"]

#!/bin/bash 

HASH_PASS=$(python3 -c 'from notebook.auth import passwd; print(passwd("admin"))')
jupyter notebook --generate-config --allow-root
echo 'c.NotebookApp.password'='"'$HASH_PASS'"' >> /root/.jupyter/jupyter_notebook_config.py
jupyter notebook --allow-root --ip=0.0.0.0 --port=8888

[disa-mhembere]

I have made a functional docker file with all the dependencies necessary to run the python here. I'm out of time so you'll have to figure out how to get it to run all the primitives without me unfortunately :(

[youngser]

Alas, got this error...

    building 'r_utils' library
    creating build/temp.linux-x86_64-3.4
    creating build/temp.linux-x86_64-3.4/rpy
    creating build/temp.linux-x86_64-3.4/rpy/rinterface
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -I./rpy/rinterface -I/usr/local/lib/R/include -c ./rpy/rinterface/r_utils.c -o build/temp.linux-x86_64-3.4/./rpy/rinterface/r_utils.o
    In file included from /usr/local/lib/R/include/Rdefines.h:36:0,
                     from ./rpy/rinterface/r_utils.c:24:
    /usr/local/lib/R/include/R_ext/Memory.h:48:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     int R_gc_running();
     ^
    In file included from /usr/local/lib/R/include/Rinternals.h:51:0,
                     from /usr/local/lib/R/include/Rdefines.h:40,
                     from ./rpy/rinterface/r_utils.c:24:
    /usr/local/lib/R/include/R_ext/Rdynload.h:38:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     typedef void * (*DL_FUNC)();
     ^
    In file included from /usr/local/lib/R/include/Rdefines.h:40:0,
                     from ./rpy/rinterface/r_utils.c:24:
    /usr/local/lib/R/include/Rinternals.h:919:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     const char *R_curErrorBuf();
     ^
    x86_64-linux-gnu-gcc-ar rc build/temp.linux-x86_64-3.4/libr_utils.a build/temp.linux-x86_64-3.4/./rpy/rinterface/r_utils.o
    running build_ext
    R version 3.4.0 (2017-04-21) -- "You Stupid Darkness"
    building 'rpy2.rinterface._rinterface' extension
    x86_64-linux-gnu-gcc -pthread -DNDEBUG -g -fwrapv -O2 -Wall -Wstrict-prototypes -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 -fPIC -DR_INTERFACE_PTRS=1 -DHAVE_POSIX_SIGJMP=1 -DRIF_HAS_RSIGHAND=1 -DCSTACK_DEFNS=1 -DHAS_READLINE=1 -I./rpy/rinterface -I/usr/local/lib/R/include -I/usr/include/python3.4m -c ./rpy/rinterface/_rinterface.c -o build/temp.linux-x86_64-3.4/./rpy/rinterface/_rinterface.o
    In file included from /usr/local/lib/R/include/R.h:87:0,
                     from ./rpy/rinterface/_rinterface.h:8,
                     from ./rpy/rinterface/_rinterface.c:52:
    /usr/local/lib/R/include/R_ext/Memory.h:48:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     int R_gc_running();
     ^
    In file included from /usr/local/lib/R/include/Rinternals.h:51:0,
                     from ./rpy/rinterface/_rinterface.h:9,
                     from ./rpy/rinterface/_rinterface.c:52:
    /usr/local/lib/R/include/R_ext/Rdynload.h:38:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     typedef void * (*DL_FUNC)();
     ^
    In file included from ./rpy/rinterface/_rinterface.h:9:0,
                     from ./rpy/rinterface/_rinterface.c:52:
    /usr/local/lib/R/include/Rinternals.h:919:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     const char *R_curErrorBuf();
     ^
    In file included from ./rpy/rinterface/_rinterface.c:64:0:
    /usr/local/lib/R/include/Rinterface.h:157:1: warning: function declaration isn’t a prototype [-Wstrict-prototypes]
     extern void (*ptr_R_ProcessEvents)();
     ^
    In file included from ./rpy/rinterface/_rinterface.c:112:0:
    ./rpy/rinterface/embeddedr.c: In function ‘SexpObject_clear’:
    ./rpy/rinterface/embeddedr.c:48:12: warning: unused variable ‘res’ [-Wunused-variable]
           SEXP res = rpy2_remove(Rf_mkString(name_buf),
                ^
    In file included from ./rpy/rinterface/_rinterface.c:115:0:
    ./rpy/rinterface/sexp.c: In function ‘Sexp_init’:
    ./rpy/rinterface/sexp.c:680:13: warning: unused variable ‘copy’ [-Wunused-variable]
       PyObject *copy = Py_True;
                 ^
    In file included from ./rpy/rinterface/_rinterface.c:118:0:
    ./rpy/rinterface/sequence.c: In function ‘VectorSexp_ass_slice’:
    ./rpy/rinterface/sequence.c:453:11: warning: variable ‘len_R’ set but not used [-Wunused-but-set-variable]
       R_len_t len_R;
               ^
    ./rpy/rinterface/_rinterface.c: In function ‘Sexp_rcall’:
    ./rpy/rinterface/_rinterface.c:1717:21: warning: assignment from incompatible pointer type
           argNameString = Rf_mkCharCE(PyBytes_AsString(pybytes), CE_UTF8);
                         ^
    ./rpy/rinterface/_rinterface.c:1718:32: warning: passing argument 1 of ‘Rf_installChar’ from incompatible pointer type
           SET_TAG(c_R, installChar(argNameString));
                                    ^
    In file included from ./rpy/rinterface/_rinterface.h:9:0,
                     from ./rpy/rinterface/_rinterface.c:52:
    /usr/local/lib/R/include/Rinternals.h:855:6: note: expected ‘SEXP’ but argument is of type ‘const char *’
     SEXP Rf_installChar(SEXP);
          ^
    In file included from ./rpy/rinterface/_rinterface.c:52:0:
    ./rpy/rinterface/_rinterface.c: At top level:
    ./rpy/rinterface/_rinterface.h:194:44: warning: ‘PyRinterface_IsInitialized’ declared ‘static’ but never defined [-Wunused-function]
       static PyRinterface_IsInitialized_RETURN PyRinterface_IsInitialized PyRinterface_IsInitialized_PROTO;
                                                ^
    ./rpy/rinterface/_rinterface.h:195:38: warning: ‘PyRinterface_FindFun’ declared ‘static’ but never defined [-Wunused-function]
       static PyRinterface_FindFun_RETURN PyRinterface_FindFun PyRinterface_FindFun_PROTO;
                                          ^
    ./rpy/rinterface/_rinterface.h:196:20: warning: ‘embeddedR_isInitialized’ defined but not used [-Wunused-variable]
       static PyObject *embeddedR_isInitialized;
                        ^
    In file included from ./rpy/rinterface/_rinterface.c:118:0:
    ./rpy/rinterface/sequence.c:1986:1: warning: ‘ComplexVectorSexp_AsSexp’ defined but not used [-Wunused-function]
     ComplexVectorSexp_AsSexp(PyObject *pyfloat) {
     ^
    In file included from ./rpy/rinterface/_rinterface.c:52:0:
    ./rpy/rinterface/_rinterface.c: In function ‘EmbeddedR_ShowFiles’:
    ./rpy/rinterface/_rinterface.h:72:5: warning: ‘gstate’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         PyGILState_Release(gstate);                 \
         ^
    ./rpy/rinterface/_rinterface.c:801:20: note: ‘gstate’ was declared here
       PyGILState_STATE gstate;
                        ^
    In file included from ./rpy/rinterface/_rinterface.c:52:0:
    ./rpy/rinterface/_rinterface.c: In function ‘EmbeddedR_ReadConsole’:
    ./rpy/rinterface/_rinterface.h:72:5: warning: ‘gstate’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         PyGILState_Release(gstate);                 \
         ^
    ./rpy/rinterface/_rinterface.c:490:20: note: ‘gstate’ was declared here
       PyGILState_STATE gstate;
                        ^
    In file included from ./rpy/rinterface/_rinterface.c:52:0:
    ./rpy/rinterface/_rinterface.c: In function ‘EmbeddedR_CleanUp’:
    ./rpy/rinterface/_rinterface.h:72:5: warning: ‘gstate’ may be used uninitialized in this function [-Wmaybe-uninitialized]
         PyGILState_Release(gstate);                 \
         ^
    ./rpy/rinterface/_rinterface.c:934:20: note: ‘gstate’ was declared here
       PyGILState_STATE gstate;
                        ^
    x86_64-linux-gnu-gcc -pthread -shared -Wl,-O1 -Wl,-Bsymbolic-functions -Wl,-z,relro -Wl,-z,relro -g -fstack-protector-strong -Wformat -Werror=format-security -D_FORTIFY_SOURCE=2 build/temp.linux-x86_64-3.4/./rpy/rinterface/_rinterface.o -L/usr/local/lib -L/usr/local/lib/R/lib -Lbuild/temp.linux-x86_64-3.4 -Wl,--enable-new-dtags,-R/usr/local/lib -Wl,--enable-new-dtags,-R/usr/local/lib/R/lib -lR -lpcre -llzma -lbz2 -lz -lrt -ldl -lm -licuuc -licui18n -lr_utils -o build/lib.linux-x86_64-3.4/rpy2/rinterface/_rinterface.cpython-34m.so -Wl,--export-dynamic -fopenmp
    /usr/bin/ld: cannot find -licuuc
    /usr/bin/ld: cannot find -licui18n
    collect2: error: ld returned 1 exit status
    error: command 'x86_64-linux-gnu-gcc' failed with exit status 1

    ----------------------------------------
Command "/usr/bin/python3 -u -c "import setuptools, tokenize;__file__='/tmp/pip-build-f40hez51/rpy2/setup.py';f=getattr(tokenize, 'open', open)(__file__);code=f.read().replace('\r\n', '\n');f.close();exec(compile(code, __file__, 'exec'))" install --record /tmp/pip-1dhnqnve-record/install-record.txt --single-version-externally-managed --compile" failed with error code 1 in /tmp/pip-build-f40hez51/rpy2/
The command '/bin/sh -c pip3 install numpy scipy python-igraph rpy2 sklearn jinja2' returned a non-zero code: 1

[disa-mhembere]

I don't have this error :/ Did you build with no cache?

docker build --no-cache -t d3m .

What I do is Google what the error is. For instance I see the error:

/usr/bin/ld: cannot find -licuuc
 /usr/bin/ld: cannot find -licui18n

so try adding libicu-dev to the apt-get install list.

Doesn't work? Just repeat the process again and it will eventually work

[youngser]

The good news is that the dockerfile works (after adding libicu-dev :-) ! But, the bad news is that the wrapper fails to run.

It looks like again rpy2 <-> R(>3.3) issue; I remember I saw a stackover page mentioned about it. I will try to change a base image that meet this requirement.

[disa-mhembere]

This is all good -- I expect this & will make the code Python3 compatible last

[youngser]

Ah, I tried another docker base image that has R 3.0 and got the same error.

BTW, the JPL guy said we may not need to have and don't want to have our own dockerfile. The TA2 team will have a docker container that will have R/Python/rpy2 installed. All we need to send is a script that has lines of installation of our R packages, which I can do.

[disa-mhembere]

So the problem is in the wrappers not the machine. Once I make it py3 compatible -- it should all work so don't worry about this -- it's on me