search

indigo-dc / onedata

Indigo mirror of http://github.com/onedata/onedata
Apache License 2.0
1 stars 1 forks source link

Building onceclient for SWAMP #18

Closed JK-STFC closed 7 years ago

JK-STFC commented 7 years ago

I'm having trouble building oneclient for the SWAMP analysis (as described in Lisbon).

The SWAMP guys have given me this info - is this correct? If so, how do we normally build it under Jenkins?

JK

The package 'oneclient' from github does not build trivially; it requires that other software, not commonly available in standard repositories, be installed and available on a system.

Details follow:

oneclient requires one mandatory package

fedora/el       debian/ubuntu
----------      -------------
folly-devel     folly-dev

and two optional packages to build

fedora/el/debian/ubuntu
-----------------------
swift-sdk-cpp
aws-sdk-cpp-s3

I've omitted the additional (quite recent) version constraints on those packages.

These packages are not available from standard repositories; they need to be configured and installed on a system before oneclient can be compiled.

I've examined this package on several of our more recent platforms:

centos-6.7
centos-7.2
fedora-23
fedora-24
debian-8.5
ubuntu-16.04

I have not located the necessary packages in the standard repositories for those platforms, nor in common alternate repositories.

caifti commented 7 years ago

You can find the dependencies in http://packages.onedata.org/, http://repo.indigo-datacloud.eu/repository/indigo/1/centos7/x86_64/third-party/, and soon also i nINDIGO-2 repositories.

groundnuty commented 7 years ago

@JK-STFC did you manage to resolve your issue thanks to @caifti suggestion?

JK-STFC commented 7 years ago

Not yet, no. I've been busy trying to install the various OneData components with Matt Viljoen today and yesterday.

I might be able to take a look next week, but I did take a quick look and I wasn't certain how the reply helps. The reply gave 3 locations where I could download things from.

The message from the SWAMP guys stated that they couldn't find it in any of the standard repositories – which are used when building things inside SWAMP.

What commands would be used to download the bits that were missing in my message? There might be a chance of putting them in a preamble or something like that. SWAMP normally takes dependencies as a list of packages and it installs them from the most appropriate place for the chosen OS, but it might be possible to put them in a preamble or something like that – but I'd need to know the commands for each of the platforms – CentOS and Ubuntu 16.04. Maybe if these are non-standard packages they could be included with Oneclient.

Cheers

JK

From: Michal Orzechowski [mailto:notifications@github.com] Sent: Tuesday, April 25, 2017 7:04 AM To: indigo-dc/onedata Cc: Kewley, John (STFC,DL,SC); Mention Subject: Re: [indigo-dc/onedata] Building onceclient for SWAMP (#18)

@JK-STFChttps://github.com/JK-STFC did you managed to resolve your issue thanks to @caiftihttps://github.com/caifti suggestion?

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHubhttps://github.com/indigo-dc/onedata/issues/18#issuecomment-296921301, or mute the threadhttps://github.com/notifications/unsubscribe-auth/AQwk2VYoCvSn3-4kfbjVL35pRvtOmBfFks5rzYzRgaJpZM4NB9Q5.

orviz commented 7 years ago

As suggested just deploy the INDIGO repositories, either installing the indigodc-release package (centos7 [1], ubuntu [2]) or via ansible [3] or puppet [4]. Double check that the third-party repository is enabled.

[1] rpm -ivh http://repo.indigo-datacloud.eu/repository/indigo/1/centos7/x86_64/base/indigodc-release-1.0.0-1.el7.centos.noarch.rpm [2] wget http://repo.indigo-datacloud.eu/repository/indigo/1/ubuntu/dists/trusty/main/binary-amd64/indigodc-release_1.0.0-1_amd64.deb && dpkg -i indigodc-release_1.0.0-1_amd64.deb [3] https://github.com/indigo-dc/ansible-role-indigo-repo [4] https://github.com/indigo-dc/puppet-indigo-repo

JK-STFC commented 7 years ago

SWAMP is building from source to help its static analyses - it doesn't run the s/w so installing from a repository isn't what is needed, unless I misunderstand what you are saying.

JK-STFC commented 7 years ago

And BTW sorry to Cristina and Michel for not responding earlier, but the responses to this ticket have been going in my Junk Email folder (probably because it looks like it is coming from a robot [because it is], but pretending to come from a person's name).

bkryza commented 7 years ago

Dear John,

below is a complete Dockerfile for creating a container which we use in our CI to compile Oneclient, including commands to add our development repositories at :

FROM ubuntu:16.04

# Get the image up to date
RUN apt-get update && \
    apt-get dist-upgrade -y && \
    apt-get install -y apt-transport-https curl && \
    apt-get clean

# Add additional package sources (couchbase, docker, onedata official repo and deps ppa)
RUN curl -L http://packages.couchbase.com/ubuntu/couchbase.key | apt-key add - && \
    curl -o /etc/apt/sources.list.d/couchbase.list http://packages.couchbase.com/ubuntu/couchbase-ubuntu1404.list && \

    apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
    echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" > /etc/apt/sources.list.d/docker.list && \

    curl -L http://onedata-dev-packages.cloud.plgrid.pl/onedata.gpg.key | apt-key add - && \
    echo "deb http://onedata-dev-packages.cloud.plgrid.pl/apt/ubuntu/xenial xenial main" >> /etc/apt/sources.list.d/onedata.list && \

    apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C211375C && \
    echo "deb http://ppa.launchpad.net/onedata/ceph/ubuntu xenial main" >> /etc/apt/sources.list.d/onedata.list && \
    echo "deb http://ppa.launchpad.net/onedata/build-deps/ubuntu xenial main" >> /etc/apt/sources.list.d/onedata.list

# Install basic packages
RUN apt-get update && \
    apt-get install -y bindfs build-essential coffeescript curl dnsutils \
                       docker-engine=1.12.0-0~xenial erlang erlang-base-hipe \
                       fuse gdb git iputils-ping lcov libcurl4-openssl-dev \
                       libdouble-conversion1v5 libboost-context1.58.0 \
                       libboost-filesystem1.58.0 libboost-iostreams1.58.0 \
                       libboost-log1.58.0 libboost-program-options1.58.0 \
                       libboost-random1.58.0 libboost-system1.58.0 \
                       libboost-thread1.58.0 libbotan1.10 libcouchbase-dev \
                       libgflags2v5 libglobus-common0 libglobus-gsi-callback0 \
                       libglu1 libgoogle-glog0v5 libnspr4 libnss3 libpoco-dev \
                       libpocofoundation9v5 libpoconetssl9v5 libpocoutil9v5 \
                       libpocoxml9v5 libprotobuf9v5 librados2 libsodium18 \
                       libtbb2 nodejs openssl python python-dev \
                       python-protobuf python-rados python-setuptools rsync \
                       rrdtool unzip nfs-common && \
    apt-get clean

# Sidestep an error of Erlang's reltool
RUN rm /usr/lib/erlang/man

# Install required Python packages
RUN easy_install dnspython Flask Flask-SQLAlchemy pytest==2.9.1 \
    pytest-bdd==2.18.0 requests boto rpyc

# Install npm, Ember-CLI and all required packages
RUN npm install -g bower ember-cli jsdoc livereload phantomjs-prebuilt && \
    npm cache clean

# Install rebar3
RUN curl -L https://github.com/erlang/rebar3/releases/download/3.2.0/rebar3 \
         -o /usr/local/bin/rebar3 && \
    chmod +x /usr/local/bin/rebar3

# Add cert bundle (required for server cert validation)
ADD ca-bundle.pem /etc/ssl/cert.pem

# Use caches
RUN mkdir /var/cache/rebar3 && chmod 777 /var/cache/rebar3
VOLUME ["/var/cache/rebar3"]

ADD rebar.config /etc/skel/.config/rebar3/rebar.config
ADD rebar.config /root/.config/rebar3/rebar.config

# Set up the environment
ENV HOME /root
RUN locale-gen en_US.UTF-8
ENV LANG en_US.UTF-8
ENV LANGUAGE en_US:en
ENV LC_ALL en_US.UTF-8

# Install development packages
RUN apt-get update && \
    apt-get install -y autoconf aws-sdk-cpp-s3 bash-completion bc \
                       build-essential ccache cmake debhelper devscripts \
                       doxygen folly-dev g++ golang-go libboost-all-dev \
                       libbotan1.10-dev libfuse-dev libgflags-dev \
                       libglobus-common-dev libglobus-gsi-callback-dev \
                       libgoogle-glog-dev libiberty-dev libltdl-dev \
                       libnspr4-dev libnss3-dev libprotobuf-dev libpython-dev \
                       librados-dev libstdc++-4.9-dev libsodium-dev libtbb-dev \
                       nano ninja-build pkg-config protobuf-compiler \
                       python-sphinx vim rpm subversion swift-sdk-cpp jekyll && \
    apt-get clean

# Use caches
RUN mkdir /var/cache/ccache && chmod 777 /var/cache/ccache
VOLUME ["/var/cache/ccache"]
ENV CCACHE_DIR /var/cache/ccache
ENV CCACHE_UMASK 000
RUN ln -s `which ccache` /usr/local/bin/gcc && \
    ln -s `which ccache` /usr/local/bin/g++ && \
    ln -s `which ccache` /usr/local/bin/cc && \
    ln -s `which ccache` /usr/local/bin/c++

Then to compile Oneclient, you need to follow these steps:

export ONEDATA_GIT_URL=https://github.com/onedata
git clone https://github.com/onedata/oneclient.git
cd oneclient
make release # or debug

Hope this helps...

Bartosz

JK-STFC commented 7 years ago

Yet again it ended up in my Junk Email folder so I missed it.

Wow that is a LOT of setup!

The other services I've been testing you just need to do something like

make release

Is all that stuff below what we do to push oneclient through the other tests we do on Jenkins?

You can add add a few commands to SWAMP to get it to download additional stuff, but it is basically a one-liner and putting all that stuff below on one line might be challenging.

Cheers

JK

From: Bartek Kryza [mailto:notifications@github.com] Sent: Wednesday, April 26, 2017 12:19 PM To: indigo-dc/onedata Cc: Kewley, John (STFC,DL,SC); Mention Subject: Re: [indigo-dc/onedata] Building onceclient for SWAMP (#18)

Dear John, below is a complete Dockerfile for creating a container which we use in our CI to compile Oneclient, including commands to add our development repositories at : FROM ubuntu:16.04

Get the image up to date

RUN apt-get update && \ apt-get dist-upgrade -y && \ apt-get install -y apt-transport-https curl && \ apt-get clean

Add additional package sources (couchbase, docker, onedata official repo and deps ppa)

RUN curl -L http://packages.couchbase.com/ubuntu/couchbase.key | apt-key add - && \ curl -o /etc/apt/sources.list.d/couchbase.list http://packages.couchbase.com/ubuntu/couchbase-ubuntu1404.list && \

apt-key adv --keyserver hkp://p80.pool.sks-keyservers.net:80 --recv-keys 58118E89F3A912897C070ADBF76221572C52609D && \
echo "deb https://apt.dockerproject.org/repo ubuntu-xenial main" > /etc/apt/sources.list.d/docker.list && \

curl -L http://onedata-dev-packages.cloud.plgrid.pl/onedata.gpg.key | apt-key add - && \
echo "deb http://onedata-dev-packages.cloud.plgrid.pl/apt/ubuntu/xenial xenial main" >> /etc/apt/sources.list.d/onedata.list && \

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys C211375C && \
echo "deb http://ppa.launchpad.net/onedata/ceph/ubuntu xenial main" >> /etc/apt/sources.list.d/onedata.list && \
echo "deb http://ppa.launchpad.net/onedata/build-deps/ubuntu xenial main" >> /etc/apt/sources.list.d/onedata.list

Install basic packages

RUN apt-get update && \ apt-get install -y bindfs build-essential coffeescript curl dnsutils \ docker-engine=1.12.0-0~xenial erlang erlang-base-hipe \ fuse gdb git iputils-ping lcov libcurl4-openssl-dev \ libdouble-conversion1v5 libboost-context1.58.0 \ libboost-filesystem1.58.0 libboost-iostreams1.58.0 \ libboost-log1.58.0 libboost-program-options1.58.0 \ libboost-random1.58.0 libboost-system1.58.0 \ libboost-thread1.58.0 libbotan1.10 libcouchbase-dev \ libgflags2v5 libglobus-common0 libglobus-gsi-callback0 \ libglu1 libgoogle-glog0v5 libnspr4 libnss3 libpoco-dev \ libpocofoundation9v5 libpoconetssl9v5 libpocoutil9v5 \ libpocoxml9v5 libprotobuf9v5 librados2 libsodium18 \ libtbb2 nodejs openssl python python-dev \ python-protobuf python-rados python-setuptools rsync \ rrdtool unzip nfs-common && \ apt-get clean

Sidestep an error of Erlang's reltool

RUN rm /usr/lib/erlang/man

Install required Python packages

RUN easy_install dnspython Flask Flask-SQLAlchemy pytest==2.9.1 \ pytest-bdd==2.18.0 requests boto rpyc

Install npm, Ember-CLI and all required packages

RUN npm install -g bower ember-cli jsdoc livereload phantomjs-prebuilt && \ npm cache clean

Install rebar3

RUN curl -L https://github.com/erlang/rebar3/releases/download/3.2.0/rebar3 \ -o /usr/local/bin/rebar3 && \ chmod +x /usr/local/bin/rebar3

Add cert bundle (required for server cert validation)

ADD ca-bundle.pem /etc/ssl/cert.pem

Use caches

RUN mkdir /var/cache/rebar3 && chmod 777 /var/cache/rebar3 VOLUME ["/var/cache/rebar3"]

ADD rebar.config /etc/skel/.config/rebar3/rebar.config ADD rebar.config /root/.config/rebar3/rebar.config

Set up the environment

ENV HOME /root RUN locale-gen en_US.UTF-8 ENV LANG en_US.UTF-8 ENV LANGUAGE en_US:en ENV LC_ALL en_US.UTF-8

Install development packages

RUN apt-get update && \ apt-get install -y autoconf aws-sdk-cpp-s3 bash-completion bc \ build-essential ccache cmake debhelper devscripts \ doxygen folly-dev g++ golang-go libboost-all-dev \ libbotan1.10-dev libfuse-dev libgflags-dev \ libglobus-common-dev libglobus-gsi-callback-dev \ libgoogle-glog-dev libiberty-dev libltdl-dev \ libnspr4-dev libnss3-dev libprotobuf-dev libpython-dev \ librados-dev libstdc++-4.9-dev libsodium-dev libtbb-dev \ nano ninja-build pkg-config protobuf-compiler \ python-sphinx vim rpm subversion swift-sdk-cpp jekyll && \ apt-get clean

Use caches

RUN mkdir /var/cache/ccache && chmod 777 /var/cache/ccache VOLUME ["/var/cache/ccache"] ENV CCACHE_DIR /var/cache/ccache ENV CCACHE_UMASK 000 RUN ln -s which ccache /usr/local/bin/gcc && \ ln -s which ccache /usr/local/bin/g++ && \ ln -s which ccache /usr/local/bin/cc && \ ln -s which ccache /usr/local/bin/c++ Then to compile Oneclient, you need to follow these steps: export ONEDATA_GIT_URL=https://github.com/onedata git clone https://github.com/onedata/oneclient.git cd oneclient make release # or debug Hope this helps... Bartosz — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub, or mute the thread.

groundnuty commented 7 years ago

I'm closing this issue, hoping it was resolved. Please file another and reference it, if needed.