search

rock-simulation / pybob

Python build scripts
4 stars 7 forks source link

Bootstrap does not work #1

Closed AlexanderFabisch closed 7 years ago

AlexanderFabisch commented 7 years ago

I use the following bootstrap script:

BUILDCONF_URL=... ./bootstrap_bolero.sh
#! /bin/bash

mkdir bolero-dev
cd bolero-dev
DEV_DIR="$( cd "$( dirname "$0" )" && pwd )"
git clone https://github.com/rock-simulation/pybob.git
cd pybob

# create default config for bolero
echo "autoprojEnv: false" > pybob.yml
echo "buildconfAddress: $BUILDCONF_URL" >> pybob.yml
echo "buildconfBranch: ''" >> pybob.yml
echo "defBuildType: debug" >> pybob.yml
echo "devDir: ${DEV_DIR}" >> pybob.yml
echo "pyScriptDir: ${DEV_DIR}/pybob" >> pybob.yml
echo "rockFlavor: master" >> pybob.yml

# clone build configuration
./pybob.py buildconf

cd ..
source bobenv.sh

# build default packages
cd pybob
./pybob.py bootstrap

echo ""
echo "To continue working with bolero in this terminal perform:"
echo ""
echo "  cd bolero-dev"
echo "  source bobenv.sh"
echo ""
echo 'Whenever you open a new terminal to work with bolero switch into bolero-dev and source the "env.sh" again to activate the bolero install for that terminal.'
echo ""

pybob creates a file bobenv.sh. The second line is

. env.sh

(1) The file env.sh does not exist.

I added the file before source bobenv.sh. It should contain at least

export PATH="/home/afabisch/tmp/test/bolero-dev/install/bin":$PATH
export LD_LIBRARY_PATH="/home/afabisch/tmp/test/bolero-dev/install/lib":$LD_LIBRARY_PATH
if [ x${PKG_CONFIG_PATH} = x ]; then
  export PKG_CONFIG_PATH="/home/afabisch/tmp/test/bolero-dev/install/lib/pkgconfig"
else
  export PKG_CONFIG_PATH="/home/afabisch/tmp/test/bolero-dev/install/lib/pkgconfig":"$PKG_CONFIG_PATH"
fi
export PYTHONPATH=$PYTHONPATH:"/home/afabisch/tmp/test/bolero-dev/install/lib/python2.7/dist-packages":"/home/afabisch/tmp/test/bolero-dev/install/lib/python2.7/site-packages"

Otherwise the build will fail.

(2) The section "Install MARS via pybob" of the readme does not work either.

(3) At the moment, you can only call source bobenv.sh directly from the directory of bobenv.sh because of the line . env.sh. It should be replaced by something like

CURRENT_DIR=`pwd`
# go to directory of bobenv.sh, I don't know how to do that at the moment
cd $CURRENT_DIR

(4) For some reason I can only call pybob functions in the pybob directory. Probably because the pybob.yml is located there. This was previously not the case.

malter commented 7 years ago

The described behavior occurs if you bootstrap while having the env.sh already sourced form another dev folder. I think in case of another dev folder you will get an error message pointing to the problem but I assume that you sourced in a dev folder that you removed afterwards and than you bootstrap again. If the AUTOPROJ_CURRENT_ROOT is set probably (which might be the case in your setup) pybob thinks you are going to install it into an existing autoproj bootstrap. In that case the "env.sh" is managed by autoproj and pybob does an extension via the "bobenv.sh". I will check that special case and try to deal with it.

AlexanderFabisch commented 7 years ago

Does ROS maybe interfere with its environment variables? I tried to clear all my environment variables. These are my environment variables:

$ echo $AUTOPROJ_CURRENT_ROOT

$ echo $PATH /home/afabisch/bin/pycharm-community-2016.1/bin/:/usr/local/cuda-7.5/bin:/home/afabisch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin $ echo $LD_LIBRARY_PATH /usr/local/cuda-7.5/lib64

$ ./bootstrap_bolero.sh 
Cloning into 'pybob'...
remote: Counting objects: 324, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 324 (delta 19), reused 0 (delta 0), pack-reused 286
Receiving objects: 100% (324/324), 76.14 KiB | 0 bytes/s, done.
Resolving deltas: 100% (203/203), done.
Checking connectivity... done.
   Fetching "git@git.hb.dfki.de:team-learning/bolero_buildconf.git" into /home/afabisch/tmp/test/bolero-dev/autoproj
  Fetching: git@git.hb.dfki.de:team-learning/package_set.git
  Fetching: https://github.com/rock-simulation/package_set.git
  Fetching: git@git.hb.dfki.de:rock-dfki/package_set.git
  Fetching: https://github.com/rock-core/rock-package_set.git
  Fetching: https://github.com/rock-core/package_set.git
  Fetching: https://github.com/orocos-toolchain/autoproj.git
Installed packages: 
[]
Time: 0:00:10.684725
Check: learning/bolero ...  Fetching /home/afabisch/tmp/test/bolero-dev/learning/bolero ...  check for patches done
Check: learning/bolero/src/controller ...  done
Check: learning/bolero/src/bl_loader ...  done
Check: learning/bolero/src/environment/cec13_test_functions ...  done
Check: learning/bolero/src/optimizer/cmaes_optimizer ...  done
Check: learning/bolero/include ...  done
Check: tools/configmaps ...  Fetching /home/afabisch/tmp/test/bolero-dev/tools/configmaps ...  check for patches done
Check: simulation/lib_manager ...  Fetching /home/afabisch/tmp/test/bolero-dev/simulation/lib_manager ...  check for patches done
Check: yaml-cpp ...  done
Check: base/cmake ...  Fetching /home/afabisch/tmp/test/bolero-dev/base/cmake ...  check for patches done
Check: pkg-config ...  done
Check: cmake ...  done
Check: python ...  done
Check: python-numpy ...  done
Check: tools/catch ...  Fetching /home/afabisch/tmp/test/bolero-dev/tools/catch ...  check for patches patch -N -p0 -d /home/afabisch/tmp/test/bolero-dev/tools/catch -i /home/afabisch/tmp/test/bolero-dev/pybob/patches/catch.patch
patching file CMakeLists.txt
patching file catch.pc.in

0
done

Check: learning/bolero ...  done
Check: learning/bolero/src/controller ...  done
Check: learning/bolero/src/bl_loader ...  done
Check: learning/bolero/src/environment/cec13_test_functions ...  done
Check: learning/bolero/src/optimizer/cmaes_optimizer ...  done
Install: ['learning/bolero/include', 'simulation/lib_manager', 'python', 'python-numpy', 'tools/catch', 'yaml-cpp', 'pkg-config', 'cmake']
learning/bolero/include configure error
simulation/lib_manager configure error
tools/catch configure error
Install: ['learning/bolero/src/bl_loader', 'base/cmake', 'learning/bolero']
learning/bolero/src/bl_loader configure error
base/cmake configure error
learning/bolero configure error
Install: ['tools/configmaps']
tools/configmaps configure error
Install: ['learning/bolero/src/controller', 'learning/bolero/src/environment/cec13_test_functions', 'learning/bolero/src/optimizer/cmaes_optimizer']
learning/bolero/src/environment/cec13_test_functions configure error
learning/bolero/src/controller configure error
learning/bolero/src/optimizer/cmaes_optimizer configure error

Errors:
 - configure: learning/bolero/include
 - configure: simulation/lib_manager
 - configure: tools/catch
 - configure: learning/bolero/src/bl_loader
 - configure: base/cmake
 - configure: learning/bolero
 - configure: tools/configmaps
 - configure: learning/bolero/src/environment/cec13_test_functions
 - configure: learning/bolero/src/controller
 - configure: learning/bolero/src/optimizer/cmaes_optimizer
Installed packages: 
[]
Time: 0:00:12.507454

To continue working with bolero in this terminal perform:

  cd bolero-dev
  source bobenv.sh

Whenever you open a new terminal to work with bolero switch into bolero-dev and source the "env.sh" again to activate the bolero install for that terminal.
malter commented 7 years ago

I tried your bootstrap script on os x. I had to rename bobenv.sh in your script by env.sh and everything installed just fine. Can you open a new terminal. Switch into a new empty folder and post the whole output of export? Afterwards perform the bootstrap script from there while replacing bobenv.sh with env.sh. Then post the output and keep the folder untouched. So that we can try to find the error of the configuration in that exact folder.

malter commented 7 years ago

The installation guide for mars still works on OS X. If we can't find the problem on your system I might have to setup a VBox and try to reproduce your system configuration. By the way, what is your os version?

AlexanderFabisch commented 7 years ago

At the moment everything works fine on Ubuntu 16.04 here. (1), (2), (4) are kind of solved. I used Ubuntu 14.04 previously, I will check it again later.

However, (3) does not work. Probably it is just outdated?

$ cat bootstrap_mars.sh
mkdir mars-dev
cd mars-dev
git clone git@github.com:rock-simulation/pybob.git
cd pybob
./pybob.py buildconf path="git@github.com:rock-simulation/simulation-buildconf.git"
cd ..
source env.sh

bob-fetch
bob-install
$ ./bootstrap_mars.sh 
Klone nach 'pybob' ...
remote: Counting objects: 324, done.
remote: Compressing objects: 100% (38/38), done.
remote: Total 324 (delta 19), reused 0 (delta 0), pack-reused 286
Empfange Objekte: 100% (324/324), 76.14 KiB | 0 bytes/s, Fertig.
Löse Unterschiede auf: 100% (203/203), Fertig.
Prüfe Konnektivität ... Fertig.
You must set a root directory where all repositories will be checked out and all packages will be installed
Enter root directory or nothing for ["/home/dfki.uni-bremen.de/afabisch/Projekte/mars-dev"]:  

You can specify the number of CORES you want to use when compiling packages.
Enter number of CORES [16]):  4

Enter default build type (debug|release) [debug]:  

Enter default rock flavor (stable|master) [master]:  

Enter branch of buildconf [default]:  ^CTraceback (most recent call last):
  File "./pybob.py", line 41, in <module>
    config.getConfiguration(cfg)
  File "/home/dfki.uni-bremen.de/afabisch/Projekte/mars-dev/pybob/config.py", line 101, in getConfiguration
    in_ = raw_input_("Enter branch of buildconf [default]: ")
  File "/home/dfki.uni-bremen.de/afabisch/Projekte/mars-dev/pybob/config.py", line 13, in raw_input_
    return raw_input()
KeyboardInterrupt
./bootstrap_mars.sh: Zeile 7: env.sh: Datei oder Verzeichnis nicht gefunden
./bootstrap_mars.sh: Zeile 9: bob-fetch: Befehl nicht gefunden
./bootstrap_mars.sh: Zeile 10: bob-install: Befehl nicht gefunden
$ env
XDG_VTNR=7
XDG_SESSION_ID=c2
CLUTTER_IM_MODULE=xim
XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/afabisch
SESSION=ubuntu
GPG_AGENT_INFO=/home/dfki.uni-bremen.de/afabisch/.gnupg/S.gpg-agent:0:1
_PMT_DEBUG_LEVEL=0
TERM=xterm-256color
SHELL=/bin/bash
VTE_VERSION=4205
QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1
WINDOWID=62914805
OLDPWD=/
UPSTART_SESSION=unix:abstract=/com/ubuntu/upstart-session/1545608986/3798
GNOME_KEYRING_CONTROL=
GTK_MODULES=gail:atk-bridge:unity-gtk-module
CUDA_HOME=/usr/local/cuda-7.5
USER=afabisch
LD_LIBRARY_PATH=:/usr/local/cuda-7.5/lib64:/usr/local/lib
LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:*.tar=01;31:*.tgz=01;31:*.arc=01;31:*.arj=01;31:*.taz=01;31:*.lha=01;31:*.lz4=01;31:*.lzh=01;31:*.lzma=01;31:*.tlz=01;31:*.txz=01;31:*.tzo=01;31:*.t7z=01;31:*.zip=01;31:*.z=01;31:*.Z=01;31:*.dz=01;31:*.gz=01;31:*.lrz=01;31:*.lz=01;31:*.lzo=01;31:*.xz=01;31:*.bz2=01;31:*.bz=01;31:*.tbz=01;31:*.tbz2=01;31:*.tz=01;31:*.deb=01;31:*.rpm=01;31:*.jar=01;31:*.war=01;31:*.ear=01;31:*.sar=01;31:*.rar=01;31:*.alz=01;31:*.ace=01;31:*.zoo=01;31:*.cpio=01;31:*.7z=01;31:*.rz=01;31:*.cab=01;31:*.jpg=01;35:*.jpeg=01;35:*.gif=01;35:*.bmp=01;35:*.pbm=01;35:*.pgm=01;35:*.ppm=01;35:*.tga=01;35:*.xbm=01;35:*.xpm=01;35:*.tif=01;35:*.tiff=01;35:*.png=01;35:*.svg=01;35:*.svgz=01;35:*.mng=01;35:*.pcx=01;35:*.mov=01;35:*.mpg=01;35:*.mpeg=01;35:*.m2v=01;35:*.mkv=01;35:*.webm=01;35:*.ogm=01;35:*.mp4=01;35:*.m4v=01;35:*.mp4v=01;35:*.vob=01;35:*.qt=01;35:*.nuv=01;35:*.wmv=01;35:*.asf=01;35:*.rm=01;35:*.rmvb=01;35:*.flc=01;35:*.avi=01;35:*.fli=01;35:*.flv=01;35:*.gl=01;35:*.dl=01;35:*.xcf=01;35:*.xwd=01;35:*.yuv=01;35:*.cgm=01;35:*.emf=01;35:*.ogv=01;35:*.ogx=01;35:*.aac=00;36:*.au=00;36:*.flac=00;36:*.m4a=00;36:*.mid=00;36:*.midi=00;36:*.mka=00;36:*.mp3=00;36:*.mpc=00;36:*.ogg=00;36:*.ra=00;36:*.wav=00;36:*.oga=00;36:*.opus=00;36:*.spx=00;36:*.xspf=00;36:
QT_ACCESSIBILITY=1
MUJOCO_PY_MJPRO_PATH=/home/dfki.uni-bremen.de/afabisch/bin/mjpro131
XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0
XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0
SSH_AUTH_SOCK=/run/user/1545608986/keyring/ssh
DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path
XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg
PATH=/home/dfki.uni-bremen.de/afabisch/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin:/usr/local/cuda-7.5/bin
DESKTOP_SESSION=ubuntu
QT_IM_MODULE=ibus
QT_QPA_PLATFORMTHEME=appmenu-qt5
QUALISYSPATH=/home/dfki.uni-bremen.de/afabisch/Data/Qualisys/Throwing
XDG_SESSION_TYPE=x11
JOB=unity-settings-daemon
PWD=/home/dfki.uni-bremen.de/afabisch/Projekte
XMODIFIERS=@im=ibus
EDITOR=vim
MUJOCO_PY_MJKEY_PATH=/home/dfki.uni-bremen.de/afabisch/Downloads/mjkey.txt
GNOME_KEYRING_PID=
LANG=de_DE.UTF-8
GDM_LANG=de_DE
MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path
IM_CONFIG_PHASE=1
COMPIZ_CONFIG_PROFILE=ubuntu
GDMSESSION=ubuntu
KRB5CCNAME=FILE:/tmp/krb5cc_1545608986_lBrsXf
SESSIONTYPE=gnome-session
GTK2_MODULES=overlay-scrollbar
HOME=/home/dfki.uni-bremen.de/afabisch
XDG_SEAT=seat0
SHLVL=1
LANGUAGE=de_DE
GNOME_DESKTOP_SESSION_ID=this-is-deprecated
UPSTART_INSTANCE=
UPSTART_EVENTS=xsession started
XDG_SESSION_DESKTOP=ubuntu
LOGNAME=afabisch
COMPIZ_BIN_PATH=/usr/bin/
QT4_IM_MODULE=xim
XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share/:/usr/share/:/var/lib/snapd/desktop
DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-6cr1VfS7mI
LESSOPEN=| /usr/bin/lesspipe %s
INSTANCE=
UPSTART_JOB=unity7
XDG_RUNTIME_DIR=/run/user/1545608986
DISPLAY=:0
RUBYOPT=rubygems
XDG_CURRENT_DESKTOP=Unity
GTK_IM_MODULE=ibus
LESSCLOSE=/usr/bin/lesspipe %s %s
XAUTHORITY=/home/dfki.uni-bremen.de/afabisch/.Xauthority
_=/usr/bin/env
malter commented 7 years ago

The problem with (3) still exists. But only if you try to extend an autoproj installation with pybob, which is pretty untested yet. ;-)

AlexanderFabisch commented 7 years ago

OK, I know why pybob thought I would have a rock installation: I had autoproj installed at /usr/local/bin/autoproj.

malter commented 7 years ago

autoproj installed into your system argh

pybob never thought about that option. ;-)

AlexanderFabisch commented 7 years ago

I don't know how that happened. :)