search

Xilinx / PYNQ

Python Productivity for ZYNQ
http://www.pynq.io/
BSD 3-Clause "New" or "Revised" License
1.96k stars 811 forks source link

[sdbuild] failed: ModuleNotFoundError: No module named '_bz2' #643

Closed m-temp closed 5 years ago

m-temp commented 6 years ago

Hi there,

I am trying to build my own sdcard image for pynq_v2.1. After setting everything up successfully (taking Issue #582 into account) the build fails with:

` make[1]: Leaving directory '/root/opencv_build/opencv-3.1.0/build'

Could this be related to Issue #582? I did fix the setup_host.sh, but obviously there is more to fix...

Best regards, Michael

PeterOgden commented 6 years ago

Can you give us more details on your build machine setup? The output of env would also be useful.

m-temp commented 6 years ago

sure. Do you need anything particular? I've tested it on both a new ubuntu16.04 vm and a ubuntu16.04 desktop. In the VM I tested a local vivado installation. On my desktop I have a (university) network installation of vivado.

Here comes the env of my VM:

XDG_VTNR=7 LC_PAPER=de_DE.UTF-8 LC_ADDRESS=de_DE.UTF-8 XDG_SESSION_ID=c1 XDG_GREETER_DATA_DIR=/var/lib/lightdm-data/michael LC_MONETARY=de_DE.UTF-8 CLUTTER_IM_MODULE=xim SESSION=ubuntu GPG_AGENT_INFO=/home/michael/.gnupg/S.gpg-agent:0:1 TERM=xterm-256color VTE_VERSION=4205 XDG_MENU_PREFIX=gnome- SHELL=/bin/bash QT_LINUX_ACCESSIBILITY_ALWAYS_ON=1 WINDOWID=75497482 LC_NUMERIC=de_DE.UTF-8 OLDPWD=/home/michael/PYNQ/sdbuild/scripts UPSTART_SESSION=unix:abstract=/com/ubuntu/upstart-session/1000/993 GNOME_KEYRING_CONTROL= GTK_MODULES=gail:atk-bridge:unity-gtk-module USER=michael LS_COLORS=rs=0:di=01;34:ln=01;36:mh=00:pi=40;33:so=01;35:do=01;35:bd=40;33;01:cd=40;33;01:or=40;31;01:mi=00:su=37;41:sg=30;43:ca=30;41:tw=30;42:ow=34;42:st=37;44:ex=01;32:.tar=01;31:.tgz=01;31:.arc=01;31:.arj=01;31:.taz=01;31:.lha=01;31:.lz4=01;31:.lzh=01;31:.lzma=01;31:.tlz=01;31:.txz=01;31:.tzo=01;31:.t7z=01;31:.zip=01;31:.z=01;31:.Z=01;31:.dz=01;31:.gz=01;31:.lrz=01;31:.lz=01;31:.lzo=01;31:.xz=01;31:.bz2=01;31:.bz=01;31:.tbz=01;31:.tbz2=01;31:.tz=01;31:.deb=01;31:.rpm=01;31:.jar=01;31:.war=01;31:.ear=01;31:.sar=01;31:.rar=01;31:.alz=01;31:.ace=01;31:.zoo=01;31:.cpio=01;31:.7z=01;31:.rz=01;31:.cab=01;31:.jpg=01;35:.jpeg=01;35:.gif=01;35:.bmp=01;35:.pbm=01;35:.pgm=01;35:.ppm=01;35:.tga=01;35:.xbm=01;35:.xpm=01;35:.tif=01;35:.tiff=01;35:.png=01;35:.svg=01;35:.svgz=01;35:.mng=01;35:.pcx=01;35:.mov=01;35:.mpg=01;35:.mpeg=01;35:.m2v=01;35:.mkv=01;35:.webm=01;35:.ogm=01;35:.mp4=01;35:.m4v=01;35:.mp4v=01;35:.vob=01;35:.qt=01;35:.nuv=01;35:.wmv=01;35:.asf=01;35:.rm=01;35:.rmvb=01;35:.flc=01;35:.avi=01;35:.fli=01;35:.flv=01;35:.gl=01;35:.dl=01;35:.xcf=01;35:.xwd=01;35:.yuv=01;35:.cgm=01;35:.emf=01;35:.ogv=01;35:.ogx=01;35:.aac=00;36:.au=00;36:.flac=00;36:.m4a=00;36:.mid=00;36:.midi=00;36:.mka=00;36:.mp3=00;36:.mpc=00;36:.ogg=00;36:.ra=00;36:.wav=00;36:.oga=00;36:.opus=00;36:.spx=00;36:.xspf=00;36: LC_TELEPHONE=de_DE.UTF-8 QT_ACCESSIBILITY=1 XDG_SESSION_PATH=/org/freedesktop/DisplayManager/Session0 UNITY_HAS_3D_SUPPORT=false XDG_SEAT_PATH=/org/freedesktop/DisplayManager/Seat0 SSH_AUTH_SOCK=/run/user/1000/keyring/ssh DEFAULTS_PATH=/usr/share/gconf/ubuntu.default.path XDG_CONFIG_DIRS=/etc/xdg/xdg-ubuntu:/usr/share/upstart/xdg:/etc/xdg UNITY_DEFAULT_PROFILE=unity-lowgfx DESKTOP_SESSION=ubuntu PATH=/opt/Xilinx/DocNav:/opt/Xilinx/Vivado/2017.4/bin:/opt/Xilinx/SDK/2017.4/bin:/opt/Xilinx/SDK/2017.4/gnu/microblaze/lin/bin:/opt/Xilinx/SDK/2017.4/gnu/arm/lin/bin:/opt/Xilinx/SDK/2017.4/gnu/microblaze/linux_toolchain/lin64_le/bin:/opt/Xilinx/SDK/2017.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin:/opt/Xilinx/SDK/2017.4/gnu/aarch32/lin/gcc-arm-none-eabi/bin:/opt/Xilinx/SDK/2017.4/gnu/aarch64/lin/aarch64-linux/bin:/opt/Xilinx/SDK/2017.4/gnu/aarch64/lin/aarch64-none/bin:/opt/Xilinx/SDK/2017.4/gnu/armr5/lin/gcc-arm-none-eabi/bin:/opt/Xilinx/SDK/2017.4/tps/lnx64/cmake-3.3.2/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin QT_IM_MODULE=ibus QT_QPA_PLATFORMTHEME=appmenu-qt5 LC_IDENTIFICATION=de_DE.UTF-8 XDG_SESSION_TYPE=x11 PWD=/home/michael/PYNQ/sdbuild JOB=unity-settings-daemon XMODIFIERS=@im=ibus GNOME_KEYRING_PID= LANG=en_US.UTF-8 GDM_LANG=en_US MANDATORY_PATH=/usr/share/gconf/ubuntu.mandatory.path LC_MEASUREMENT=de_DE.UTF-8 COMPIZ_CONFIG_PROFILE=ubuntu IM_CONFIG_PHASE=1 XILINX_VIVADO=/opt/Xilinx/Vivado/2017.4 GDMSESSION=ubuntu SESSIONTYPE=gnome-session GTK2_MODULES=overlay-scrollbar SHLVL=1 HOME=/home/michael XDG_SEAT=seat0 LANGUAGE=en_US LIBGL_ALWAYS_SOFTWARE=1 GNOME_DESKTOP_SESSION_ID=this-is-deprecated UPSTART_INSTANCE= UPSTART_EVENTS=xsession started XDG_SESSION_DESKTOP=ubuntu LOGNAME=michael COMPIZ_BIN_PATH=/usr/bin/ DBUS_SESSION_BUS_ADDRESS=unix:abstract=/tmp/dbus-Rbn8zpQoDZ XDG_DATA_DIRS=/usr/share/ubuntu:/usr/share/gnome:/usr/local/share:/usr/share:/var/lib/snapd/desktop:/var/lib/snapd/desktop QT4_IM_MODULE=xim LESSOPEN=| /usr/bin/lesspipe %s INSTANCE= UPSTART_JOB=unity7 XDG_RUNTIME_DIR=/run/user/1000 DISPLAY=:0 XDG_CURRENT_DESKTOP=Unity GTK_IM_MODULE=ibus LESSCLOSE=/usr/bin/lesspipe %s %s LC_TIME=de_DE.UTF-8 LC_NAME=deDE.UTF-8 XAUTHORITY=/home/michael/.Xauthority =/usr/bin/env

and here the ENV of my desktop:

MODULE_VERSION_STACK=3.2.8 LC_PAPER=de_DE.UTF-8 XDG_SESSION_ID=16 LC_ADDRESS=de_DE.UTF-8 LC_MONETARY=de_DE.UTF-8 TERM=xterm SHELL=/bin/bash SSH_CLIENT=10.152.249.213 33806 22 LC_NUMERIC=de_DE.UTF-8 SSH_TTY=/dev/pts/9 USER=gu95jil LC_TELEPHONE=de_DE.UTF-8 MODULE_VERSION=3.2.8 MAIL=/var/mail/gu95jil PATH=/nfs/tools/xilinx/SDK/2017.4/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/microblaze/lin/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/arm/lin/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/microblaze/linux_toolchain/lin64_le/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/aarch32/lin/gcc-arm-linux-gnueabi/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/aarch32/lin/gcc-arm-none-eabi/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/aarch64/lin/aarch64-linux/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/aarch64/lin/aarch64-none/bin:/nfs/tools/xilinx/SDK/2017.4/gnu/armr5/lin/gcc-arm-none-eabi/bin:/nfs/tools/xilinx/SDK/2017.4/tps/lnx64/cmake-3.3.2/bin:/nfs/tools/xilinx/Vivado/2017.4/bin:/nfs/tools/xilinx/DocNav:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/opt/qemu/bin:/opt/crosstool-ng/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin QT_QPA_PLATFORMTHEME=appmenu-qt5 LC_IDENTIFICATION=de_DE.UTF-8 PWD=/home/gu95jil LMFILES=/nfs/tools/environment_modules/3.2.8/modulefiles/xilinx/vivado/2017.4 LANG=en_US.UTF-8 MODULEPATH=/nfs/tools/environment_modules/versions:/nfs/tools/environment_modules/$MODULE_VERSION/modulefiles:/nfs/tools/environment_modules/modulefiles LOADEDMODULES=xilinx/vivado/2017.4 LC_MEASUREMENT=de_DE.UTF-8 XILINX_VIVADO=/nfs/tools/xilinx/Vivado/2017.4 LM_LICENSE_FILE=5280@cadencelic.regent.e-technik.tu-muenchen.de:2100@xilinxlic.regent.e-technik.tu-muenchen.de:1717@mentorlic.regent.e-technik.tu-muenchen.de:1702@tueisec-licenses.sec.ei.tum.de:2300@licenses.lis.ei.tum.de:27000@licenses.lis.ei.tum.de: KRB5CCNAME=FILE:/tmp/krb5cc_3164845_zrqHF6lJwE SHLVL=1 HOME=/home/gu95jil LOGNAME=gu95jil XDG_DATA_DIRS=/usr/local/share:/usr/share:/var/lib/snapd/desktop SSH_CONNECTION=10.152.249.213 33806 10.152.249.103 22 MODULESHOME=/nfs/tools/environment_modules/3.2.8 XDG_RUNTIME_DIR=/run/user/3164845 LC_TIME=de_DE.UTF-8 LC_NAME=de_DE.UTF-8 BASH_FUNC_module%%=() { eval /nfs/tools/environment_modules/$MODULE_VERSION/bin/modulecmd bash $* } _=/usr/bin/env

PeterOgden commented 6 years ago

That all looks reasonable - can you try building again but capturing the entire build log? I find it easiest to do make 2>&1 | tee build.log.

PeterOgden commented 6 years ago

Scratch that - it appears to be a matplotlib issue https://github.com/matplotlib/matplotlib/issues/10866 which has been fixed upstream but still waiting to be packaged.

Can you try modifying sdbuild/packages/python_packages/qemu.sh and change matplotlib to matplotlib==2.1.2 (line 16)?

m-temp commented 6 years ago

Ok, with this patch the error is gone. However it is very surprising that a "make" in "image_v2.1" does create different images on each run. Actually I expected to rebuild image_v2.1 and that I don't have different version after each run. It seams also that there is a new kernel with options not covered in your .config. Therefore make 2>&1 | tee log . fails because it is then unable to read stdin.

This is a problem, because now, there is a new error at the end of the build process without a clear error message:

Processing /home/xilinx/pynq_git Installing collected packages: pynq Running setup.py install for pynq ... done Successfully installed pynq-2.1 You are using pip version 9.0.1, however version 10.0.1 is available. You should consider upgrading via the 'pip install --upgrade pip' command.

Obviously my hostname ist not pynq and there is no user xilinx, but I'd expected that to be happen inside the chroot? "Please manually reboot the board" also sounds wired as an error during the build process. Should I open a new issue for that?

PeterOgden commented 6 years ago

The reboot message is the result of the script we use to change the hostname inside the chroot so you can ignore that - the reason you see things change back is because changing the host name inside the chroot also changes (temporarily) the hostname of the host.

The xilinx user is created very early on in the image creation process just after all of the Ubuntu packages get installed. One thing to be aware of is that make doesn't remove image files when the build fails (because if it did they couldn't be unmounted easily) so it's possible that on a previous run the first stage of the build failed resulting in an intermediate images that doesn't have the xilinx user added.

Can you try running a make clean before running make and see if the same thing happens?

We're also in the process of changing this flow quite drastically to better support additional boards and move to Ubuntu 18.04. It runs a lot faster if that's of interest to you. The branch we're doing development on is in my fork - https://github.com/PeterOgden/PYNQ/tree/sdbuild_refactor

m-temp commented 6 years ago

nope. Make clean does not make any difference. I can try to produce a logfile at the next run....

m-temp commented 6 years ago

However in my VM it worked - don't know why it's not working on my workstation. I will provide you the build.log as soon as it fails again...

m-temp commented 6 years ago

Here comes the log - and the log is too big for github...

https://syncandshare.lrz.de/dl/fiNLReqxJx47b4EGC6mJHgT

PeterOgden commented 6 years ago

I don't know what's going on. Can you sudo losetup -a and make sure nothing's mounted before the system starts? Failing that can you mount the final image and the .build/rootfs-stage1.img and dump the contents of /etc/passwd and see of the xilinx user is in them?

m-temp commented 6 years ago

The xilinx user is there. And it also generates the rootfs.img inside the ./sdbuild-Folder. So it looks like everything works expect the unmounting.. I can live with that, however it's not ideal.