search

ilg-archived / openocd

The GNU MCU Eclipse OpenOCD
http://gnuarmeclipse.github.io/openocd/
GNU General Public License v2.0
234 stars 62 forks source link

Feedback: pre-build requirements #21

Closed TommyMurphyTM1234 closed 6 years ago

TommyMurphyTM1234 commented 7 years ago

I ran through the build process on a clean Ubuntu 16.04 64 bit installation. In case it helps to refine the installation instructions at all I found that the script failed a few times until I installed each of the following in turn:

sudo apt install automake sudo apt install texinfo sudo apt install makeinfo

Maybe it's too much trouble to enumerate all required packages for all possible build platforms in which case running the script and installing things on a trial and error basis is probably OK?

Curiously the GCC build script did not complain about these tools being missing and did not install them (otherwise they would still be available now for the openocd script) so I don't know if it uses and installs them in the docker container while the openocd script installs and uses them on the host or something?!

One other thing - at the very end of the build process I got this but I don't know if it's significant?

/Host/Work/openocd/scripts/build-helper.sh: line 743: DEPLOY_FOLDER_NAME: unbound variable

ilg-ul commented 7 years ago

the build process on a clean Ubuntu 16.04 64 bit installation. ... the script failed

that's bad. the script has (had?) some tests at the beginning to check the prerequisites, and should not start if there is something missing.

could you list the executables that failed, to extend the initial tests?

installing things on a trial and error basis is probably OK?

in my opinion it is not ok, the script should complain from the beginning.

DEPLOY_FOLDER_NAME

that's bad too, it means something was wrong in the first part of the script, where variables are set.

ilg-ul commented 7 years ago

while the openocd script installs and uses them on the host

good point. I thought most of the interesting actions happen inside the container, where things should already be present.

could you remove those packages from the ubuntu host, retry the script, and take a closer look where things break? if the actions are ok to happen on the host, then the executables need to be tested at the beginning. if the actions are executed on the host by mistake, the script needs to be fixed.

TommyMurphyTM1234 commented 7 years ago

Yes - I will do this later on and post feedback on it.

TommyMurphyTM1234 commented 7 years ago

Apologies for the delay on this but I only got to look at it again now. I double checked with a clean (VM) install of Ubuntu 16.04 64 bit desktop, nothing additional installed other than available updates. I found that the script would not run successfully until the following were installed on the host:

Perhaps it is a mistake that the script looks for these on the host rather than in the docker container(s)?

Also as mentioned above at the end of the build process I got this again:

Changing owner to non-root Linux user...
/Host/Work/openocd/scripts/build-helper.sh: line 743: DEPLOY_FOLDER_NAME: unbound variable

P.S. one other small thing is that the website instructions refer to the working folder as "work" when, in fact, it is "Work" (e.g. ~/Work).

More detailed info if needed:

I started with a clean (VM) install of Ubuntu 16.04 64 bit desktop mode. I didn't add anything other than installing all available updates.

I followed these instructions for installing/configuring docker:

https://gnu-mcu-eclipse.github.io/developer/build-binaries-prerequisites/

The RISC-V tools build instructions link to this page and also to the docker installation instructions but the OpenOCD build instructions only link to the latter - the former are simpler to follow so maybe the OpenOCD page could also link to them?

I then followed the instructions here for building OpenOCD. While the page does mention some required packages it also says that the script should check for and install them if necessary so I didn't manually install any packages at this stage. I simply ran:

bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

This failed because automake was not installed (note that the RISC-V build script did not fail on this):

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:15:14 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...

So I installed automake:

sudo apt-get install automake

and ran the script again but this time it failed on makeinfo:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:15:47 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Checking host makeinfo...
/home/user/Downloads/openocd-build.git/scripts/build.sh: line 444: makeinfo: command not found

So I installed texinfo which provides makeinfo:

sudo apt-get install texinfo

I ran the script again but this time it failed on libtool:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Fri Sep  1 18:16:41 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.1-ce, build 874a737

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.

Checking host makeinfo...
texi2any (GNU texinfo) 6.1
/home/user/Downloads/openocd-build.git/scripts/build.sh: Error: libtool is required

So I installed libtool

sudo apt-get install libtool

and ran the script again - this time it worked.

Hope this helps.

ilg-ul commented 7 years ago

thank you for your detailed message. I'll take a look and try to fix this, but right now I'm very busy with the future project template.

TommyMurphyTM1234 commented 7 years ago

OK - thanks. Just to clarify is it a mistake that the script is trying to find these tools on the host rather than in the docker container? If so and I can spot the difference between the RISC-V GCC build script and the OpenOCD build script I'll see if I can do anything to address this issue (and the unbound variable issue).

ilg-ul commented 7 years ago

Just to clarify is it a mistake that the script is trying to find these tools on the host rather than in the docker container?

I guess the problem is a wrong assumption that the script runs on macOS. in this case the script should check the prerequisites on the host, since there are no macOS Docker containers.

probably a pair of if/fi to test the host will solve the problem.

TommyMurphyTM1234 commented 7 years ago

I guess the problem is a wrong assumption that the script runs on macOS. in this case the script should check the prerequisites on the host, since there are no macOS Docker containers.

probably a pair of if/fi to test the host will solve the problem.

Yes - that seems to be the problem alright - I commented out the calls to/checks for automake, makeinfo and libtool and did a clean build and it worked fine on Ubuntu. So I guess that these calls/checks are macOS specific. I thought that comparing these to the RISC-V GCC tools build script would make it obvious how to wrap them in a host test but that script doesn't have such a test and yet works fine. I'm also not clear on how to fix the unbound variable issue. So I will leave it to you to make the necessary changes whenever you get time. Thanks for the feedback on this.

ilg-ul commented 7 years ago

I updated the build script and the helper, be sure you update the git and the submodule.

I also updated the build page, which refers to the global Build Prerequisites page.

the page defines the packages required on Ubuntu. it would be great to try again with a fresh Ubuntu with those packages.

I also added the missing DEPLOY_FOLDER_NAME.

please remove the openocd build folder and try again.

ilg-ul commented 7 years ago

one small note: the checkout-stable and checkout-dev are not implemented, you need to manually checkout the desired branch.

ilg-ul commented 6 years ago

is this issue still actual?

TommyMurphyTM1234 commented 6 years ago

Hi Liviu

There still seem to be problems as far as I can see.

I started with a clean Ubuntu 16.04 64 bit VM installation with the latest VirtualBox guest extensions and all available Ubuntu updates installed - nothing else manually installed.

I pulled the latest (I presume?) script as follows:

git clone --recurse-submodules https://github.com/gnu-mcu-eclipse/openocd-build.git ~/Downloads/openocd-build.git

I kicked off a build of the win32 and deb32 targets but only got this far before the script terminated:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh --win32 --deb32

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Tue Sep 26 10:21:30 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking Docker...
Docker version 17.06.2-ce, build cec0b72

Checking host automake...

I installed automake on the Ubuntu host and kicked the build off again but I got this (not sure if this is significant?):

Copying openocd-build.git from Downloads...
fatal: Not a git repository (or any of the parent directories): .git

and then it failed with this:

bootstrap...
./bootstrap: Error: libtool is required

I installed libtool and tried again but again it failed with an error that seems to relate to the docker container (/Host/Work/...) so I'm not really sure how to work around that.

Running OpenOCD configure...
bash: /Host/Work/openocd/openocd.git/configure: No such file or directory

I don't think that I am doing anything wrong but if I am please let me know. Thanks.

Hope this helps Regards Tommy

TommyMurphyTM1234 commented 6 years ago

Looking at the contents of ~/Work/openocd/openocd.git the problem looks to be that there is no configure script because bootstrap has not been run first.

ilg-ul commented 6 years ago

I'll take a look and fix it during the next few days. (I had a similar problem with the gcc script, the changes are small, I need to identify them and apply to this script too).

I might also change the configuration to use git submodules instead of downloding the helper, at least this keeps the scripts in sync.

ilg-ul commented 6 years ago

bootstrap has not been run first

bootstrap was somewhere in the script, probably something went wrong before trying to run it.

TommyMurphyTM1234 commented 6 years ago

bootstrap was somewhere in the script, probably something went wrong before trying to run it.

build.sh does not call bootstrap for openocd. I modified ~/Downloads/openocd.git/scripts/build.sh to add this:

if [ "${target_name}" == "win" ]
  then

    cd "${build_folder_path}/openocd"

    bash "${work_folder_path}/${OPENOCD_FOLDER_NAME}/bootstrap"

but then it fails with this:

Running OpenOCD configure...
+ aclocal
aclocal: error: 'configure.ac' is required

I'm not sure why because ~/Work/openocd/openocd.git/configure.ac does exist.

ilg-ul commented 6 years ago

build.sh does not call bootstrap for openocd

it looks like I messed the script recently, I remember before it was there.

TommyMurphyTM1234 commented 6 years ago

Actually I may be wrong. The script does seem to call openocd's bootstrap but only if build.sh bootstrap is called. Am I supposed to build as follows?

build.sh bootstrap 
build.sh --win32

I tried that but still get the error with configure.ac.

ilg-ul commented 6 years ago

I'm a bit busy now, I'll try to fix it tomorrow afternoon.

TommyMurphyTM1234 commented 6 years ago

OK - thanks.

Is the "if false ..." snippet correct here or could this be relevant to the current problems?

# ----- Process "pull|checkout-dev|checkout-stable" actions. -----

do_repo_action() {

  # $1 = action (pull, checkout-dev, checkout-stable)

  # Update current branch and prepare autotools.
  echo
  if [ "${ACTION}" == "pull" ]
  then
    echo "Running git pull..."
  elif [ "${ACTION}" == "checkout-dev" ]
  then
    echo "Running git checkout gnu-mcu-eclipse-dev & pull..."
  elif [ "${ACTION}" == "checkout-stable" ]
  then
    echo "Running git checkout gnu-mcu-eclipse & pull..."
  fi

  if [ -d "${PROJECT_GIT_FOLDER_PATH}" ]
  then
    echo
    if [ "${USER}" == "ilg" ]
    then
      echo "If asked, enter ${USER} GitHub password for git pull"
    fi

    cd "${PROJECT_GIT_FOLDER_PATH}"

    if [ "${ACTION}" == "checkout-dev" ]
    then
      git checkout gnu-mcu-eclipse-dev
    elif [ "${ACTION}" == "checkout-stable" ]
    then
      git checkout gnu-mcu-eclipse
    fi

    if false
    then

      git pull --recurse-submodules
      git submodule update --init --recursive --remote

      git branch

      do_host_bootstrap

      rm -rf "${BUILD_FOLDER_PATH}/${APP_LC_NAME}"

      echo
      if [ "${ACTION}" == "pull" ]
      then
        echo "Pull completed. Proceed with a regular build."
      else
        echo "Checkout completed. Proceed with a regular build."
      fi

    else

      echo "Not implemented."
      exit 1

    fi

    exit 0
  else
    echo "No git folder."
    exit 1
  fi

}

FWIW I tried build.sh checkout-dev and got this:

user@ubuntu-16:~$ bash ~/Downloads/openocd-build.git/scripts/build.sh checkout-dev

Work folder: "/home/user/Work/openocd".
Helper script: "/home/user/Downloads/openocd-build.git/scripts/helper/build-helper.sh".
Script "/home/user/Downloads/openocd-build.git/scripts/build.sh" started at Tue Sep 26 11:31:21 IST 2017.

Running on Ubuntu 64-bits.

Checking host curl...
curl 7.47.0 (x86_64-pc-linux-gnu) libcurl/7.47.0 GnuTLS/3.4.10 zlib/1.2.8 libidn/1.32 librtmp/2.3

Checking host git...
git version 2.7.4

Checking host automake...
automake (GNU automake) 1.15

Checking host patch...
GNU patch 2.7.5

Checking host tar...
tar (GNU tar) 1.28
Copyright (C) 2014 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by John Gilmore and Jay Fenlason.

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
Copying openocd-build.git from Downloads...

Running git checkout gnu-mcu-eclipse-dev & pull...

fatal: Not a git repository (or any of the parent directories): .git

I don't know if this is a different problem or related to the original build problem(s) above.

ilg-ul commented 6 years ago

fatal: Not a git repository (or any of the parent directories): .git

this is a tricky issue that I'm not very happy with. the problem is that for convenience the initial *-build.git is cloned in Downloads, but later on its content is needed in the Work folder, which might be on a different file system; so I copy it, but doing so is not that easy, since it includes hidden folders. in the gcc build script I fixed this, we'll do the same here.

ilg-ul commented 6 years ago

the 'Not a git repository (or any of the parent directories): .git' problem was solved:

https://github.com/gnu-mcu-eclipse/openocd-build/commit/8b8e0c1085aff1a7df75e8a003e2a60279e47834

ilg-ul commented 6 years ago

bootstrap

the bootstrap script is always executed right after cloning the openocd git, so it should not be required to do it manually. but if bootstrap fails, you need to run it manually before restarting the script. or you can remove everything and start from scratch.

the bootstrap script requires automake, and this is the reason automake is checked to be present on the host. the Prerequisites page also asks for automake.

I suggest you retry with a clean Ubuntu, install the packages listed in the Prerequisites page, and run the script.

if the script still fails, there must be something else missing, and we need to identify the packages, and add them to the Prerequisites page.

ilg-ul commented 6 years ago

I managed to install a new Ubuntu 16 VM and you were right, libtool was also required; I updated the Prerequisites page.

in my environment the build script was successful for all 4 platforms.

TommyMurphyTM1234 commented 6 years ago

Thanks Liviu. But I started with a clean Ubuntu 16.04 64 bit VM, did all the prerequisite stuff and then ran

bash ~/Downloads/openocd-build.git/scripts/build.sh --win32

but it failed with this:

Checking host unzip...
UnZip 6.00 of 20 April 2009, by Debian. Original by Info-ZIP.
Copying openocd-build.git from Downloads...
Cloning into '/home/user/Work/openocd/openocd-build.git'...
done.
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now

Just in case I did

rm -rf ~/Work

and tried again but got the same error.

ilg-ul commented 6 years ago

this looks like a damaged file. define DEBUG=-x before starting the script and you should be able to see all script lines.

TommyMurphyTM1234 commented 6 years ago

Yes - the contents of the libusb-1.0.20.tar.bz2 file are HTML including

We're sorry -- the Sourceforge site is currently in Disaster Recovery mode, and currently requires
the use of javascript to function.  Please check back later.

so it's a problem with SourceForge. I'll try again later or tomorrow morning.

ilg-ul commented 6 years ago

hmmm... SourceForge is less and less reliable :-(

probably I should re-post those files on GitHub and update the script to get them from there.

TommyMurphyTM1234 commented 6 years ago

SourceForge is less and less reliable

Is this a common occurrence? I can download the zips using a browser (presumably because it supports JavaScript as the message above says is required) but getting them with curl or wget results in a HTML file with that error message.

TommyMurphyTM1234 commented 6 years ago

I can download the zips using a browser (

Oh now I can't and I get this in a browser:

The sourceforge.net website is temporarily in static offline mode. Only a very limited set of project pages are available until the main website returns to service.

I'll leave it until the morning to try again.

Thanks a lot for the work on this - much appreciated.

ilg-ul commented 6 years ago

I already published the binaries on GitHub:

https://github.com/gnu-mcu-eclipse/files/tree/master/libs

TommyMurphyTM1234 commented 6 years ago

Thanks - I tried to hack the script to take them from there but I can't seem to figure out the magic URL that will download the raw file and I just keep getting HTML e.g.:

Downloading "libusb-1.0.20.tar.bz2"...
+ cd /home/user/Work/openocd/download
+ curl -L https://github.com/gnu-mcu-eclipse/files/blob/master/libs/libusb-1.0.20.tar.bz2 --output libusb-1.0.20.tar.bz2
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 32729    0 32729    0     0  32565      0 --:--:--  0:00:01 --:--:-- 32598
+ '[' '!' -d /home/user/Work/openocd/libusb-1.0.20 ']'
+ cd /home/user/Work/openocd
+ tar -xjvf /home/user/Work/openocd/download/libusb-1.0.20.tar.bz2
bzip2: (stdin) is not a bzip2 file.
tar: Child returned status 2
tar: Error is not recoverable: exiting now
user@ubuntu-16:~$ head -20 libusb-1.0.20.tar.bz2 

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="utf-8">
  <link rel="dns-prefetch" href="https://assets-cdn.github.com">
  <link rel="dns-prefetch" href="https://avatars0.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars1.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars2.githubusercontent.com">
  <link rel="dns-prefetch" href="https://avatars3.githubusercontent.com">
  <link rel="dns-prefetch" href="https://github-cloud.s3.amazonaws.com">
  <link rel="dns-prefetch" href="https://user-images.githubusercontent.com/">
ilg-ul commented 6 years ago

I already updated the script to use the 'magic' urls:

https://github.com/gnu-mcu-eclipse/openocd-build/commit/5a4b4e919e35e8cbe06c41fe6abb72cc65e5116f

TommyMurphyTM1234 commented 6 years ago

Thanks a lot. Works great now - win32 build worked fine for me and deb32 churning away now. Thanks! 👍

Edit: deb32 build completed successfully too. Feel free to close the issue if you see fit to do so. Thanks again.

ilg-ul commented 6 years ago

you're welcome!

I'll probably update the other build script to use GitHub archives too.