SueChaplain commented 6 years ago

Covering:

If users need more pointers for using Docker (E.g. how to map host directories, Issue #13 )
If users don't want to use Docker and want to build on their own systems (other linux distros, other platforms)

Location of build instructions:

Does it make sense to put these instructions into the openj9/buildenv .... or to keep them in a markdown file on the website repo?

SueChaplain commented 6 years ago

I've created a more detailed set of build instructions for OpenJDK9 with OpenJ9. There are placeholders for AIX and Windows there, when we are ready. This all needs checking and verifying, and there may be more instructions to interleave. @keithc-ca @davew16 @mgaudet @mstoodle ... anyone else welcome to chip in.

Where should these go? In this repo, linked to from the website or in the openj9 repo, alongside the Dockerfiles?

Pasting the content of my markdown file here:

Building OpenJDK 9 with OpenJ9

Our website describes a simple build process that uses Docker and Dockerfiles to create a build environment that contains everything you need to easily build OpenJDK 9 with the Eclipse OpenJ9 virtual machine. If you are new to Docker and you need a few more pointers to help you get going, read the expanded build process here:

Building with Docker :whale:

However, if you want to build OpenJDK 9 with OpenJ9 directly on your system or in a virtual machine without Docker, follow these instructions for your platform:

Linux :penguin:
AIX :blue_book:
Windows :ledger:

Building with Docker

:whale: Docker technology makes it very easy to create a precise build environment in which you can build OpenJDK with OpenJ9 with minimal impact to your local system.

1. Prepare your system

:whale: The first thing you need to do is install Docker. You can download the free Community edition from here, which also contains instructions for installing Docker on your system. You should also read the Getting started guide to familiarise yourself with the basic Docker concepts and terminology.

We've created Dockerfiles that you can use to build and run containers that have all the correct pre-requisites. Dockerfiles are available for the following platforms:

Either download one of these Dockerfiles to your local system or copy and paste one of the following commands:

For Linux on 64-bit x86 systems, run:

wget https://raw.githubusercontent.com/eclipse/openj9/master/buildenv/docker/jdk9/x86_64/ubuntu16/Dockerfile

For Linux on 64-bit Power systems, run:

wget https://raw.githubusercontent.com/eclipse/openj9/master/buildenv/docker/jdk9/ppc64le/ubuntu16/Dockerfile

For Linux on 64-bit z Systems, run:

wget https://raw.githubusercontent.com/eclipse/openj9/master/buildenv/docker/jdk9/s390x/ubuntu16/Dockerfile

Next, run the following command to build a Docker image, called openj9:

docker build -t openj9 Dockerfile .

Start a Docker container from the openj9 image with the following command:

docker run -v <host_directory>:/root/hostdir -it openj9

where -v maps any directory, <host_directory>, on your local system to the containers /root/hostdir directory so that you can store the binaries, once they are built.

:pencil: Depending on your Docker system configuration, you might need to prefix these commands with sudo. Now that you have the Docker image running, you are ready to move to the next step.

2. Get the source

:whale: First you need to clone the Extensions for OpenJDK for OpenJ9 project. This repository is a git mirror of OpenJDK without the HotSpot JVM, but with an openj9 branch that contains a few necessary patches. Run the following command:

git clone https://github.com/ibmruntimes/openj9-openjdk-jdk9

Cloning this repository can take a while because OpenJDK is a large project! When the process is complete, change directory into the cloned repository:

cd openj9-openjdk-jdk9

Now fetch additional sources from the Eclipse OpenJ9 project and its clone of Eclipse OMR:

bash ./get_source.sh

3. Configure

:whale: When you have all the source files that you need, run the configure script, which detects how to build in the current build environment. You must specify one extra jar file, which has already been installed into the Docker container.

bash ./configure --with-freemarker-jar=/root/freemarker.jar

:pencil: You must give an absolute path to freemarker.jar

4. Build

:whale: Now you're ready to build OpenJDK with OpenJ9:

make all

:warning: If you just type make, rather than make all your build will fail, because the default make target is exploded-image. If you want to specify make instead of make all, you must add --default-make-target=images when you run the configure script.

Two Java builds are produced: a full developer kit (jdk) and a runtime environment (jre):

build/<platform>/images/jdk
build/<platform>/images/jre

where <platform> reflects the Dockerfile you chose. For example, the directory for Linux 64-bit on x86 systems is: linux-x86_64-normal-server-release

Copy the binaries to the containers /root/hostdir directory so that you can access them on your local system. You'll find them in the directory you set for <host_directory> when you started your Docker container (See 1. Prepare your system.).

5. Test

:whale: For a simple test, try running the java -version command. Change to <host_directory> on your local system and run:

./bin/java -version

Here is some sample output:

openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-adhoc..openj9-openjdk-jdk9)
Eclipse OpenJ9 VM (build 2.9, JRE 9 Linux amd64-64 Compressed References 20170912_000000 (JIT enabled, AOT enabled)
J9VM - e6ca543
JIT  - e6ca543
OMR  - a0813c2
OpenJDK  - 8593b2f based on )

:whale: Congratulations! :tada:

Linux

:penguin: This build process provides detailed instructions for building a Linux x86-64 OpenJDK 9 with OpenJ9 binary on a Ubuntu 16.04 system.

If you are using a different Linux distribution, you might have to review the list of libraries that are bundled with your distribution and/or modify the instructions to use equivalent commands to the Advanced Packaging Tool (APT). For example, for Centos, substitute the apt-get command with yum.

If you want to build a binary for Linux on a different architecture, such as Power Systems™ or z Systems™, the process is very similar and any additional information for those architectures are included as Notes :pencil: as we go along.

1. Prepare your system

:penguin: All the pre-requisites that you need on your system are described in this Dockerfile, which simply lists the dependencies that must be installed on the system, plus a few configuration steps.

:pencil: For Linux on other architectures, refer to the appropriate DockerFile: Linux on Power systems, Linux on z Systems.

We'll break this down into individual steps.

Install the dependencies that can be obtained with the apt-get command. Copy and paste the list from the Dockerfile to your command prompt. If you already have some of these packages installed, just omit them. It is important to take the list of dependencies from the Dockerfile because this is kept right up to date. The list looks something like this:

apt-get update \
  && apt-get install -qq -y --no-install-recommends \
    autoconf \
    ca-certificates \
    ccache \
    cpio \
    file \
    g++-4.8 \
    gcc-4.8 \
    git \
    git-core \
    libasound2-dev \
    libcups2-dev \
    libelf-dev \
    libfreetype6-dev \
    libnuma-dev \
    libx11-dev \
    libxext-dev \
    libxrender-dev \
    libxt-dev \
    libxtst-dev \
    make \
    openjdk-8-jdk \
    pkg-config \
    realpath \
    ssh \
    unzip \
    wget \
    zip \
  && rm -rf /var/lib/apt/lists/*

:pencil: For Linux on other architectures, download the IBM SDK for Java 8 rather than the openjdk-8-jdk package. The IBM versions contain a JIT compiler that will significantly accelerate compile time.

This build uses the same gcc and g++ compiler levels as OpenJDK, which might be backlevel compared with the versions you use on your system. Create links for the compilers using the following command:

ln -s g++ /usr/bin/c++ \
  && ln -s g++-4.8 /usr/bin/g++ \
  && ln -s gcc /usr/bin/cc \
  && ln -s gcc-4.8 /usr/bin/gcc

Download and setup freemarker.jar into the /root directory:

cd /root \
  && wget https://sourceforge.net/projects/freemarker/files/freemarker/2.3.8/freemarker-2.3.8.tar.gz/download -O freemarker.tgz \
  && tar -xzf freemarker.tgz freemarker-2.3.8/lib/freemarker.jar --strip=2 \
  && rm -f freemarker.tgz

2. Get the source

:penguin: First you need to clone the Extensions for OpenJDK for OpenJ9 project. This repository is a git mirror of OpenJDK without the HotSpot JVM, but with an openj9 branch that contains a few necessary patches. Run the following command:

git clone https://github.com/ibmruntimes/openj9-openjdk-jdk9

Cloning this repository can take a while because OpenJDK is a large project! When the process is complete, change directory into the cloned repository:

cd openj9-openjdk-jdk9

Now fetch additional sources from the Eclipse OpenJ9 project and its clone of Eclipse OMR:

bash ./get_source.sh

3. Configure

:penguin: When you have all the source files that you need, run the configure script, which detects how to build in the current build environment.

bash ./configure --with-freemarker-jar=/root/freemarker.jar

4. Build

:penguin: Now you're ready to build OpenJDK with OpenJ9:

make all

:warning: If you just type make, rather than make all your build will fail, because the default make target is exploded-image. If you want to specify make instead of make all, you must add --default-make-target=images when you run the configure script.

Two Java builds are produced: a full developer kit (jdk) and a runtime environment (jre)

build/linux-x86_64-normal-server-release/images/jdk
build/linux-x86_64-normal-server-release/images/jre

:pencil: On other architectures the /jdk and /jre directories are in build/linux-ppc64le-normal-server-release/images (Linux on 64-bit Power systems) and build/linux-s390x-normal-server-release/images (Linux on 64-bit z Systems)

5. Test

:penguin: For a simple test, try running the java -version command. Change to the /jre directory:

cd build/linux-x86_64-normal-server-release/images/jre

Run:

./bin/java -version

Here is some sample output:

openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-adhoc..openj9-openjdk-jdk9)
Eclipse OpenJ9 VM (build 2.9, JRE 9 Linux amd64-64 Compressed References 20170912_000000 (JIT enabled, AOT enabled)
J9VM - e6ca543
JIT  - e6ca543
OMR  - a0813c2
OpenJDK  - 8593b2f based on )

:penguin: Congratulations! :tada:

AIX

:blue_book:

:construction: We haven't created a full build process for AIX yet? Watch this space!

Windows

:ledger:

:construction: We haven't created a full build process for Windows yet? Watch this space!

SueChaplain commented 6 years ago

@mstoodle @pshipton @DanHeidinga Please see the last comment. This is an attempt (work in progress) to produce a more detailed set of build instructions for those who need more help to build with docker or want to build on another platform. This one is for v9, but the idea is to duplicate/amend per version and link to it from the simplest process we have on the website. There are still gaps, and there is still untested content, but I would like to put it somewhere where others can contribute to it as they progress platform builds. Idea was to place it in openj9/buildenv/docs? so that it sits alongside the dockerfiles (and a new aix directory for a yum shell <text_file> command that sets out AIX pre-reqs in <text_file> in a similar fashion to the dockerfiles).

Do you have a different view on the location or shall I go ahead and create a PR?

keithc-ca commented 6 years ago

The apt-get install command is missing (the recently added) libdwarf-dev package.

keithc-ca commented 6 years ago

The sample -version output is old: it still lists a JIT sha and is missing the jdk-181 tag.

keithc-ca commented 6 years ago

I suggest this should go under the existing doc directory, perhaps in a new buildenv folder.

SueChaplain commented 6 years ago

Thanks @keithc-ca

"The apt-get install command is missing (the recently added) libdwarf-dev package."

the process does say "It is important to take the list of dependencies from the Dockerfile because this is kept right up to date. The list looks something like this: ...." because it will be hard to keep track of every and all changes. However, if this process sat alongside the dockerfiles, developers could amend it when they make changes there.

"The sample -version output is old: it still lists a JIT sha and is missing the jdk-181 tag."

if you can pass over a recent java -version output I will update it.

"I suggest this should go under the existing doc directory, perhaps in a new buildenv folder."

What is that doc directory for? Looks a little random to me. Is this doc that is a placeholder and intended to go into the website (when it exists) .. or is this something else? I'd be concerned about putting the build processes in there if it led users to think that this was the place for all docs about OpenJ9. We're still hoping to produce an MkDocs website with a set of contributed user documentation in the near future. That would be in a different repo frm OpenJ9 .... infact maybe it should go there, although I still think there is virtue in having the processes close to the dockerfiles.

mgaudet commented 6 years ago

The docs directory is intended to be documentation about OpenJ9 from a developer perspective. What's the thinking on splitting documentation into yet-another-repo?

DanHeidinga commented 6 years ago

My preference is to put the doc as close to the code as possible so @SueChaplain's suggestion of a new openj9/buildenv/docs directory sounds like the right approach.

I'd even go one step further and do away with the docs directory by adding the content to a openj9/buildenv/Build_Instructions.md or similarly named file.

mstoodle commented 6 years ago

Looks really good, @SueChaplain ! Thanks for putting this together and I agree with Dan's comment about where it should go.

SueChaplain commented 6 years ago

Hi @mgaudet

"The docs directory is intended to be documentation about OpenJ9 from a developer perspective. What's the thinking on splitting documentation into yet-another-repo?"

the website is our shop window: eclipse/openj9-website
the user docs that contain how to configure/tune/troubleshoot/cmd ref (a.k.a what used to be in the old IBM SDK guide) would go into an MkDocs hosted site, probably eclipse/openj9-docs. This is not a new plan, just something that we haven't been able to do yet.
If the developer focused docs more directly support the API doc they might be best retained with the openj9 source. Or they could also become part of the MkDocs hosted site (when we get it up and running)?

mgaudet commented 6 years ago

I'd love to see the OpenJ9 repo itself get processed into an MkDocs site, for the purpose of building out "hacking on OpenJ9" documentation.

keithc-ca commented 6 years ago

Current 'java -version' output:

openjdk version "9-internal"
OpenJDK Runtime Environment (build 9-internal+0-adhoc.keith.openj9-openjdk-jdk9)
Eclipse OpenJ9 VM (build 2.9, JRE 9 Linux amd64-64 Compressed References 20171030_000000 (JIT enabled, AOT enabled)
OpenJ9   - 731f323
OMR      - 7c3d3d7
OpenJDK  - 1983043 based on jdk-9+181)

pshipton commented 6 years ago

Steps 2, 3, 4, 5 should be the same for both docker and non-docker, although I see there is an extra suggestion for docker (Copy the binaries to the containers /root/hostdir directory). Can we have a single copy of these steps, with a docker addendum in the steps that need it? This means only a single version to keep up to date, and the steps won't diverge from each other. I see the docker version of step 3 already has extra information missing from the non-docker version (You must give an absolute path to freemarker.jar).

keithc-ca commented 6 years ago

The danger of relying on people seeing that clause about looking in Dockerfile is that they won't (I didn't). If the command were intentionally incomplete, they won't have a choice. Perhaps the advice ought to be to find the command that begins with the following:

apt-get update \
  && apt-get install -qq -y --no-install-recommends \
    autoconf \
    ca-certificates \
    ...

SueChaplain commented 6 years ago

Build instructions now visible here: https://github.com/eclipse/openj9/blob/master/buildenv/Build_Instructions_V9.md

We still need to link to these from the website. Further platform support will be added as and when available.

SueChaplain commented 6 years ago

Windows added in https://github.com/eclipse/openj9/pull/521

SueChaplain commented 6 years ago

Two sets now in place. New issues will be created to capture work for new platforms.

https://github.com/eclipse/openj9/blob/master/buildenv/Build_Instructions_V9.md and https://github.com/eclipse/openj9/blob/master/buildenv/Build_Instructions_V8.md

eclipse-openj9 / openj9-website

Create / link to more detailed build instructions #25

Building OpenJDK 9 with OpenJ9

Building with Docker

1. Prepare your system

2. Get the source

3. Configure

4. Build

5. Test

Linux

1. Prepare your system

2. Get the source

3. Configure

4. Build

5. Test

AIX

Windows