search

vmware-tanzu / oss-httpd-build

This project is a schema to build Apache HTTP Server (httpd), along with a number of frequently updated library components (dependencies), on Linux or Windows. The results of this build are also distributed periodically to the general public from the https://network.tanzu.vmware.com/products/p-apache-http-server (login required)
Apache License 2.0
3 stars 6 forks source link

readme

= VMware OSS Build and Instance Management Schema for Apache HTTP Server

Copyright (C) 2017-2022 VMware, Inc.

This program and the accompanying materials are made available under the terms of the under the Apache License, Version 2.0 (the "License”); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.

== Overview

This project provides a complete illustration of building the httpd project and its many component dependencies, as a monolithic package. These may rely on many established components updated in modern linux distributions, which are reasonably stable. On Windows, every dependency must be built from source.

The build target is /opt/apache/webserver/httpd-2.4/ on Linux and Windows, this is simply a symlink. Each has a longer pathname, e.g. httpd-2.4.53{-bldno}/ which is symlinked from this httpd-2.4 path. Non-release builds have additional pathname components such as -snapshot, -bleed, etc, and may have a TAG to distinguish two otherwise apparently identical builds. By associating the httpd-2.4 symlink to an updated build, all of the associated server instances are redirected to the new release or test runtime at once.

Build \{pkg} reflects the particular component, either the offical designation (httpd-2.4.25/), an indication of interim packages (httpd-2.4.26-rc1/), or the exact provinance of the file (httpd-2.4.26-dev-r1872199/).

Several \{bld} schemas are expected; 'release' for the official packages, 'candidate' for packages anticipated for release, 'snapshot' for the working trees of the component's current maintenance trees, and 'bleed' for next-major or minor development trees.

mak/ Build script files

src/ Complete build sources {pkg}/ Component source (for VPATH build) pkgs/ Source component package downloads {bld}-manifest-vars Description of {pkg} names and versions for {bld}

bld/ Build trees of components by build type {pkg}/ Component build tree (vpath or copied from src/)

dst/ Resulting intermediate binary packages {httpd-pkg}{tag}/ Temporary installation for testing and relocation

pkg/ Final binary package {httpd-pkg}{tag}/ Relocateable binaries for distribution

The structure above can be problematic for running many different builds in parallel. Specifically, the configured bld/{component} can only be --prefix directed at one dst/ target path. If building multiple flavors, use parallel build trees, naming them as desired (e.g. bld-snapshot/, bld-bleed/.)

== Platform Requirements

The development toolchain package group from the respective OS distribution is required to build this package. In addition, a number of stable, well-maintained components are leveraged from the OS distribution, rather than building and installing potentially conflicting versions.

On all platforms, pre-install the Text::Template package from CPAN to assure you are building OpenSSL with the latest tools. In order to run Makefile.test there is a list of CPAN dependencies required to invoke the perl-based Apache::Test framework. These can be found under src/httpdtest-trunk/README after first running the Makefile.gather and Makefile.download phases. (An immediate goal of this toolchain is to automate the local provisioning of those dependencies if not already installed.)

=== RedHat/CentOS/Oracle/Fedora Linux development package dependencies:

Tested With RHEL and CentOS version 7.0 and Fedora 29.

For all steps the complete group of development tools is recommended, e.g.;

$ yum groupinstall 'Development Tools'

Additional required tools which may not be included by default;

$ yum install cmake imake

For the actual Makefile.build phase, a number of system component -devel packages are needed.

$ yum install libuuid-devel expat-devel jansson-devel libxml2-devel \ lua-devel pcre2-devel zlib-devel docbook2X

Perl is used by the openssl documentation schema (Text::Template) and many Perl packages are used by the httpdtest perl framework. To deploy many of these using RHEL 7 packages, use

$ yum install perl-core perl-Module-Load-Conditional \ perl-HTTP-DAV perl-LWP-Protocol-https perl-AnyEvent-HTTP \ perl-Crypt-SSLeay perl-Net-SSLGlue perl-DateTime \ perl-Module-Build-Tiny perl-Test-LeakTrace perl-Test-TCP

And use CPAN for the packages not distributed with the OS as listed above (note the RHEL 7 package perl-Text-Template is too old for use by OpenSSL);

$ cpan install Apache::Test5005compat Text::Template Protocol::HTTP2::Client

In RHEL 8 and Fedora 30 and later, the libxcrypt package is required (although it may often be installed by default), while the Text::Template and Protocol::HTTP2 perl packages become available;

$ dnf groupinstall 'Development Tools'

$ dnf install cmake imake

$ dnf install libuuid-devel expat-devel jansson-devel libxml2-devel \ libxcrypt-devel lua-devel pcre2-devel zlib-devel docbook2X

$ dnf install perl-core perl-Module-Load-Conditional \ perl-HTTP-DAV perl-LWP-Protocol-https perl-AnyEvent-HTTP \ perl-Crypt-SSLeay perl-Net-SSLGlue perl-DateTime \ perl-Module-Build-Tiny perl-Test-LeakTrace perl-Test-TCP \ perl-Text-Template Protocol-HTTP2

$ cpan install Apache::Test5005compat

=== RedHat or Fedora Linux deployment package dependencies:

Tested With RHEL and CentOS version 7.0 and Fedora 29.

These base packages are required for deployment (and are incidently installed with the -devel packages above; most are already present by default);

$ yum install libuuid expat jansson libxml2 lua pcre zlib

Note that following RHEL 7 / Fedora 21, yum becomes the dnf utility.

In RHEL 8 and Fedora 30 and later, the libxcrypt package is required (although it may often be installed by default);

$ dnf install libuuid expat jansson libxcrypt libxml2 lua pcre zlib

=== Ubuntu or Debian development package dependencies:

Tested with Ubuntu 18.04 and 20.04, install the following required tools which may not be included by default;

$ apt-get install build-essential autoconf cmake docbook2x git \ libtool libtool-bin subversion wget xutils-dev

For the actual Makefile.build phase, a number of system component -devel packages are needed.

$ apt-get install libexpat1-dev libjansson-dev libpcre2-dev \ uuid-dev libxml2-dev liblua5.3-dev zlib1g-dev

Perl is used by the openssl documentation schema (Text::Template) and many Perl packages are used by the httpdtest perl framework. To deploy these using Ubuntu 16.04 packages, use;

$ apt install perl-modules libtext-template-perl libcrypt-ssleay-perl \ libnet-sslglue-perl libhttp-dav-perl libanyevent-http-perl \ libdatetime-perl libmodule-build-perl libmodule-build-tiny-perl \ libtest-leaktrace-perl libtest-tcp-perl

And use CPAN for the packages not distributed with the OS as listed above;

$ cpan install Protocol::HTTP2::Client

=== Ubuntu or Debian deployment package dependencies:

These base packages are required for deployment (and are incidently installed with the -devel packages above; most are already present by default);

$ apt-get install libexpat1 libjansson4 libpcre3 libxml2 \ liblua5.3-0 libuuid1 zlib1g

=== Microsoft Windows dependencies

. Microsoft Visual Studio 2017 or 2022 . NASM Assembler . ActiveState or Strawberry Perl . unxutils or gnuwin32 Windows-native unix command line tools (Note mingw and cygwin are not supported) . Info-zip command line zip . curl and awk (or name gawk from unxutils as awk) . Subversion and GIT command line tools

== Phase 1: Gather and Download Sources

$ cd src/ $ make -f ../mak/Makefile.gather [BLD={type}] [GRP=complete] [targets] $ make -f ../mak/Makefile.download [BLD={type}]

BLD defines the build type, one of : release - candidate - snapshot - bleed (case sensitive) where release is the default.

Gathers the manifest of source code packages or source checkouts for all packages into a {type}-manifest-vars file into the source tree, providing the package origins, version identifiers and directory names. Then download the list provided by that manifest file. If that manifest file from the Makefile.gather step has not changed, there would be no need to repeat the remaining steps in this process.

This will gather all components if GRP=complete is specified, otherwise the linux system package sources of expat, lua, pcre, jansson, libxml2 and zlib will not be gathered, downloaded or compiled. Two packages not included in the GRP=complete all target are the "openldap" library for the httpd ldap modules and the Tomcat "tcnative" connector. Add these explicitly to the targets list followed by the explicit "all" target, as desired.

These makefiles are run from the source directory root (e.g. src/), and must be performed as updates to the source packages are released or committed. The resulting manifest from Makefile.gather can be compared to the previously created manifest to determine whether any sources have been updated.

Makefile.preconfig must immediately follow when the manifest has changeed, owing to newly downloaded directories to be preconfigured.

== Phase 2: Preconfigure Sources

$ cd src/ $ make -f ../mak/Makefile.preconfig [BLD={type}]

Prepare configuration scripts of packages, particularly from source control where autoconf etc have not been invoked yet, or where release and candidate source packages are not distributed with an autotools step completed.

This makefile is run after Makefile.gather+Makefile.download from the source directory root, and must be performed following updates to the source packages as indicated by manifest changes. Only source code packages corresponding to the specific BLD target are updated.

The result of this step is suitable for archive, or escrow and distribution to multiple build systems, resuming from the following Makefile.build step.

== Phase 3: Build Sources

$ cd bld/ $ make -f ../mak/Makefile.build [BLD={type}] [TAG={-suffix}]

Build all components described by the manifest into the intermediate/ temporary installation tree, using that intermediate tree as the component reference for later components.

TAG defines the target directory and package name suffix such as a datestamp, checkout revision, or continuous build revision number. By default there is no suffix tag.

This makefile is run after Makefile.gather and Makefile.preconfig and may be based on a snapshot of the build tree from those two previous steps from another continuous build job.

This makefile must be run from the build (not source) subdirectory, such as bld/. The build tree uses the same component directory names as the source tree. The components are initially installed into the DESTDIR which is the ../dst/httpd component directory name with the TAG variable suffixed. SRCDIR references the source tree (typically ../src) and would typically not need to be overridden.

The TARGET directory, /opt/apache/webserver/$(httpd_srcdir)$(TAG) would typically not be overridden, and refers to the anticipated installation path of the resulting package. Use this to ensure the generated suexec binaries are recognized as valid.

== Phase 4: Test Source and Intermediate Installation

$ cd bld/ $ make -f ../mak/Makefile.test [BLD={type}] [TAG={-suffix}]

Test all components described by the manifest and the intermediate/ temporary installation httpd server.

This makefile must be run from the build (not source) subdirectory. Where a component has an integrated test target these are processed within the build tree. The Apache httpd perl test framework is invoked against the intermediate installation in the $DESTDIR path.

== Phase 5: Package Installation Binaries

$ cd pkg/ $ make -f ../mak/Makefile.package [BLD={type}] [TAG={-suffix}]

Copy the intermediate/temporary installation httpd server and dependent binaries into the dst/webserver/ tree to rewrite configurations files and scripts with as relocatable paths, add the instance management scripts, split the debugging symbols from the binaries, and tar up the package.

This makefile is run from the pkg (not src, bld or dst) subdirectory. WARNING; running this in the dst subdirectory will wipe out the last build target directory; please use caution.

Distribute the resulting .tar.bz2 files as desired.

== Installation Phase ==

Installing these binaries to a target machine consists of untarring the package, relocating references to the desired installation path and creating a symlink to use as the 'generic' reference to the now-current httpd.

$ mkdir -p /opt/apache/webserver $ cd /opt/apache/webserver $ tar -xjvf {pkgname} $ ./httpd-2.4.29{tag}/bin/fixrootpath.pl $ ln -sf httpd-2.4.29{tag} httpd-2.4

Packages may be installed in parallel; in order to switch the running httpd version, simply reassign the symlink to the desired version and restart the server instances.

== Instance Creation ==

To create an instance /opt/apache/webserver/{hostname}, use the following commands;

$ cd /opt/apache/webserver $ ./httpd-2.4/bin/newserver.pl --server {hostname}

The resulting directory includes bin, conf, htdocs, cgi-bin, ssl and logs subdirectories. The bin directory includes an environment script for consuming the instance's and then binaries distributed in httpd-2.4/bin, as well as an httpd control script httpdctl.

See the README.md file for more specific details about package installation and instance management.