WIP: add script to build release tarball

[adammoody]

This adds a script to create a release tarball. The tarball packages up mpiFileUtils along with the source for DTCMP, LWGRP, and libcircle all at appropriate release versions. It compiles all source files into libmfu. This reduces the number of dependencies that a user has to install, it speeds up the build, and avoids errors due to version mismatches.

To build a release tarball:

cd dist
./builddist v0.11

Then to build:

  tar -zxf mpifileutils-v0.11.tgz
  pushd mpifileutils-v0.11
    mkdir build
    pushd build
      make clean
      cmake \
        -DCMAKE_INSTALL_PREFIX=../install \
        -DCMAKE_BUILD_TYPE=Debug \
        -DCMAKE_VERBOSE_MAKEFILE=true \
        -DENABLE_LUSTRE=ON \
        -DENABLE_EXPERIMENTAL=ON \
        -DWITH_LibArchive_PREFIX=/path/to/libarchive \
        ..

      make -j install

In theory, libarchive could also be included in a similar way, but the build system for that package is more complicated.

This tarball would be attached to the release page for each release. One can also create a tarball of the latest branch for each component, which is handy when trying to provide an end user with a pre-release.

[adammoody]

@dsikich , when you get a chance, would you mind giving this a try with DAOS? If you checkout this branch, follow the steps above to build a tarball. Then you should be able to build without having to mess with LWGRP, DTCMP, and libcircle. It should still have libarchive and DAOS as external dependencies.

I think you should still be able to disable libarchive, but I haven't tried that yet. I also haven't done much testing so far... just checking that it builds and installs.

[dsikich]

@dsikich , when you get a chance, would you mind giving this a try with DAOS? If you checkout this branch, follow the steps above to build a tarball. Then you should be able to build without having to mess with LWGRP, DTCMP, and libcircle. It should still have libarchive and DAOS as external dependencies.

I think you should still be able to disable libarchive, but I haven't tried that yet. I also haven't done much testing so far... just checking that it builds and installs.

@adammoody so following these instructions I am getting an error:

CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find DTCMP (missing: DTCMP_LIBRARIES DTCMP_INCLUDE_DIRS)
Call Stack (most recent call first):
  /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindDTCMP.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:78 (FIND_PACKAGE)

I am running the cmake command from within the build directory. I also created an install directory one level above the build directory or inside of /home/dsikich/daos-mpifileutils/mpifileutils/dist/mpifileutils-v0.11 which I didn't see a command to create in the instructions but it is being used in the example above.

[adammoody]

Thanks @dsikich . Hmm, it shouldn't be trying to run the FindDTCMP script. I wonder if it's picking up the CMakeLists.txt from mpiFileUtils rather than the higher-level one. Can you verify if you're running this CMakeLists.txt:

mpifileutils-v0.11/CMakeLists.txt

vs this one

mpifileutils-v0.11/mpifileutils/CMakeLists.txt

If it's the second, can you try again using the first?

If that's what's going on, perhaps I should add a command to delete the second one when creating the tarball.

[dsikich]

@adammoody Not sure what you mean because I am just following the instructions you posted here.

cd dist
./builddist v0.11
tar -zxf mpifileutils-v0.11.tgz
cd mpifileutils-v0.11
mkdir build
cd build
[dsikich@boro-54 build]$ make clean
make: *** No rule to make target `clean'.  Stop.
[dsikich@boro-54 build]$ cmake3 ../mpifileutils -DWITH_DAOS_PREFIX=/home/dsikich/daos-clean/daos/install -DCMAKE_INSTALL_PREFIX=../install -DENABLE_LIBARCHIVE=OFF -DENABLE_HDF5=ON -DENABLE_DAOS=ON; make install;
-- The C compiler identification is GNU 4.8.5
-- The CXX compiler identification is GNU 4.8.5
-- Check for working C compiler: /usr/bin/cc
-- Check for working C compiler: /usr/bin/cc - works
-- Detecting C compiler ABI info
-- Detecting C compiler ABI info - done
-- Detecting C compile features
-- Detecting C compile features - done
-- Check for working CXX compiler: /usr/bin/c++
-- Check for working CXX compiler: /usr/bin/c++ - works
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Looking for byteswap.h
-- Looking for byteswap.h - found
-- Found MPI_C: /usr/lib64/mpich/lib/libmpi.so (found version "3.1") 
-- Found MPI_CXX: /usr/lib64/mpich/lib/libmpicxx.so (found version "3.1") 
-- Found MPI: TRUE (found version "3.1")  
-- MPI C Compile Flags:  
-- MPI C Include Path:   /usr/include/mpich-x86_64
-- MPI C Link Flags:     -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags
-- MPI C Libraries:      /usr/lib64/mpich/lib/libmpi.so
-- MPI CXX Compile Flags: 
-- MPI CXX Include Path:  /usr/include/mpich-x86_64
-- MPI CXX Link Flags:    -Wl,-rpath -Wl,/usr/lib64/mpich/lib -Wl,--enable-new-dtags
-- MPI CXX Libraries:     /usr/lib64/mpich/lib/libmpicxx.so;/usr/lib64/mpich/lib/libmpi.so
-- MPI Executable:       /usr/lib64/mpich/bin/mpiexec
-- MPI Num Proc Flag:    -n
CMake Error at /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:164 (message):
  Could NOT find DTCMP (missing: DTCMP_LIBRARIES DTCMP_INCLUDE_DIRS)
Call Stack (most recent call first):
  /usr/share/cmake3/Modules/FindPackageHandleStandardArgs.cmake:445 (_FPHSA_FAILURE_MESSAGE)
  cmake/FindDTCMP.cmake:22 (FIND_PACKAGE_HANDLE_STANDARD_ARGS)
  CMakeLists.txt:78 (FIND_PACKAGE)

-- Configuring incomplete, errors occurred!
See also "/home/dsikich/daos-mpifileutils/mpifileutils/dist/mpifileutils-v0.11/build/CMakeFiles/CMakeOutput.log".
make: *** No rule to make target `install'.  Stop.

Shouldn't I be able to run the cmake command from the build directory?

[adammoody]

Oh, I see. Yes, I think that's using the CMakeLists.txt from the mpifileutils subdirectory rather than the CMakeLists.txt in the directory above. Try changing this bit from:

cmake3 ../mpifileutils

to:

cmake3 ..

[dsikich]

Oh, I see. Yes, I think that's using the CMakeLists.txt from the mpifileutils subdirectory rather than the CMakeLists.txt in the directory above. Try changing this bit from:

cmake3 ../mpifileutils

to:

cmake3 ..

@adammoody

[dsikich@boro-54 build]$ cmake3 .. -DWITH_DAOS_PREFIX=/home/dsikich/daos-clean/daos/install -DCMAKE_INSTALL_PREFIX=../install -DENABLE_LIBARCHIVE=OFF -DENABLE_HDF5=ON -DENABLE_DAOS=ON; make install;
CMake Error: The source directory "/home/dsikich/daos-mpifileutils/mpifileutils/dist/mpifileutils-v0.11" does not appear to contain CMakeLists.txt.
Specify --help for usage, or press the help button on the CMake GUI.
make: *** No rule to make target `install'.  Stop.

I do not see a CMakeLists.txt in the directory above?

[dsikich@boro-54 build]$ ls ..
build  cmake  dtcmp  install  libcircle  lwgrp  mpifileutils

[adammoody]

@daltonbohning , I was working with @dsikich to debug a problem she was seeing in this release tarball. This is meant to simplify the build by packaging a bunch of source into one tarball. I've gone ahead and merged this PR, but I think the problem is still out there. If you have a few minutes, could you give this a shot?

You can pull from master to get the code at this point, since it's merged. You'll see a new "dist" subdirectory. You should then be able to create a "release" tarball with:

cd dist
./builddist main

That should produce a tarball named mpifileutils-master.tgz. This includes mpiFileUtils, lwgrp, dtcmp, and libcircle in one package. You'll still need libarchive installed separately, or disable it during the build.

Then you should be able to build with something like:

  tar -zxf mpifileutils-master.tgz
  cd mpifileutils-master
  mkdir build
  cd build
  cmake \
        -DCMAKE_INSTALL_PREFIX=../install \
        -DCMAKE_BUILD_TYPE=Debug \
        -DCMAKE_VERBOSE_MAKEFILE=true \
        -DENABLE_LUSTRE=ON \
        -DENABLE_EXPERIMENTAL=ON \
        -DWITH_LibArchive_PREFIX=/path/to/libarchive \
        ..
make -j install

Would you please let me know if that process works for you?

Last we tried, @dsikich was missing the top-level CMakeLists.txt file.

[daltonbohning]

@adammoody I started with a fresh clone of mpifileutils and followed your instructions, with two different cmake configs. (I disabled libarchive and lustre, as they are not currently installed on my development system) Both seemed to work correctly. mpifileutils/dist/mpifileutils-master/install/bin contains the executables, and they appear functional.

cmake \
    -DCMAKE_INSTALL_PREFIX=../install \
    -DCMAKE_BUILD_TYPE=Debug \
    -DCMAKE_VERBOSE_MAKEFILE=true \
    -DENABLE_LIBARCHIVE=off \
    -DENABLE_EXPERIMENTAL=ON \
     ..

cmake \
    -DCMAKE_INSTALL_PREFIX=../install \
    -DCMAKE_BUILD_TYPE=Debug \
    -DCMAKE_VERBOSE_MAKEFILE=true \
    -DENABLE_LIBARCHIVE=off \
    -DENABLE_EXPERIMENTAL=ON \
    -DWITH_DAOS_PREFIX=~/daos/install \
    -DENABLE_DAOS=on \
     ..

[adammoody]

Thanks for trying this, @daltonbohning . That makes me feel better about it. Maybe there is something different in @dsikich 's environment or I had a problem in an older version. We can try again later to run it down.

In the meantime, I've attached one of these release tarballs for v0.11. We can create these tarballs for each new release going forward. I've also pushed some commits to the documentation to direct users to build with this tarball as the preferred way. That should be easier, faster, and less error prone for most people.

Since we're a bit slow to stamp releases, DAOS users might still want to build using the longer method by cloning the repo. I moved those instructions to a "development build" section. You should also be able to zip up a snapshot of the latest development branch to pass to people with this script, e.g., builddist main.

The downside is that we now have two CMakeLists.txt files to keep in mind when adding/removing source files.

[daltonbohning]

@adammoody Sounds good, no problem! Until the DAOS changes become more stable / less frequent, I think we'll probably continue recommending users build from the latest source. At least until our internal build+test process uses these simplified methods.