search

open-mpi / ompi

Open MPI main development repository
https://www.open-mpi.org
Other
2.12k stars 856 forks source link

Missing orte/mca/common/Makefile.am at ./autogen.sh line 94. #1276

Closed jeffhammond closed 8 years ago

jeffhammond commented 8 years ago

summary

I cannot being to even build the v1.10 release branch from source because autogen.sh fails.

git info

Start with current v1.10

egonza2-mobl1:github jrhammon$ git checkout remotes/release/v1.10 -b v1.10
...
egonza2-mobl1:github jrhammon$ git pull
Already up-to-date.
egonza2-mobl1:github jrhammon$ git info

## Remote URLs:

origin  git@github.com:open-mpi/ompi.git (fetch)
origin  git@github.com:open-mpi/ompi.git (push)
release https://github.com/open-mpi/ompi-release.git (fetch)
release https://github.com/open-mpi/ompi-release.git (push)

## Remote Branches:

  origin/HEAD -> origin/master
  origin/master
  release/v0.9
  release/v1.0
  release/v1.1
  release/v1.10
  release/v1.2
  release/v1.2lanl
  release/v1.3
  release/v1.4
  release/v1.5
  release/v1.6
  release/v1.7
  release/v1.8
  release/v1.8.1
  release/v2.x

## Local Branches:

  master
* v1.10

## Most Recent Commit:

commit 0e3b111db98682f6ae1f611b2f90ea88685fc7c8
Merge: d7b5b25 4c81052
Author: rhc54 <rhc@open-mpi.org>

    Merge pull request #876 from ggouaillardet/topic/v1.10/mpi_neighbor_alltoallx_fixes

Type 'git log' for more commits, or 'git show <commit id>' for full commit details.

## Configuration (.git/config):

user.name=Jeff Hammond
user.email=jeff.science@gmail.com
push.default=simple
merge.tool=vimdiff
core.editor=vim
core.excludesfile=/Users/jrhammon/.gitignore_global
http.proxy=...
https.proxy=...
git.proxy=...
url.https://.insteadof=git://
filter.media.clean=git-media-clean %f
filter.media.smudge=git-media-smudge %f
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=git@github.com:open-mpi/ompi.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.master.remote=origin
branch.master.merge=refs/heads/master
remote.release.url=https://github.com/open-mpi/ompi-release.git
remote.release.fetch=+refs/heads/*:refs/remotes/release/*
branch.v1.10.remote=release
branch.v1.10.merge=refs/heads/v1.10

autogen.sh

egonza2-mobl1:github jrhammon$ ./autogen.sh 
Open MPI autogen (buckle up!)

1. Checking tool versions

   Searching for autoconf
     Found autoconf version 2.69; checking version...
       Found version component 2 -- need 2
       Found version component 69 -- need 65
     ==> ACCEPTED
   Searching for libtoolize
  libtoolize not found
   Searching for glibtoolize
     Found glibtoolize version 2.4.6; checking version...
       Found version component 2 -- need 2
       Found version component 4 -- need 2
     ==> ACCEPTED
   Searching for automake
     Found automake version 1.15; checking version...
       Found version component 1 -- need 1
       Found version component 15 -- need 12
     ==> ACCEPTED

2. Running template-generating scripts

=== ompi/include/mpif-values.pl
creating Fortran header files (with common constants)...
./ompi/include/mpif-handles.h unchanged; not written
./ompi/include/mpif-constants.h unchanged; not written
./ompi/include/mpif-io-handles.h unchanged; not written
./ompi/include/mpif-io-constants.h unchanged; not written
./ompi/mpi/fortran/use-mpi-f08/constants.h unchanged; not written

3. Searching for projects, MCA frameworks, and MCA components

*** Found opal project

=== Found opal / backtrace framework
    Found framework configure.m4 file
--- Found opal / backtrace / execinfo component
    Found configure.m4 file
--- Found opal / backtrace / none component
    Found configure.m4 file
--- Found opal / backtrace / printstack component
    Found configure.m4 file

=== Found opal / common framework
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/cuda -> Excluded
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/libfabric -> Excluded
--- Found opal / common / pmi component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/sm -> Excluded
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/ugni -> Excluded
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/verbs -> Excluded
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/common/verbs_usnic -> Excluded

=== Found opal / compress framework
--- Found opal / compress / bzip component
--- Found opal / compress / gzip component

=== Found opal / crs framework
--- Found opal / crs / blcr component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/crs/criu -> Excluded
--- Found opal / crs / dmtcp component
    Found configure.m4 file
--- Found opal / crs / none component
--- Found opal / crs / self component
    Found configure.m4 file

=== Found opal / db framework
--- Found opal / db / hash component
--- Found opal / db / pmi component
    Found configure.m4 file
--- Found opal / db / print component
--- Found opal / db / sqlite component
    Found configure.m4 file

=== Found opal / dl framework
    Found framework configure.m4 file
--- Found opal / dl / dlopen component
    Found configure.m4 file
--- Found opal / dl / libltdl component
    Found configure.m4 file

=== Found opal / event framework
    Found framework configure.m4 file
--- Found opal / event / external component
    Found configure.m4 file
--- Found opal / event / libevent2021 component
    Found configure.m4 file
    Found subdir: libevent (will process later)
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/event/libevent2022 -> Excluded

=== Found opal / hwloc framework
    Found framework configure.m4 file
--- Found opal / hwloc / external component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/hwloc/hwloc1111 -> Excluded
--- Found opal / hwloc / hwloc191 component
    Found configure.m4 file

=== Found opal / if framework
--- Found opal / if / bsdx_ipv4 component
    Found configure.m4 file
--- Found opal / if / bsdx_ipv6 component
    Found configure.m4 file
--- Found opal / if / linux_ipv6 component
    Found configure.m4 file
--- Found opal / if / posix_ipv4 component
    Found configure.m4 file
--- Found opal / if / solaris_ipv6 component
    Found configure.m4 file

=== Found opal / installdirs framework
    Found framework configure.m4 file
--- Found opal / installdirs / config component
    Found configure.m4 file
--- Found opal / installdirs / env component
    Found configure.m4 file

=== Found opal / memchecker framework
    Found framework configure.m4 file
--- Found opal / memchecker / valgrind component
    Found configure.m4 file

=== Found opal / memcpy framework
    Found framework configure.m4 file

=== Found opal / memory framework
    Found framework configure.m4 file
--- Found opal / memory / linux component
    Found configure.m4 file
--- Found opal / memory / malloc_solaris component
    Found configure.m4 file

=== Found opal / pstat framework
    Found framework configure.m4 file
--- Found opal / pstat / linux component
    Found configure.m4 file
--- Found opal / pstat / test component
    Found configure.m4 file

=== Found opal / sec framework
--- Found opal / sec / basic component
--- Found opal / sec / keystone component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/opal/mca/sec/munge -> Excluded

=== Found opal / shmem framework
--- Found opal / shmem / mmap component
    Found configure.m4 file
--- Found opal / shmem / posix component
    Found configure.m4 file
--- Found opal / shmem / sysv component
    Found configure.m4 file

=== Found opal / timer framework
    Found framework configure.m4 file
--- Found opal / timer / aix component
    Found configure.m4 file
--- Found opal / timer / altix component
    Found configure.m4 file
--- Found opal / timer / catamount component
    Found configure.m4 file
--- Found opal / timer / darwin component
    Found configure.m4 file
--- Found opal / timer / linux component
    Found configure.m4 file
--- Found opal / timer / solaris component
    Found configure.m4 file

*** Found orte project

=== Found orte / common framework
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/common/alps -> Excluded

=== Found orte / dfs framework
--- Found orte / dfs / app component
--- Found orte / dfs / orted component
--- Found orte / dfs / test component

=== Found orte / errmgr framework
--- Found orte / errmgr / default_app component
--- Found orte / errmgr / default_hnp component
--- Found orte / errmgr / default_orted component
--- Found orte / errmgr / default_tool component

=== Found orte / ess framework
--- Found orte / ess / alps component
    Found configure.m4 file
--- Found orte / ess / env component
--- Found orte / ess / hnp component
--- Found orte / ess / lsf component
    Found configure.m4 file
--- Found orte / ess / pmi component
    Found configure.m4 file
--- Found orte / ess / singleton component
--- Found orte / ess / slurm component
    Found configure.m4 file
--- Found orte / ess / tm component
    Found configure.m4 file
--- Found orte / ess / tool component

=== Found orte / filem framework
--- Found orte / filem / raw component

=== Found orte / grpcomm framework
--- Found orte / grpcomm / bad component
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/grpcomm/brks -> Excluded
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/grpcomm/direct -> Excluded
--- Found orte / grpcomm / hier component
    => Ignored (found .ompi_ignore file)
--- Found orte / grpcomm / pmi component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/grpcomm/rcd -> Excluded

=== Found orte / iof framework
--- Found orte / iof / hnp component
--- Found orte / iof / mr_hnp component
--- Found orte / iof / mr_orted component
--- Found orte / iof / orted component
--- Found orte / iof / tool component

=== Found orte / odls framework
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/odls/alps -> Excluded
--- Found orte / odls / default component
    Found configure.m4 file

=== Found orte / oob framework
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/oob/alps -> Excluded
--- Found orte / oob / tcp component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/orte/mca/oob/ud -> Excluded

=== Found orte / plm framework
--- Found orte / plm / alps component
    Found configure.m4 file
--- Found orte / plm / isolated component
    Found configure.m4 file
--- Found orte / plm / lsf component
    Found configure.m4 file
--- Found orte / plm / rsh component
    Found configure.m4 file
--- Found orte / plm / slurm component
    Found configure.m4 file
--- Found orte / plm / tm component
    Found configure.m4 file

=== Found orte / ras framework
--- Found orte / ras / alps component
    Found configure.m4 file
--- Found orte / ras / gridengine component
    Found configure.m4 file
--- Found orte / ras / loadleveler component
    Found configure.m4 file
--- Found orte / ras / lsf component
    Found configure.m4 file
--- Found orte / ras / simulator component
--- Found orte / ras / slurm component
    Found configure.m4 file
--- Found orte / ras / tm component
    Found configure.m4 file

=== Found orte / rmaps framework
--- Found orte / rmaps / mindist component
    Found configure.m4 file
--- Found orte / rmaps / ppr component
--- Found orte / rmaps / rank_file component
    Found configure.m4 file
--- Found orte / rmaps / resilient component
--- Found orte / rmaps / round_robin component
--- Found orte / rmaps / seq component
--- Found orte / rmaps / staged component

=== Found orte / rml framework
--- Found orte / rml / ftrm component
    Found configure.m4 file
--- Found orte / rml / oob component

=== Found orte / routed framework
--- Found orte / routed / binomial component
--- Found orte / routed / debruijn component
--- Found orte / routed / direct component
--- Found orte / routed / radix component

=== Found orte / sensor framework
--- Found orte / sensor / file component
    Found configure.m4 file
--- Found orte / sensor / ft_tester component
    Found configure.m4 file
--- Found orte / sensor / heartbeat component
    Found configure.m4 file
--- Found orte / sensor / resusage component
    Found configure.m4 file

=== Found orte / snapc framework
--- Found orte / snapc / full component
    Found configure.m4 file

=== Found orte / sstore framework
--- Found orte / sstore / central component
    Found configure.m4 file
--- Found orte / sstore / stage component
    Found configure.m4 file

=== Found orte / state framework
--- Found orte / state / app component
--- Found orte / state / dvm component
--- Found orte / state / hnp component
--- Found orte / state / novm component
--- Found orte / state / orted component
--- Found orte / state / staged_hnp component
--- Found orte / state / staged_orted component
--- Found orte / state / tool component

*** Found ompi project

=== Found ompi / allocator framework
--- Found ompi / allocator / basic component
--- Found ompi / allocator / bucket component

=== Found ompi / bcol framework
--- Found ompi / bcol / basesmuma component
--- Found ompi / bcol / ptpcoll component

=== Found ompi / bml framework
--- Found ompi / bml / r2 component
    Found configure.m4 file

=== Found ompi / btl framework
--- Found ompi / btl / openib component
    Found configure.m4 file
--- Found ompi / btl / portals4 component
    Found configure.m4 file
--- Found ompi / btl / scif component
    Found configure.m4 file
--- Found ompi / btl / self component
--- Found ompi / btl / sm component
    Found configure.m4 file
--- Found ompi / btl / smcuda component
    Found configure.m4 file
--- Found ompi / btl / tcp component
    Found configure.m4 file
--- Found ompi / btl / ugni component
    Found configure.m4 file
--- Found ompi / btl / usnic component
    Found configure.m4 file
--- Found ompi / btl / vader component
    Found configure.m4 file

=== Found ompi / coll framework
--- Found ompi / coll / basic component
--- Found ompi / coll / cuda component
    Found configure.m4 file
--- Found ompi / coll / fca component
    Found configure.m4 file
--- Found ompi / coll / hcoll component
    Found configure.m4 file
--- Found ompi / coll / hierarch component
    Found configure.m4 file
--- Found ompi / coll / inter component
--- Found ompi / coll / libnbc component
--- Found ompi / coll / ml component
    Found configure.m4 file
--- Found ompi / coll / portals4 component
    Found configure.m4 file
--- Found ompi / coll / self component
--- Found ompi / coll / sm component
--- Found ompi / coll / tuned component

=== Found ompi / common framework
--- Found ompi / common / cuda component
    Found configure.m4 file
--- Found ompi / common / sm component
    Found configure.m4 file
--- Found ompi / common / ugni component
    Found configure.m4 file
--- Found ompi / common / verbs component
    Found configure.m4 file
--- Found ompi / common / verbs_usnic component
    Found configure.m4 file

=== Found ompi / crcp framework
--- Found ompi / crcp / bkmrk component
    Found configure.m4 file

=== Found ompi / dpm framework
--- Found ompi / dpm / orte component
    Found configure.m4 file

=== Found ompi / fbtl framework
    Found framework configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/fbtl/plfs -> Excluded
--- Found ompi / fbtl / posix component
    Found configure.m4 file
--- Found ompi / fbtl / pvfs2 component
    Found configure.m4 file

=== Found ompi / fcoll framework
    Found framework configure.m4 file
--- Found ompi / fcoll / dynamic component
--- Found ompi / fcoll / individual component
--- Found ompi / fcoll / static component
--- Found ompi / fcoll / two_phase component
--- Found ompi / fcoll / ylib component

=== Found ompi / fs framework
    Found framework configure.m4 file
--- Found ompi / fs / lustre component
    => Ignored (found .ompi_ignore file)
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/fs/plfs -> Excluded
--- Found ompi / fs / pvfs2 component
    Found configure.m4 file
--- Found ompi / fs / ufs component

=== Found ompi / io framework
    Found framework configure.m4 file
--- Found ompi / io / ompio component
--- Found ompi / io / romio component
    Found configure.m4 file
    Found subdir: romio (will process later)
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/io/romio314 -> Excluded

=== Found ompi / mpool framework
--- Found ompi / mpool / gpusm component
    Found configure.m4 file
--- Found ompi / mpool / grdma component
--- Found ompi / mpool / rgpusm component
    Found configure.m4 file
--- Found ompi / mpool / sm component
--- Found ompi / mpool / udreg component
    Found configure.m4 file

=== Found ompi / mtl framework
--- Found ompi / mtl / mxm component
    Found configure.m4 file
--- Found ompi / mtl / ofi component
    Found configure.m4 file
--- Found ompi / mtl / portals4 component
    Found configure.m4 file
--- Found ompi / mtl / psm component
    Found configure.m4 file
--- Found ompi / mtl / psm2 component
    Found configure.m4 file

=== Found ompi / op framework

=== Found ompi / osc framework
--- Found ompi / osc / portals4 component
    Found configure.m4 file
--- Found ompi / osc / pt2pt component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/osc/rdma -> Excluded
--- Found ompi / osc / sm component

=== Found ompi / pml framework
--- Found ompi / pml / bfo component
    Found configure.m4 file
--- Found ompi / pml / cm component
--- Found ompi / pml / crcpw component
    Found configure.m4 file
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/pml/monitoring -> Excluded
--- Found ompi / pml / ob1 component
    Found configure.m4 file
--- Found ompi / pml / ucx component
    Found configure.m4 file
--- Found ompi / pml / v component
    Found configure.m4 file
--- Found ompi / pml / yalla component
    Found configure.m4 file

=== Found ompi / pubsub framework
--- Found ompi / pubsub / orte component
    Found configure.m4 file
--- Found ompi / pubsub / pmi component
    Found configure.m4 file

=== Found ompi / rcache framework
--- Found ompi / rcache / vma component

=== Found ompi / rte framework
    Found framework configure.m4 file
--- Found ompi / rte / orte component
    Found configure.m4 file

=== Found ompi / sbgp framework
--- Found ompi / sbgp / basesmsocket component
    Found configure.m4 file
--- Found ompi / sbgp / basesmuma component
--- Found ompi / sbgp / p2p component
    Found configure.m4 file

=== Found ompi / sharedfp framework
    Found framework configure.m4 file
--- Found ompi / sharedfp / individual component
--- Found ompi / sharedfp / lockedfile component
--- Found ompi / sharedfp / sm component

=== Found ompi / topo framework
--- Found ompi / topo / basic component
    => No sentinel file found in /Users/jrhammon/Work/MPI/OMPI/github/ompi/mca/topo/treematch -> Excluded

=== Found ompi / vprotocol framework
--- Found ompi / vprotocol / example component
    => Ignored (found .ompi_ignore file)
--- Found ompi / vprotocol / pessimist component

*** Found oshmem project

=== Found oshmem / atomic framework
--- Found oshmem / atomic / basic component
--- Found oshmem / atomic / mxm component
    Found configure.m4 file
--- Found oshmem / atomic / ucx component
    Found configure.m4 file

=== Found oshmem / memheap framework
    Found framework configure.m4 file
--- Found oshmem / memheap / buddy component
--- Found oshmem / memheap / ptmalloc component

=== Found oshmem / scoll framework
--- Found oshmem / scoll / basic component
--- Found oshmem / scoll / fca component
    Found configure.m4 file
--- Found oshmem / scoll / mpi component

=== Found oshmem / spml framework
    Found framework configure.m4 file
--- Found oshmem / spml / ikrit component
    Found configure.m4 file
--- Found oshmem / spml / ucx component
    Found configure.m4 file
--- Found oshmem / spml / yoda component

=== Found oshmem / sshmem framework
--- Found oshmem / sshmem / mmap component
    Found configure.m4 file
--- Found oshmem / sshmem / sysv component
    Found configure.m4 file
--- Found oshmem / sshmem / verbs component
    Found configure.m4 file
Missing orte/mca/common/Makefile.am at ./autogen.sh line 94.
rhc54 commented 8 years ago

The problem is a lingering remnant from your update. When there is a difference in the file system (e.g., a directory is removed), but you have built the code before and so there is build product in that directory, git doesn't remove it. Thus, autogen sees the common directory and looks for the Makefile.am that should be in it, but doesn't find it because that file no longer exists.

Bottom line: whenever you do an update to a branch like this, you should do a "git clean -dfx" before building to ensure that all the cruft gets removed.

jeffhammond commented 8 years ago

I have not built the code and there is no build directory yet. Obviously, I cannot run configure if autogen.pl fails...

In any case, I can run autogen.pl if I rm -rf * && git checkout ., but I don't consider that an acceptable solution. This sort of issue has never happened with any other Autotools-based project I use...

hppritcha commented 8 years ago

The Open MPI project is pretty complex and dynamic. If you are working off of master or one of the other active git branches, you can expect this kind of behavior. One option to avoid these sorts of issues is to work off of the release tarballs at http://www.open-mpi.org/software/ompi/v1.10/ or friends. Then you don't need to use autogen.pl at all.

rhc54 commented 8 years ago

Or, as I said, you can just run "git clean -dfx" to remove any stale stuff and then run autogen et al. It's just a git artifact caused by having multiple branches in the same repo - when you change between them, the directories that are in one and not the other branch carry across. Thus, you have to remove them with the "clean" command.

jeffhammond commented 8 years ago

@hppritcha

Yes, I'm aware of the benefits of rehearse tarballs every time I have to build the latest Autotools from source in order to configure MPICH ;-)

In this case, I'm hoping to diagnose a bug so that it can be fixed before 1.10 drops, in which case tracking Git will save me the trouble of starting from scratch if a patch is made available.

ggouaillardet commented 8 years ago

As far as I am concerned, I have three clones (master, v2.x and v1.10) and I never checkout from one branch to an other. that saves some time since I only run autogen.pl/configure when needed, and I do not run into these kind of troubles. I set remote repos, but I only use them for cherry-picking purpose. on my laptop, I also mirror ompi and ompi-release (git clone --mirror) so i do not have to connect multiple times to github when updating my clones.

my 0.02 US$

jeffhammond commented 8 years ago

@ggouaillardet I don't have the luxury of making those space-time trade-offs:

$ df -h
Filesystem                          Size   Used  Avail Capacity  iused   ifree %iused  Mounted on
/dev/disk1                         233Gi  219Gi   13Gi    95% 57578528 3408991   94%   /

This is the state of things after extensive pruning. As much as I'd love to delete my Windows VM to clear up 100 GB or so, it's essential to my day job :rage2:

Anyways, feel free to close this ticket. Now that I know the cause of the issue, I will be able to solve it whenever it appears again in the future.

PS If you did not already know about Wolfenstein emoji support on Github, you're welcome :octocat: :trophy: :unlock: