search

ComputationalRadiationPhysics / graybat

Graph Approach for Highly Generic Communication Schemes Based on Adaptive Topologies :satellite:
Other
8 stars 4 forks source link

spack package #144

Open ax3l opened 6 years ago

ax3l commented 6 years ago

Update: Development updates are now in https://github.com/ComputationalRadiationPhysics/spack-repo

Here is a draft of a spack package. https://github.com/ax3l/spack/tree/topic-graybat

The main things:

ccing @erikzenker @fabian-jung

Snapshot

(branch above will be regularly rebased if necessary)

from spack import *

class Graybat(CMakePackage):
    """Graph Approach for Highly Generic Communication Schemes Based on Adaptive Topologies"""

    homepage = "https://github.com/ComputationalRadiationPhysics/graybat"
    url      = "https://github.com/ComputationalRadiationPhysics/graybat/archive/1.2.0.tar.gz"

    # version('develop', branch='remove_grpc',
    #         git='https://github.com/fabian-jung/graybat.git')
    version('develop', branch='dev',
            git='https://github.com/ComputationalRadiationPhysics/graybat.git')
    version('master', branch='master',
            git='https://github.com/ComputationalRadiationPhysics/graybat.git')
    # todo: only add first working version
    version('1.2.0', 'd47200e99d712d33e6e6a5f0ad946c6a')

    # this should not be necessary when all variants are properly optional
    build_targets = ['all', 'signaling']

    # C++14
    conflicts('%gcc@:5')
    conflicts('%clang@:3.4')

    variant('mpi', default=True,
            description='Enable MPI communication policy')
    variant('zeromq', default=True,
            description='Enable ZeroMQ communication policy')
    variant('metis', default=False,
            description='Enable graph partitioning mapping')
    variant('grpc', default=True,
            description='Enable gRPC signaling')

    depends_on('cmake@3.0.2:', type='build')
    depends_on('boost@1.61.0:1.62.0~mpi', type='link', when='~mpi')
    depends_on('boost@1.61.0:1.62.0+mpi', type='link', when='+mpi')
    depends_on('mpi', type='link', when='+mpi')
    depends_on('cppzmq@4.2.2:', type='link', when='+zeromq')
    depends_on('metis@5.1.0:', type='link', when='+metis')
    depends_on('protobuf@3.4.0:')
    depends_on('grpc@1.6.0+shared', when='+grpc')
erikzenker commented 6 years ago

Cool, thank you very much!!!

I just tested the package and you are right it does not succeed. On my computer it fails on the linking step do you have the same issue?

Nevertheless, I will provide a version with optional zmq, gprc, protobuf and signaling. Such that, it will be possible to build with mpi only.

ax3l commented 6 years ago

Nevertheless, I will provide a version with optional zmq, gprc, protobuf and signaling. Such that, it will be possible to build with mpi only.

Yes, that would be excellent! ✨ You could even make MPI itself optional, too if that is possible. (Isn't it already optional? :) )

I just tested the package and you are right it does not succeed. [...]

I have currently the following patches done in upstream projects:

When the protobuf and gRPC fixes are merged I still need to publish the spack gRPC package. For graybat, I would make all the above mentioned dependencies "variants" of the graybat spack package as soon as they are optional.

ax3l commented 6 years ago

@erikzenker @fabian-jung just a note: I moved our in-development spack packages to a central repo in https://github.com/ComputationalRadiationPhysics/spack-repo

Please see the instructions therein, so you don't need to depend on some weird branches of mine.

ax3l commented 6 years ago

grpc@develop dependency in spack-repo updated:

(there is still no gRPC 1.5.0 release we can put a fix on)

ax3l commented 6 years ago

currently I also experience a missing zlib when trying to build graybat@develop:

libz.so.1: error adding symbols: DSO missing from command line
ax3l commented 6 years ago

ccing @fabian-jung @erikzenker

ax3l commented 6 years ago

@fabian-jung @erikzenker I just managed to build graybat 1.2.0 and dev with spack and gRPC 1.6.0.

Like this (using gcc 6.3.0: %gcc@6.3.0 as a suffix in spack install graybat)

# the standard spack install
git clone https://github.com/llnl/spack.git $HOME/src/spack
. $HOME/src/spack/share/spack/setup-env.sh

# our graybat recipe
git clone https://github.com/ComputationalRadiationPhysics/spack-repo.git $HOME/src/spack-repo
spack repo add $HOME/src/spack-repo

# now build graybat
spack install graybat@1.2.0
spack install graybat@develop

@erikzenker any progress on #145? :)

erikzenker commented 6 years ago

145 is in good shape. I will test it with the current spack version.

ax3l commented 6 years ago

perfect, as soon as #145 is in we can add proper spack variants

erikzenker commented 6 years ago

I failed to install protobuf. I will continue to investigate after my vacations in 2 weeks.

==> Installing protobuf
==> Fetching https://github.com/google/protobuf/archive/v3.4.0.tar.gz
######################################################################## 100,0%
==> Error: ChecksumError: md5 checksum failed for /home/erik/projects/spack/var/spack/stage/protobuf-3.4.0-vp2si5mlgalhsqre64yw35vaga62ge6k/v3.4.0.tar.gz
    Expected 4f47de212ef665ea619f5f97083c6781 but got 1d077a7d4db3d75681f5c333f2de9b1a
ChecksumError: ChecksumError: md5 checksum failed for /home/erik/projects/spack/var/spack/stage/protobuf-3.4.0-vp2si5mlgalhsqre64yw35vaga62ge6k/v3.4.0.tar.gz
    Expected 4f47de212ef665ea619f5f97083c6781 but got 1d077a7d4db3d75681f5c333f2de9b1a

/home/erik/projects/spack/lib/spack/spack/package.py:962, in do_fetch:
     29            self._fetch_time = time.time() - start_time
     30    
     31            if spack.do_checksum and self.version in self.versions:
  >> 32                self.stage.check()
     33    
     34            self.stage.cache_local()
ax3l commented 6 years ago

oh, true. that's a weird issue currently in spack with the way it discovers potential URL candidates for versions: https://github.com/LLNL/spack/issues/5356

the only thing you can do right now is: open the (/home/erik/projects/spack/) var/spack/repos/builtin/packages/protobuf/packages.py file and just put in the md5sum you see on the terminal behind "but got ...". this will let it pass for now until the issue is fixed upstream

ax3l commented 6 years ago

@erikzenker I have a PR open in https://github.com/LLNL/spack/pull/5373 that should fix your fetch/md5 issue in protobuf :) update: merged :)