COMPSs/PyCOMPSs Framework

Website • Documentation • Releases • Slack • ✉

COMP Superscalar (COMPSs) is a programming model which aims to ease the development of applications for distributed infrastructures, such as Clusters, Grids and Clouds. COMP Superscalar also features a runtime system that exploits the inherent parallelism of applications at execution time.

Repository Structure

builders: Packages, scripts for local installations, scripts for supercomputers installation and package building scripts
compss : Programming Model, Bindings and Runtime source code
dependencies : Embeded dependencies
files : Dependency files (i.e. paraver configurations)
tests : Integration tests
utils : Misc utils (i.e. OVA scripts, Docker generation, Storage implementations)

Supported Systems

COMPSs/PyCOMPSs fully supports Linux systems for x86_64, amd64, ppc64, arm64 and riscv64 architectures. macOS systems are also supported with some limitations.

Building From Sources

Follow the next steps to build COMPSs in your current machine.

1. Install dependencies

For an updated list of COMPSs dependencies and how to install them for different systems visit the dependencies section of the COMPSs documentation website.

2. Get GIT submodules

Before installing COMPSs you need to download the git submodules that contain its dependencies. To do that execute the following two commands at the root of the repository.

./submodules_get.sh

3. Build COMPSs

Note: Remember to install the COMPSs dependencies and to get the GIT submodules before trying to build COMPSs from sources.

Building COMPSs for all users (not supported in macOS)

cd builders/
INSTALL_DIR=/opt/COMPSs/
sudo -E ./buildlocal [options] ${INSTALL_DIR}

Building COMPSs for current user

cd builders/

INSTALL_DIR=$HOME/opt/COMPSs/
./buildlocal [options] ${INSTALL_DIR}

For macOS:

cd builders/
alias libtoolize=/usr/local/bin/glibtoolize
alias readlink=/usr/local/bin/greadlink

export LIBTOOL=`which glibtool`
export LIBTOOLIZE=`which glibtoolize`

INSTALL_DIR=$HOME/opt/COMPSs/
./buildlocal -K -T -M ${INSTALL_DIR}

Many COMPSs modules can be activated/deactivated during the build using different options in the buildlocal command. You may check the available options by running the following command:

cd builders
./buildlocal -h

Running docker tests

1. Install Docker and docker-py

Follow these instructions

Docker for Mac. Or, if you prefer to use Homebrew.
Docker for Ubuntu.
Docker for Arch Linux.

Add user to docker group to run docker as non-root user.

Instructions.

2. Build the docker image

Run the following command at the root of the project to build the image that will used for testing. The command create an image named compss and install the current branch into the image.

docker build --target=ci -t compss .

3. Run the tests

To run the tests inside the docker image use the script found in ./tests/scripts/docker_main. This command is a wrapper for the ./main test command so it has de the syntax and options. For example, you can run the first test without retrials as follows:

./docker_main -R -t 1

The docker main command creates a new docker container each time you run it (replacing the last one used). It copies the current framework inside it and runs its tests. Note: the testing scripts assumes you have named the testing image compss.

Please be aware that:

Code changes affecting the tests sources, config files (e.g. local.cfg, and scripts (like ./local) will be visible inside the newly created container.
Code changes affecting the installation will not be visible in the installation because framework is not reinstalled. To do that rebuild the docker image as explained in step 3.
If you run the command once, the container will be available for manual inspection (such as logs). You can log into in issuing docker exec --user jenkins -it compss_test bash and use the CLI as usual.
To delete the created image issue docker rmi compss
To delete the compss_test container use docker rm -f compss_test.

4. Run the tests locally on macOS

In order to run Jenkins tests locally on macOS, GNU sed (gsed) is needed. To install it, use:

brew install gsed

Some environment variables also need to be defined to ensure gsed is used instead of macOS sed.

export PATH="/usr/local/opt/gnu-sed/libexec/gnubin:$PATH

Finally, the NIO_mac.cfg file needs to be updated with any specific features of the local macOS environment, commonly variables such as the java_home, compss_home and runcompss_opts parameters should be updated.

Support

For support please send and e-mail to support-compss@bsc.es

(c) Workflows and Distributed Computing Group (WDC) - Department of Computer Science (CS) - Barcelona Supercomputing Center (BSC)

bsc-wdc / compss

readme