Clean and define a service deployment template relying on Docker Compose

[VincenzoFerme]

@simonedavico define a deployment descriptor we should define to add a BenchFlow service (e.g., collectors or monitors) to BenchFlow.

Here some discussed examples (where the # are generated):

#The service name should be "benchflowServiceName_BoundServiceName"
mysql:
  image: 'benchflow/collectors:mysql_dev'
  # container_name: mysql_db_TRIAL_ID
  environment:
    - KAFKA_HOST=${BENCHFLOW_ENV_KAFKA_IP}
    - MINIO_ALIAS=benchflow
    - MINIO_HOST=http://${BENCHFLOW_ENV_MINIO_IP}:${BENCHFLOW_ENV_MINIO_PORT}
    - MINIO_ACCESSKEYID=${BENCHFLOW_ENV_MINIO_ACCESSKEYID}
    - MINIO_SECRETACCESSKEY=${BENCHFLOW_ENV_MINIO_SECRETACCESSKEY}

    # - BENCHFLOW_EXPERIMENT_ID=camunda
    # - BENCHFLOW_TRIAL_ID=camunda_1O
    # - BENCHFLOW_TRIAL_TOTAL_NUM=1
    - MYSQL_DB_NAME=${BENCHFLOW_BENCHMARK_CONFIG_MYSQL_DB_NAME}
    - TABLE_NAMES=${BENCHFLOW_BENCHMARK_CONFIG_TABLE_NAMES}

    # the IP can be the local IP
    - MYSQL_HOST=${BENCHFLOW_BENCHMARK_BOUNDSERVICE_IP}
    - MYSQL_PORT=${BENCHFLOW_BENCHMARK_BOUNDSERVICE_PORT}
    - MYSQL_USER=${BENCHFLOW_BENCHMARK_CONFIG_MYSQL_USER}
    - MYSQL_USER_PASSWORD=${BENCHFLOW_BENCHMARK_CONFIG_MYSQL_USER_PASSWORD}

    # - BENCHFLOW_CONTAINER_NAME=mysql_db_TRIAL_ID
    - BENCHFLOW_COLLECTOR_NAME=mysql
    - BENCHFLOW_DATA_NAME=mysql

    # - "constraint:node==bull"
  expose:
    - 8080
  ports:
    - '8080' #192.168.41.128::8080

#The service name should be "benchflowServiceName_BoundServiceName"
stats:
  image: 'benchflow/collectors:stats_dev'
  # container_name: stats_camunda_TRIAL_ID
  environment:
    - KAFKA_HOST=${BENCHFLOW_ENV_KAFKA_IP}
    - MINIO_ALIAS=benchflow
    - MINIO_HOST=http://${BENCHFLOW_ENV_MINIO_IP}:${BENCHFLOW_ENV_MINIO_PORT}
    - MINIO_ACCESSKEYID=${BENCHFLOW_ENV_MINIO_ACCESSKEYID}
    - MINIO_SECRETACCESSKEY=${BENCHFLOW_ENV_MINIO_SECRETACCESSKEY}

    # - BENCHFLOW_EXPERIMENT_ID=camunda
    # - BENCHFLOW_TRIAL_ID=camunda_1O
    # - BENCHFLOW_TRIAL_TOTAL_NUM=1
    - CONTAINERS=${BENCHFLOW_BENCHMARK_BOUNDSERVICE_CONTAINER_NAME}

    # - BENCHFLOW_CONTAINER_NAME=stats_camunda_TRIAL_ID
    - BENCHFLOW_COLLECTOR_NAME=stats
    - BENCHFLOW_DATA_NAME=stats

    # - "constraint:node==lisa1"

  volumes:
    - /var/run/docker.sock:/var/run/docker.sock:ro
  expose:
    - 8080
  ports:
    - '8080' #192.168.41.105::8080

[simonedavico]

generated fields

The following fields will be modified/generated when specialising a template like the ones above in a docker-compose file:

• service name will be updated to BenchflowServiceName_collector_BoundServiceName;
• container_name will be set to BenchflowServiceName_collector_ServiceName_TrialId;
• BENCHFLOW_EXPERIMENT_ID: the experiment id;
• BENCHFLOW_TRIAL_ID: the trial id;
• BENCHFLOW_TRIAL_TOTAL: total number of runs;
• CONTAINER_NAME: same as container_name;
• constraint:node==BOUNDSERVICE_SERVER_ALIAS: the alias of the server on which the service will be located;

variables resolution

BenchFlow provides a way to specify some properties, or a part of them, as variables that will be resolved by BenchFlow itself. All these variables' names have to start with the keyword BENCHFLOW; then, BenchFlow provides different ways to access different informations:

• BENCHFLOW_ENV_VARNAME: resolve VARNAME from the BenchFlow environment;
• BENCHFLOW_BENCHMARK_CONFIG_VARNAME: resolve VARNAME from the config section under each service-collector binding;
• BENCHFLOW_BOUNDSERVICE_VARNAME: resolve VARNAME from the service the collector is bound to. Possible values for VARNAME are IP, PORT, CONTAINER_NAME.

[VincenzoFerme]

As discussed on the following issue: https://github.com/benchflow/drivers-maker/issues/14#issue-129881344, we are going to have BenchFlow services (currently monitors and collectors) which deployment is described as per in https://github.com/benchflow/collectors/issues/38#issue-132146550.

The user can currently specify the mapping only for the collector in the benchflow-benchmark file as discussed on the following issue: https://github.com/benchflow/drivers-maker/issues/16#issuecomment-181581031. So we assume that when the user specifies a mapping to a BenchFlow service, this mapping refers to a collector.

We also need to define the mapping between a collector and the monitors that need to be enabled, if the given collector is used. This should be done in the deployment description of the collector itself, so that the https://github.com/benchflow/drivers-maker can resolve the dependency and automatically add the right monitors given the collector specified by the users.

A possible proposal to be integrated in the examples above:

dependencies:
   monitors: cpu, mysql

[VincenzoFerme]

Each collector service also expose at most two APIs, one for starting the collection and one for stopping it. The start API is optional since some of the collector just work offline after the execution (as it is for the mysql one). We also need to define this information in the deployment descriptor. A proposal follows:

endpoints:
   start: /start
   stop: /stop