search

ldbc / ldbc_snb_interactive_v1_impls

Reference implementations for LDBC Social Network Benchmark's Interactive workload.
https://ldbcouncil.org/benchmarks/snb-interactive
Apache License 2.0
97 stars 85 forks source link
snb

readme

LDBC logo

LDBC SNB Interactive v1 workload implementations

Build Status

Reference implementations of the LDBC Social Network Benchmark's Interactive workload (paper, specification on GitHub pages, specification on arXiv).

To get started with the LDBC SNB benchmarks, check out our introductory presentation: The LDBC Social Network Benchmark (PDF).

Notes

:warning: Please keep in mind the following when using this repository.

Implementations

We provide three reference implementations:

Additional implementations:

For detailed instructions, consult the READMEs of the projects.

To build a subset of the projects, use Maven profiles, e.g. to build the reference implementations, run:

mvn clean package -DskipTests -Pcypher,postgres

User's guide

Building the project

This project uses Java 11.

To build the project, run:

scripts/build.sh

Inputs

The benchmark framework relies on the following inputs produced by the SNB Datagen:

Driver modes

For each implementation, it is possible to perform to perform the run in one of the SNB driver's three modes: create validation parameters, validate, and benchmark. The execution in all three modes should be started after the initial data set was loaded into the system under test.

  1. Create validation parameters with the driver/create-validation-parameters.sh script.

    • Inputs:
      • The query substitution parameters are taken from the directory set in ldbc.snb.interactive.parameters_dir configuration property.
      • The update streams are the updateStream_0_0_{forum,person}.csv files from the location set in the ldbc.snb.interactive.updates_dir configuration property.
      • For this mode, the query frequencies are set to a uniform 1 value to ensure the best average test coverage.
    • Output: The results will be stored in the validation parameters file (e.g. validation_params.csv) file set in the create_validation_parameters configuration property.
    • Parallelism: The execution must be single-threaded to ensure a deterministic order of operations.

  2. Validate against an existing reference output (called "validation parameters") with the driver/validate.sh script.

    • Input:
      • The query substitution parameters are taken from the validation parameters file (e.g. validation_params.csv) file set in the validate_database configuration property.
      • The update operations are also based on the content of the validation parameters file.
    • Output:
      • The validation either passes of fails.
      • The per query results of the validation are printed to the console.
      • If the validation failed, the results are saved to the validation_params-failed-expected.json and validation_params-failed-actual.json files.
    • Parallelism: The execution must be single-threaded to ensure a deterministic order of operations.

  3. Run the benchmark with the driver/benchmark.sh script.

    • Inputs:
      • The query substitution parameters are taken from the directory set in ldbc.snb.interactive.parameters_dir configuration property.
      • The update streams are the updateStream_*_{forum,person}.csv files from the location set in the ldbc.snb.interactive.updates_dir configuration property.
        • To get 2n write threads, the framework requires n updateStream_*_forum.csv and n updateStream_*_person.csv files.
        • If you are generating the data sets from scratch, set ldbc.snb.datagen.serializer.numUpdatePartitions to n in the data generator to get produce these.
      • The goal of the benchmark is the achieve the best (lowest possible) time_compression_ratio value while ensuring that the 95% on-time requirement is kept (i.e. 95% of the queries can be started within 1 second of their scheduled time). If your benchmark run returns "failed schedule audit", increase this number (which lowers the time compression rate) until it passes.
      • Set the thread_count property to the size of the thread pool for read operations.
      • For audited benchmarks, ensure that the warmup and operation_count properties are set so that the warmup and benchmark phases last for 30+ minutes and 2+ hours, respectively.
    • Output:
      • Passed or failed the "schedule audit" (the 95% on-time requirement).
      • The throughput achieved in the run (operations/second).
      • The detailed results of the benchmark are printed to the console and saved in the results/ directory.
    • Parallelism: Multi-threaded execution is recommended to achieve the best result.

For more details on validating and benchmarking, visit the driver's documentation.

Developer's guide

To create a new implementation, it is recommended to use one of the existing ones: the Neo4j implementation for graph database management systems and the PostgreSQL implementation for RDBMSs.

The implementation process looks roughly as follows:

  1. Create a bulk loader which loads the initial data set to the database.
  2. Implement the complex and short reads queries (22 in total).
  3. Implement the 7 update queries.
  4. Test the implementation against the reference implementations using various scale factors.
  5. Optimize the implementation.

Data sets

Benchmark data sets

To generate the benchmark data sets, use the Hadoop-based LDBC SNB Datagen.

The key configurations are the following:

Pre-generated data sets

Producing large-scale data sets requires non-trivial amounts of memory and computing resources (e.g. SF100 requires 24GB memory and takes about 4 hours to generate on a single machine). To mitigate this, we have pregenerated data sets using 9 different serializers and the update streams using 17 different partition numbers:

The data sets are available at the SURF/CWI data repository. We also provide direct links and a download script (which stages the data sets from tape storage if they are not immediately available).

Validation parameters

We provide validation parameters for SF0.1 to SF10. These were produced using the Neo4j reference implementation.

Test data set

Small test data sets are placed in the cypher/test-data/ directory for Neo4j and in the postgres/test-data/ for the SQL systems.

To generate a data set with the same characteristics, see the documentation on generating the test data set.

Preparing for an audited run

Implementations of the Interactive workload can be audited by a certified LDBC auditor. The Auditing Policies chapter of the specification describes the auditing process and the required artifacts. If you are considering commissioning an LDBC SNB audit, please study the auditing process document and the audit questionnaire.

Determining the best TCR

  1. Select a scale factor and configure the driver/benchmark.properties file as described in the Driver modes section.
  2. Load the data set with scripts/load-in-one-step.sh.
  3. Create a backup with scripts/backup-database.sh.
  4. Run the driver/determine-best-tcr.sh.
  5. Once the "best TCR" value has been determined, test it with a full workload (at least 0.5h for warmup operation and at least 2h of benchmark time), and make further adjustments if necessary.

Recommendations

We have a few recommendations for creating audited implementations. (These are not requirements – implementations are allowed to deviate from these recommendations.)