search

ldbc / ldbc_snb_interactive_v1_impls

Reference implementations for LDBC Social Network Benchmark's Interactive workload.
https://ldbcouncil.org/benchmarks/snb-interactive
Apache License 2.0
97 stars 85 forks source link

LdbcShortQuery4MessageContent is not listed in benchmark result #379

Closed cpuwar closed 1 year ago

cpuwar commented 1 year ago

I have tested benchmark several times with the following setting. Even though I increased operation count to 30000, LdbcShortQuery4MessageContent is not found in the report. So I disabled all queries except it and tested operation count up to 1000000, but the benchmark still didn't execute the query.


dh@bigroom:/mnt/nvme/ldbc_snb_interactive_impls/cypher$ driver/benchmark.sh 
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
13:15:17.048 [main] INFO  ExecuteWorkloadMode - Driver Configuration
13:15:17.054 [main] INFO  ExecuteWorkloadMode - Workload Start Time:    2023-03-27 - 04:15:22.046
Parameters:
    Mode:                           execute_benchmark
    Name:                           LDBC-SNB
    DB:                             org.ldbcouncil.snb.impls.workloads.cypher.interactive.CypherInteractiveDb
    Workload:                       org.ldbcouncil.snb.driver.workloads.interactive.LdbcSnbInteractiveWorkload
    Operation Count:                30,000
    Warmup Count:                   1,500
    Skip Count:                     0
    Worker Threads:                 1
    Status Display Interval:        00:01.000 (m:s.ms)
    Time Unit:                      MILLISECONDS
    Results Directory:              /mnt/nvme/ldbc_snb_interactive_impls/cypher/results
    Time Compression Ratio:         0.0010000
    Validation Parameter Size:      0
    Validation Serialization Check: true
    Validation Database File:       null
    Spinner Sleep Duration:         00:00.001 (m:s.ms) / 1 (ms)
    Print Help:                     false
    Ignore Scheduled Start Times:   false
    User-defined parameters:
        endpoint = bolt://localhost:7687
        flush_log = false
        ldbc.snb.interactive.LdbcDelete1RemovePerson_enable = true
        ldbc.snb.interactive.LdbcDelete2RemovePostLike_enable = false
        ldbc.snb.interactive.LdbcDelete3RemoveCommentLike_enable = false
        ldbc.snb.interactive.LdbcDelete4RemoveForum_enable = true
        ldbc.snb.interactive.LdbcDelete5RemoveForumMembership_enable = false
        ldbc.snb.interactive.LdbcDelete6RemovePostThread_enable = false
        ldbc.snb.interactive.LdbcDelete7RemoveCommentSubthread_enable = false
        ldbc.snb.interactive.LdbcDelete8RemoveFriendship_enable = false
        ldbc.snb.interactive.LdbcInsert1AddPerson_enable = true
        ldbc.snb.interactive.LdbcInsert2AddPostLike_enable = false
        ldbc.snb.interactive.LdbcInsert3AddCommentLike_enable = false
        ldbc.snb.interactive.LdbcInsert4AddForum_enable = true
        ldbc.snb.interactive.LdbcInsert5AddForumMembership_enable = false
        ldbc.snb.interactive.LdbcInsert6AddPost_enable = false
        ldbc.snb.interactive.LdbcInsert7AddComment_enable = false
        ldbc.snb.interactive.LdbcInsert8AddFriendship_enable = false
        ldbc.snb.interactive.LdbcQuery10_enable = false
        ldbc.snb.interactive.LdbcQuery11_enable = false
        ldbc.snb.interactive.LdbcQuery12_enable = false
        ldbc.snb.interactive.LdbcQuery13a_enable = false
        ldbc.snb.interactive.LdbcQuery13b_enable = false
        ldbc.snb.interactive.LdbcQuery14a_enable = false
        ldbc.snb.interactive.LdbcQuery14b_enable = false
        ldbc.snb.interactive.LdbcQuery1_enable = true
        ldbc.snb.interactive.LdbcQuery2_enable = false
        ldbc.snb.interactive.LdbcQuery3a_enable = false
        ldbc.snb.interactive.LdbcQuery3b_enable = false
        ldbc.snb.interactive.LdbcQuery4_enable = true
        ldbc.snb.interactive.LdbcQuery5_enable = false
        ldbc.snb.interactive.LdbcQuery6_enable = false
        ldbc.snb.interactive.LdbcQuery7_enable = false
        ldbc.snb.interactive.LdbcQuery8_enable = false
        ldbc.snb.interactive.LdbcQuery9_enable = false
        ldbc.snb.interactive.LdbcShortQuery1PersonProfile_enable = true
        ldbc.snb.interactive.LdbcShortQuery2PersonPosts_enable = false
        ldbc.snb.interactive.LdbcShortQuery3PersonFriends_enable = false
        ldbc.snb.interactive.LdbcShortQuery4MessageContent_enable = true
        ldbc.snb.interactive.LdbcShortQuery5MessageCreator_enable = false
        ldbc.snb.interactive.LdbcShortQuery6MessageForum_enable = false
        ldbc.snb.interactive.LdbcShortQuery7MessageReplies_enable = false
        ldbc.snb.interactive.parameters_dir = ../parameters-sf1/
        ldbc.snb.interactive.scale_factor = 1
        ldbc.snb.interactive.short_read_dissipation = 0.2
        ldbc.snb.interactive.updates_dir = ../update-streams-sf1/
        log_delayed = true
        password = admin
        peer_identifiers = 
        printQueryNames = false
        printQueryResults = false
        printQueryStrings = false
        queryDir = queries/
        results_log = true
        user = neo4j
        workload_statistics = false
 --------------------
 --- Warmup Phase ---
 --------------------
------------------------------------------------------------------------------
Operation Count:                        677
Duration:                               00:10.548.000 (m:s.ms.us)
Throughput:                             64.18 (op/s)
------------------------------------------------------------------------------
    LdbcDelete1RemovePerson       Count: 1     Mean: 212.00 ms
    LdbcDelete4RemoveForum        Count: 3     Mean: 69.00 ms
    LdbcInsert1AddPerson          Count: 2     Mean: 53.00 ms
    LdbcInsert4AddForum           Count: 23    Mean: 22.43 ms
    LdbcQuery1                    Count: 251   Mean: 5.39 ms
    LdbcQuery4                    Count: 181   Mean: 28.78 ms
    LdbcShortQuery1PersonProfile  Count: 216   Mean: 1.62 ms
------------------------------------------------------------------------------
 -----------------
 --- Run Phase ---
 -----------------
------------------------------------------------------------------------------
Operation Count:                        12,293
Duration:                               03:25.798.000 (m:s.ms.us)
Throughput:                             59.73 (op/s)
Start Time (Korean Standard Time):      2023-03-27 - 13:15:43.493
Finish Time (Korean Standard Time):     2023-03-27 - 13:19:09.291
------------------------------------------------------------------------------
    LdbcDelete1RemovePerson
        Units:              MILLISECONDS
        Count:              3
        Min:                36
        Max:                166
        Mean:               87.67
        50th Percentile:    61
        90th Percentile:    166
        95th Percentile:    166
        99th Percentile:    166
    LdbcDelete4RemoveForum
        Units:              MILLISECONDS
        Count:              12
        Min:                10
        Max:                24
        Mean:               18.08
        50th Percentile:    19
        90th Percentile:    24
        95th Percentile:    24
        99th Percentile:    24
    LdbcInsert1AddPerson
        Units:              MILLISECONDS
        Count:              28
        Min:                3
        Max:                29
        Mean:               12.46
        50th Percentile:    13
        90th Percentile:    16
        95th Percentile:    18
        99th Percentile:    29
    LdbcInsert4AddForum
        Units:              MILLISECONDS
        Count:              443
        Min:                2
        Max:                13
        Mean:               8.83
        50th Percentile:    11
        90th Percentile:    12
        95th Percentile:    12
        99th Percentile:    13
    LdbcQuery1
        Units:              MILLISECONDS
        Count:              4,701
        Min:                0
        Max:                12
        Mean:               0.65
        50th Percentile:    1
        90th Percentile:    1
        95th Percentile:    1
        99th Percentile:    2
    LdbcQuery4
        Units:              MILLISECONDS
        Count:              3,171
        Min:                12
        Max:                50
        Mean:               26.76
        50th Percentile:    27
        90th Percentile:    31
        95th Percentile:    33
        99th Percentile:    37
    LdbcShortQuery1PersonProfile
        Units:              MILLISECONDS
        Count:              3,935
        Min:                0
        Max:                11
        Mean:               0.08
        50th Percentile:    0
        90th Percentile:    0
        95th Percentile:    1
        99th Percentile:    1
------------------------------------------------------------------------------

dh@bigroom:/mnt/nvme/ldbc_snb_interactive_impls/cypher$ scripts/restore-database.sh 

dh@bigroom:/mnt/nvme/ldbc_snb_interactive_impls/cypher$ driver/benchmark.sh 
WARNING: sun.reflect.Reflection.getCallerClass is not supported. This will impact performance.
13:40:02.152 [main] INFO  ExecuteWorkloadMode - Driver Configuration
13:40:02.158 [main] INFO  ExecuteWorkloadMode - Workload Start Time:    2023-03-27 - 04:40:07.150
Parameters:
    Mode:                           execute_benchmark
    Name:                           LDBC-SNB
    DB:                             org.ldbcouncil.snb.impls.workloads.cypher.interactive.CypherInteractiveDb
    Workload:                       org.ldbcouncil.snb.driver.workloads.interactive.LdbcSnbInteractiveWorkload
    Operation Count:                1,000,000
    Warmup Count:                   1,500
    Skip Count:                     0
    Worker Threads:                 1
    Status Display Interval:        00:01.000 (m:s.ms)
    Time Unit:                      MILLISECONDS
    Results Directory:              /mnt/nvme/ldbc_snb_interactive_impls/cypher/results
    Time Compression Ratio:         0.0010000
    Validation Parameter Size:      0
    Validation Serialization Check: true
    Validation Database File:       null
    Spinner Sleep Duration:         00:00.001 (m:s.ms) / 1 (ms)
    Print Help:                     false
    Ignore Scheduled Start Times:   false
    User-defined parameters:
        endpoint = bolt://localhost:7687
        flush_log = false
        ldbc.snb.interactive.LdbcDelete1RemovePerson_enable = false
        ldbc.snb.interactive.LdbcDelete2RemovePostLike_enable = false
        ldbc.snb.interactive.LdbcDelete3RemoveCommentLike_enable = false
        ldbc.snb.interactive.LdbcDelete4RemoveForum_enable = false
        ldbc.snb.interactive.LdbcDelete5RemoveForumMembership_enable = false
        ldbc.snb.interactive.LdbcDelete6RemovePostThread_enable = false
        ldbc.snb.interactive.LdbcDelete7RemoveCommentSubthread_enable = false
        ldbc.snb.interactive.LdbcDelete8RemoveFriendship_enable = false
        ldbc.snb.interactive.LdbcInsert1AddPerson_enable = false
        ldbc.snb.interactive.LdbcInsert2AddPostLike_enable = false
        ldbc.snb.interactive.LdbcInsert3AddCommentLike_enable = false
        ldbc.snb.interactive.LdbcInsert4AddForum_enable = false
        ldbc.snb.interactive.LdbcInsert5AddForumMembership_enable = false
        ldbc.snb.interactive.LdbcInsert6AddPost_enable = false
        ldbc.snb.interactive.LdbcInsert7AddComment_enable = false
        ldbc.snb.interactive.LdbcInsert8AddFriendship_enable = false
        ldbc.snb.interactive.LdbcQuery10_enable = false
        ldbc.snb.interactive.LdbcQuery11_enable = false
        ldbc.snb.interactive.LdbcQuery12_enable = false
        ldbc.snb.interactive.LdbcQuery13a_enable = false
        ldbc.snb.interactive.LdbcQuery13b_enable = false
        ldbc.snb.interactive.LdbcQuery14a_enable = false
        ldbc.snb.interactive.LdbcQuery14b_enable = false
        ldbc.snb.interactive.LdbcQuery1_enable = false
        ldbc.snb.interactive.LdbcQuery2_enable = false
        ldbc.snb.interactive.LdbcQuery3a_enable = false
        ldbc.snb.interactive.LdbcQuery3b_enable = false
        ldbc.snb.interactive.LdbcQuery4_enable = false
        ldbc.snb.interactive.LdbcQuery5_enable = false
        ldbc.snb.interactive.LdbcQuery6_enable = false
        ldbc.snb.interactive.LdbcQuery7_enable = false
        ldbc.snb.interactive.LdbcQuery8_enable = false
        ldbc.snb.interactive.LdbcQuery9_enable = false
        ldbc.snb.interactive.LdbcShortQuery1PersonProfile_enable = false
        ldbc.snb.interactive.LdbcShortQuery2PersonPosts_enable = false
        ldbc.snb.interactive.LdbcShortQuery3PersonFriends_enable = false
        ldbc.snb.interactive.LdbcShortQuery4MessageContent_enable = true
        ldbc.snb.interactive.LdbcShortQuery5MessageCreator_enable = false
        ldbc.snb.interactive.LdbcShortQuery6MessageForum_enable = false
        ldbc.snb.interactive.LdbcShortQuery7MessageReplies_enable = false
        ldbc.snb.interactive.parameters_dir = ../parameters-sf1/
        ldbc.snb.interactive.scale_factor = 1
        ldbc.snb.interactive.short_read_dissipation = 0.2
        ldbc.snb.interactive.updates_dir = ../update-streams-sf1/
        log_delayed = true
        password = admin
        peer_identifiers = 
        printQueryNames = false
        printQueryResults = false
        printQueryStrings = false
        queryDir = queries/
        results_log = true
        user = neo4j
        workload_statistics = false

 --------------------
 --- Warmup Phase ---
 --------------------
------------------------------------------------------------------------------
Operation Count:                        0
Duration:                               00:00.000.000 (m:s.ms.us)
Throughput:                             NaN (op/s)
------------------------------------------------------------------------------
 -----------------
 --- Run Phase ---
 -----------------
------------------------------------------------------------------------------
Operation Count:                        0
Duration:                               00:00.000.000 (m:s.ms.us)
Throughput:                             NaN (op/s)
Start Time (Korean Standard Time):      1970-01-01 - 08:59:59.999
Finish Time (Korean Standard Time):     1970-01-01 - 08:59:59.999
------------------------------------------------------------------------------
szarnyasg commented 1 year ago

Hi,

Great question. The way short queries are defined is that they are triggered by complex queries and other short queries. The output of these complex/short queries are used as the input of the triggered short queries.

The precise connection is in the specification in Table 5.2: Short read queries executed after given complex read and short read queries:

https://ldbcouncil.org/ldbc_snb_docs/ldbc-snb-specification.pdf#page=75

Screenshot 2023-03-27 at 17 02 48

To trigger short query 4, you need complex queries 2, 7, 8, or 9 enabled. Alternatively, short queries 2 and 7 can also trigger it. So you'll need to enable some of these queries.

Your log is useful because it shows that all of these queries are currently disabled.

Gabor

cpuwar commented 1 year ago

Thank you for the kind explanation. Now I understand it was by design.

Dongho.