search

flock-lab / flock

Flock: A Low-Cost Streaming Query Engine on FaaS Platforms
https://flock-lab.github.io/flock/
GNU Affero General Public License v3.0
291 stars 39 forks source link

fix: add nexmark q11 on cloud functions #393

Closed gangliao closed 2 years ago

gangliao commented 2 years ago

Which issue does this PR close?

Closes #.

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

By submitting this pull request

gangliao commented 2 years ago 
[2021-12-30T16:35:37Z INFO  nexmark_bench] Running the NEXMark benchmark with the following options: NexmarkBenchmarkOpt { query_number: 11, debug: true, generators: 1, seconds: 30, events_per_second: 1000, data_sink_type: 0, async_type: false, memory_size: 128 }
[2021-12-30T16:35:37Z INFO  nexmark_bench] Creating lambda function: flock_datasource
[2021-12-30T16:35:37Z INFO  nexmark_bench] Creating lambda function group: Group(("q11-00", 8))
[2021-12-30T16:35:37Z INFO  nexmark_bench] Creating function member: q11-00-00
[2021-12-30T16:35:38Z INFO  nexmark_bench] Creating function member: q11-00-01
[2021-12-30T16:35:38Z INFO  nexmark_bench] Creating function member: q11-00-02
[2021-12-30T16:35:39Z INFO  nexmark_bench] Creating function member: q11-00-03
[2021-12-30T16:35:39Z INFO  nexmark_bench] Creating function member: q11-00-04
[2021-12-30T16:35:40Z INFO  nexmark_bench] Creating function member: q11-00-05
[2021-12-30T16:35:40Z INFO  nexmark_bench] Creating function member: q11-00-06
[2021-12-30T16:35:41Z INFO  nexmark_bench] Creating function member: q11-00-07
[2021-12-30T16:35:41Z INFO  nexmark_bench] [OK] Invoking NEXMark source function: flock_datasource by generator 0
[2021-12-30T16:35:41Z INFO  nexmark_bench] Waiting for the current invocations to be logged.
[2021-12-30T16:35:46Z INFO  flock::driver::logwatch::tail] Sending log request FilterLogEventsRequest { end_time: None, filter_pattern: None, limit: Some(100), log_group_name: "/aws/lambda/flock_datasource", log_stream_name_prefix: None, log_stream_names: None, next_token: None, start_time: Some(1640882086895) }
[2021-12-30T16:35:48Z INFO  flock::driver::logwatch::tail] [OK] Got response from AWS CloudWatch Logs.
[2021-12-30T16:35:48Z INFO  flock::services] Got a lastlog response
[2021-12-30T16:35:48Z INFO  flock::services] Waiting 5s before requesting logs again...
[2021-12-30T16:35:53Z INFO  flock::driver::logwatch::tail] Sending log request FilterLogEventsRequest { end_time: None, filter_pattern: None, limit: Some(100), log_group_name: "/aws/lambda/flock_datasource", log_stream_name_prefix: None, log_stream_names: None, next_token: None, start_time: Some(1640882086895) }
[2021-12-30T16:35:56Z INFO  flock::driver::logwatch::tail] [OK] Got response from AWS CloudWatch Logs.
2021-12-30 16:35:43 START RequestId: d76eaf26-d9cd-484b-9f6f-11215c67fa4f Version: $LATEST
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Generating events for 30s over 1 partitions.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::nexmark::source] Nexmark Benchmark: Query 11
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::nexmark::source] NEXMarkSource { config: Config { args: {"seconds": "30", "threads": "1", "events-per-second": "1000"} }, window: SessionWindow(Seconds(10)) }
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::nexmark::source] [OK] Generate nexmark events.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 0: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 1: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 2: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 3: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 4: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 5: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 6: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 7: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 8: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 9: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 10: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 11: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 12: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 13: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 14: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 15: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 16: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 17: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 18: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 19: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 20: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 21: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 22: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 23: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 24: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 25: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::window] Session window -> function name: q11-00-04
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-04.
2021-12-30 16:35:43 [2021-12-30T16:35:43Z INFO  flock::window] [OK] Event 0 - function payload bytes: 1740
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 26: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 27: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 28: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] Session window -> function name: q11-00-00
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] Session window -> function name: q11-00-00
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] Session window -> function name: q11-00-00
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-00.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-00.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Event 0 - function payload bytes: 1525
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Event 0 - function payload bytes: 2249
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-00.
2021-12-30 16:35:44 [2021-12-30T16:35:44Z INFO  flock::window] [OK] Event 0 - function payload bytes: 2800
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::datasource::nexmark::nexmark] Selecting events for epoch 29: 20 persons, 60 auctions, 920 bids.
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] Session window -> function name: q11-00-02
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] Session window -> function name: q11-00-02
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] Session window -> function name: q11-00-02
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] Session window -> function name: q11-00-02
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-02.
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-02.
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Event 0 - function payload bytes: 1690
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Event 0 - function payload bytes: 2153
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-02.
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Send 1 events from a session window to function: q11-00-02.
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Event 0 - function payload bytes: 2025
2021-12-30 16:35:46 [2021-12-30T16:35:46Z INFO  flock::window] [OK] Event 0 - function payload bytes: 1670
2021-12-30 16:35:47 END RequestId: d76eaf26-d9cd-484b-9f6f-11215c67fa4f
2021-12-30 16:35:47 REPORT RequestId: d76eaf26-d9cd-484b-9f6f-11215c67fa4f      Duration: 3768.45 ms    Billed Duration: 3781 ms        Memory Size: 2048 MB    Max Memory Used: 34 MBInit Duration: 12.03 ms
[2021-12-30T16:35:56Z INFO  flock::services] Got a Token response
-------------------------------------------------------------
gangliao commented 2 years ago

The problem with consistent hashing is that closer timestamps will be mapped to the same function. We need a better way to handle this.