flock-lab / flock

Flock: A Low-Cost Streaming Query Engine on FaaS Platforms
https://flock-lab.github.io/flock/
GNU Affero General Public License v3.0
291 stars 39 forks source link

baseline: use S3 as function comm. medium #373

Closed gangliao closed 2 years ago

gangliao commented 2 years ago

Which issue does this PR close?

Closes #366

Rationale for this change

What changes are included in this PR?

Are there any user-facing changes?

By submitting this pull request

gangliao commented 2 years ago

S3:

[2021-12-19T18:50:47Z INFO  s3_bench] Running the NEXMark benchmark [S3] with the following options: NexmarkBenchmarkOpt { query_number: 5, debug: true, generators: 1, seconds: 10, events_per_second: 100000, data_sink_type: 0, async_type: false }
[2021-12-19T18:50:47Z INFO  s3_bench::nexmark_bench] Creating lambda function: flock_datasource
[2021-12-19T18:50:48Z INFO  s3_bench::nexmark_bench] Creating lambda function group: Group(("q5-00", 8))
[2021-12-19T18:50:48Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-00
[2021-12-19T18:50:48Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-01
[2021-12-19T18:50:49Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-02
[2021-12-19T18:50:49Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-03
[2021-12-19T18:50:49Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-04
[2021-12-19T18:50:51Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-05
[2021-12-19T18:50:52Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-06
[2021-12-19T18:50:52Z INFO  s3_bench::nexmark_bench] Creating function member: q5-00-07
[2021-12-19T18:50:53Z INFO  s3_bench] [OK] Invoking NEXMark source function: flock_datasource
[2021-12-19T18:50:56Z INFO  s3_bench] Recieved response from the source function: Object({"function": String("q5-00-01"), "bucket": String("flock-lab"), "key": String("q5-00-01_payload"), "uuid": String("{\"tid\":\"q5-1639939854\",\"seq_num\":0,\"seq_len\":1}"), "encoding": String("\"Zstd\"")})
[2021-12-19T18:50:56Z INFO  s3_bench] [OK] Invoking NEXMark worker function: q5-00-01
[2021-12-19T18:51:10Z INFO  s3_bench] [OK] Received response: Object({"name": String("q5-00-01"), "sink_type": String("Empty"), "status": String("success")})
[2021-12-19T18:51:10Z INFO  s3_bench] [OK] The NEXMark benchmark [S3] took 17267 milliseconds
gangliao commented 2 years ago

We are at least 5x faster than baseline.