Problems

[vagetablechicken]

Setup

• Dir /data just a readme doc, and /dataset is empty. Not good. And the way downloading data wget -r -np -R "index.html*" http://43.138.115.238/download/data/; cp -r <dataset directory> ./dataset is not good too. Use wget -r -np -R "index.html*" http://43.138.115.238/download/data/ -P ./dataset

• Give the openmldb quickstart url, users don't know how to start OpenMLDB cluster. https://github.com/decis-bench/febench/blob/fc0d88bb6aa96405ca5b35089aa061d0c4961b8e/README.md#L55-L56

• OpenMLDB cluster settings

  • https://openmldb.ai/docs/zh/main/maintain/faq.html#java-lang-outofmemoryerror-java-heap-space default setting is not enough, set bigger
  • zk config maxClientCnxns default 60, need to set bigger?

• Give a script to convert. And mention that we should generate a new file named 'conf.properties' from 'conf.properties.template'. https://github.com/decis-bench/febench/blob/fc0d88bb6aa96405ca5b35089aa061d0c4961b8e/README.md#L58 e.g. sed '' xx.template > xx

• Comment about the DATASET_ID will be set by compile_test.sh/test.sh. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/conf/conf.properties.template#L4

• default thread is 1? The result will show poor performance. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L56

Test Code

• even we do sync load data, but if failed, won't stop the whole test. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L208
• readParquetFolder is for insert, give the explaination about it, how many file we used in each dataset, and why? e.g. why dataset 0 reads 40 files, but dataset 1,2,3... only read 1 file. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L511-L512
• what's this? https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L542

  error handle

• if create table throws exception, abort the test, don't just print error. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L103
• check returned value, if false, there's no deployment, test can't continue. Make sure deployments exist. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L171

[vagetablechicken]

Update 23/08/07

Doc

• [x] Step 4 怎么修改不清晰，配置项也不同步
• [x] maven http有特殊要求，版本推荐3.8.0，3.8.1级以上可能需要特殊配置，问题在 https://stackoverflow.com/a/67002852

下载使用tar压缩包，分文件下载能下14h+ Q3等dataset必然要使用超过1g默认memory配置，文档写明测试前必须要检查spark配置，而不是错误了再修正配置。local需要改driver memory，yarn需要改execute memory

Usage

• logs目录要提前配置，否则报错，也不应该叫logs，result/report更合适，并且代码或者repo中建好目录 https://github.com/decis-bench/febench/blob/5b2f7c2ea5f66cdf2113748d890f2cc728c582fb/flink/src/src/main/java/org/example/join/BenchmarkConfig.java#L89
• 脚本中途失败应该推出，不要继续跑，加set -ex，不要合并cmd https://github.com/decis-bench/febench/blob/91dac5f16a5c076d136b1299d118e07927775cab/OpenMLDB/compile_test.sh
• flink测试不可以配置flink集群？为什么还会有openmldb配置https://github.com/decis-bench/febench/blob/5facee2f136645e6cf2c0ac4b89bb5b0b2b51ec0/flink/conf/conf.properties.template
• 增加日志系统，不要只stdout，日志不分级非常影响调试
  • init期可能因为无db而删除db失败，打印异常很干扰，适合加入info级日志

    Code Style

• 代码命名不好，java有下划线，全改为驼峰

[zhouxh19]

Setup

• Dir /data just a readme doc, and /dataset is empty. Not good. And the way downloading data wget -r -np -R "index.html*" http://43.138.115.238/download/data/; cp -r <dataset directory> ./dataset is not good too. Use wget -r -np -R "index.html*" http://43.138.115.238/download/data/ -P ./dataset
• Give the openmldb quickstart url, users don't know how to start OpenMLDB cluster. https://github.com/decis-bench/febench/blob/fc0d88bb6aa96405ca5b35089aa061d0c4961b8e/README.md#L55-L56

• OpenMLDB cluster settings

  • https://openmldb.ai/docs/zh/main/maintain/faq.html#java-lang-outofmemoryerror-java-heap-space default setting is not enough, set bigger
  • zk config maxClientCnxns default 60, need to set bigger?

• Give a script to convert. And mention that we should generate a new file named 'conf.properties' from 'conf.properties.template'. https://github.com/decis-bench/febench/blob/fc0d88bb6aa96405ca5b35089aa061d0c4961b8e/README.md#L58

  e.g. sed '' xx.template > xx

• Comment about the DATASET_ID will be set by compile_test.sh/test.sh. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/conf/conf.properties.template#L4
• default thread is 1? The result will show poor performance. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L56

Test Code

• even we do sync load data, but if failed, won't stop the whole test. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L208
• readParquetFolder is for insert, give the explaination about it, how many file we used in each dataset, and why? e.g. why dataset 0 reads 40 files, but dataset 1,2,3... only read 1 file. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L511-L512
• what's this? https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L542

error handle

• if create table throws exception, abort the test, don't just print error. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L103
• check returned value, if false, there's no deployment, test can't continue. Make sure deployments exist. https://github.com/decis-bench/febench/blob/f63a480ea16b850fd49d00167c307a46d6da8025/OpenMLDB/src/src/main/java/com/_4paradigm/benchmark/OpenMLDBPerfBenchmark_Cluster.java#L171

updated :)

[vagetablechicken]

docker needs update after code improved

[vagetablechicken]

https://github.com/decis-bench/febench/blob/523d304ddcfe1eb7aa336e4c4f746ce41ad06b20/README.md?plain=1#L194

not step 5, start from step 4(modify conf file)

sbin style for openmldb cluster deploy

导入需要注意物理内存，测试Q3 local 32会吃多少内存，以此为参考，用户物理内存少的，必须调整local核数