search

husky-team / husky

A more expressive and most importantly, more efficient system for distributed data analytics.
http://www.husky-project.com/
Other
98 stars 55 forks source link

[IO] Add Redis Connector #289

Open gypleon opened 7 years ago

gypleon commented 7 years ago

Test cases

- Preparation

  1. Flush Redis data using "/data/opt/course/redis/flushall-cluster.sh"
  2. Check configuration file "$HUSKY_ROOT/test/test-redis-config.ini"

- Test 1: CMake

  1. Follow the standard Cmake commands of Husky.
  2. make "Master".
  3. make "TestRedisOutputFormat".
  4. make "TestRedisInputOutputFormat".
  5. make "WordCountRedis".

Expectation: running well. Notes:

- Test 2: enwiki from MongoDB to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m30s. Notes: MUST wait for 10 minutes before Test 3, because for the first time data were loaded into Redis cluster, it takes time for master-slave synchronization.

- Test 3: enwiki from Redis to Redis:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./TestRedisInputOutputFormat -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 1m. Notes: On Husky-worker side, "waiting for keys" logs may be printed as expected, which means that some workers are waiting for keys from the next batch.

- Test 4: enwiki WordCount:

  1. Run "./Master -C test-redis-config.ini"
  2. Run "time /data/opt/tmp/exec.sh ./WordCountRedis -C test-redis-config.ini"

Expectation: running well. 4825150 records to be imported. around 50s. Notes: Can be performed consecutively without restart Husky-Master.

gypleon commented 7 years ago

OMG!!! I merged these commits for several hours... but they come out again... maybe I made a ridiculous mistake that override the merge operations.... OMG.... actually, "[IO] Add Redis Connector. Fixed merge conflicts." (38ce3c2) is supposed to be the ONLY ONE left.

I am currently searching for safely merging solutions.

gypleon commented 7 years ago

Hi Legend, I think the commit problem has been fixed. Please review and leave me comments. Thanks.

kygx-legend commented 7 years ago

This PR is including other commits in master branch which are not merged into experimental branch. Could you redo the rebase?

gypleon commented 7 years ago

No problem.

kygx-legend commented 7 years ago

This branch is updated to master now. Please rebase again!

kygx-legend commented 7 years ago

And you can run ./scripts/clang-format.py -o check to see what does not match the coding style and run ./scripts/clang-format.py -o replace to fix those errors automatically. Then use lint.py to double check.