make-distribution.sh with --skip-java-test fails against master

[MLnick]

Attempting to run spark-perf against master fails:

./make-distribution.sh --skip-java-test
The following shell command finished with a non-zero returncode (1): ./make-distribution.sh --skip-java-test

The issue is with the --skip-java-test flag, which was recently removed in https://github.com/apache/spark/commit/6cf51a7007bd72eb93ade149ca9fc53be5b32a17. It's still used in Spark 1.4- (https://github.com/apache/spark/blob/branch-1.4/make-distribution.sh#L146)

There are 2 potential fixes:

• Amend spark-perf to check if --skip-java-test exists in make-distribution. Something like:

...
    with cd(target_dir):
        logger.info("Building spark at version %s; This may take a while...\n" % commit_id)
        # Spark version 1.5+ no longer uses the --skip-java-test flag in make-distribution.sh
        skip_java_test_code = run_cmd('cat make-distribution.sh | grep "skip-java-test"', exit_on_fail=False)
        skip_java_test_str =  "--skip-java-test" if skip_java_test_code == 0 else ""
        # According to the SPARK-1520 JIRA, building with Java 7+ will only cause problems when
        # running PySpark on YARN or when running on Java 6.  Since we'll be building and running
        # Spark on the same machines and using standalone mode, it should be safe to
        # disable this warning:
        if is_yarn_mode:
            run_cmd("./make-distribution.sh %s -Pyarn %s" % (skip_java_test_str, additional_make_distribution_args))
        else:
            run_cmd("./make-distribution.sh %s %s" % (skip_java_test_str, additional_make_distribution_args))
...

• Amend make-distribution.sh to give a warning such as suggested in the commit. e.g.

echo "Warning: '--skip-java-test' is deprecated and has no effect."
;;

I can submit a PR for option 1 or 2 as required, let me know which you prefer.