search

databrickslabs / remorph

Cross-compiler and Data Reconciler into Databricks Lakehouse
Other
29 stars 18 forks source link

Updated Spark setup script #875

Closed vijaypavann-db closed 1 week ago

vijaypavann-db commented 2 weeks ago

  1. Updated Spark setup script to check whether spark gzip file exists

  2. Refactored the script to remove the warnings:

    • Double quote to prevent globbing and word splitting
    • Use 'cd ... || exit' or 'cd ... || return' in case cd fails.
    • Consider using 'grep -c' instead of 'grep|wc -l'.
    • Usage: sleep seconds, converted 2m to 120 seconds

  3. Tested the following Scenarios:

    1. Scenario: Extracted Spark folder is already present Outcome: Directly starts the spark server using sbin/start-connect-server.sh
    2. Scenario: Extracted Spark folder is not present, and Zip file (spark-<VERSION>.tgz) is present Outcome: Extract the zip file and start the spark server
    3. Scenario: Extracted Spark folder is not present, and Zip file is not present Outcome: Download, Extract and start the spark server
github-actions[bot] commented 2 weeks ago

Coverage tests results

430 tests  ±0   382 :white_check_mark: ±0   5s :stopwatch: ±0s   5 suites ±0     0 :zzz: ±0    5 files   ±0    48 :x: ±0 

For more details on these failures, see this check.

Results for commit be2c45c6. ± Comparison against base commit 6794006d.

:recycle: This comment has been updated with latest results.