make --driver-library-path mandatory for python projects

SimpleDataLabsInc / prophecy-build-tool

Prophecy-built-tool (PBT) allows you to quickly build projects generated by Prophecy (your standard Spark Scala and PySpark pipelines) to integrate them with your own CI / CD (e.g. Github Actions), build system (e.g. Jenkins), and orchestration (e.g. Databricks Workflows).

https://pypi.org/project/prophecy-build-tool

Apache License 2.0

28 stars 3 forks source link

make --driver-library-path mandatory for python projects #103

Open neontty opened 3 months ago

neontty commented 3 months ago

make --driver-library-path mandatory for python projects

neontty commented 3 months ago

only for test-v2 command. pending discussion whether we should just put a better error if correct lib version is not detected.

neontty commented 3 months ago

enhance pbt so it takes the version of plibs from pbt_project.yml and attempts to use the correct version if it's in the M2 cache otherwise error out.

neontty commented 3 months ago

--driver-library-path should take any of the following:

full path to exact jar which matches version in pbt_project.yml
path to directory that contains a jar which matches version in pbt_project.yml
path which ends in .m2
empty (search for libs in: ['$M2_REPO', '$HOME/.m2/repository/'])

if correct plibs version is not found in any of those places, error out. accept the latest $SPARK_VERSION out of any libs that exist.