Pyspark-iris Windows CI fix - Githubissues

kedro-org / kedro-starters

Templates for your Kedro projects.

Apache License 2.0

64 stars 59 forks source link

Pyspark-iris Windows CI fix #100

Closed SajidAlamQB closed 2 years ago

SajidAlamQB commented 2 years ago

Signed-off-by: SajidAlamQB 90610031+SajidAlamQB@users.noreply.github.com

Motivation and Context

Related issue: https://github.com/kedro-org/kedro-starters/issues/95

The windows run tests do not pass due to a problem with pyspark-iris. This PR will look to fix this issue.

Notes:

PySpark needs hadoop binaries installed.
Using Hadoop 2.7.1 binaries as 2.6.3 (Used on the framework) resulted in errors.
JDK comes pre-installed with this windows orb therefore does not need to be added.
hadoop.dll needs to be placed in C:\windows\system32.
HADOOP_HOME environment variable needs to be set to 'C:\hadoop'.
winutils.exe needs to be added to 'C:\hadoop'.

How has this been tested?

Checklist

[ ] Opened this PR as a 'Draft Pull Request' if it is work-in-progress
[x] Assigned myself to the PR
[ ] Added tests to cover my changes