ClickHouse / spark-clickhouse-connector

Spark ClickHouse Connector build on DataSourceV2 API
https://clickhouse.com/docs/en/integrations/apache-spark
Apache License 2.0
187 stars 66 forks source link

Test: Tune tests to support MS Windows #341

Closed pan3793 closed 3 months ago

pan3793 commented 4 months ago

Summary

This PR aims to tune the code to make the test runnable on Microsoft Windows.

Unfortunately, my MacBook got damaged and I have to use a Windows machine these days, when I try to run the test with .\gradlew.bat clean test, I get the following errors

xenon.clickhouse.HashSuite *** ABORTED *** (0 milliseconds)
  java.lang.RuntimeException: Unable to load a Suite class xenon.clickhouse.HashSuite that was discovered in the runpath: requirement failed: illegal path: D:\Projects\spark-clickhouse-connector\clickhouse-core-it\build\classes\scala\test      
  at org.scalatest.tools.DiscoverySuite$.getSuiteInstance(DiscoverySuite.scala:80)
  at org.scalatest.tools.DiscoverySuite.$anonfun$nestedSuites$1(DiscoverySuite.scala:38)
  at scala.collection.TraversableLike.$anonfun$map$1(TraversableLike.scala:286)
  at scala.collection.Iterator.foreach(Iterator.scala:943)
  at scala.collection.Iterator.foreach$(Iterator.scala:943)
  at scala.collection.AbstractIterator.foreach(Iterator.scala:1431)
  at scala.collection.IterableLike.foreach(IterableLike.scala:74)
  at scala.collection.IterableLike.foreach$(IterableLike.scala:73)
  at scala.collection.AbstractIterable.foreach(Iterable.scala:56)
  at scala.collection.TraversableLike.map(TraversableLike.scala:286)

this is because Windows uses backslash as the path separator while UNIX-like OS uses the slash, it can be easily fixed by replacing path string match with Java NIO methods.

for developers who want to run tests on Windows, you need to:

  1. install JDK 8 or 11 or 17 - download and unarchive a JDK tarball, set env JAVA_HOME and add %JAVA_HOME%\bin to PATH
  2. install Hadoop winutils - just clone the repo and set HADOOP_HOME to the-path-of\hadoop-3.0.0
  3. install Docker Desktop

after applying patch, I can run tests successfully

 .\gradlew.bat clean test

...

Tests: succeeded 74, failed 0, canceled 0, ignored 0, pending 0
All tests passed.

BUILD SUCCESSFUL in 4m 18s
26 actionable tasks: 26 executed

Checklist

Delete items not relevant to your PR:

pan3793 commented 4 months ago

cc @wForget, can you do me a favor to test it on Windows if you have time? thanks

wForget commented 4 months ago

cc @wForget, can you do me a favor to test it on Windows if you have time? thanks

Sure, , I will do it later.