search

starlake-ai / starlake

Declarative text based tool for data analysts and engineers to extract, load, transform and orchestrate their data pipelines.
http://starlake.ai/
Apache License 2.0
57 stars 22 forks source link

[BUG] - NoSuchFileException during load unit tests #888

Closed tiboun closed 6 months ago

tiboun commented 6 months ago

Using unit testing on load without transform generate an exception while launching starlake test --load

java.nio.file.NoSuchFileException: /Users/ME/starlake-unit-testing/test-reports/transform/index.html
    at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:92)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111)
    at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116)
    at java.base/sun.nio.fs.UnixFileSystemProvider.newByteChannel(UnixFileSystemProvider.java:219)
    at java.base/java.nio.file.spi.FileSystemProvider.newOutputStream(FileSystemProvider.java:478)
    at java.base/java.nio.file.Files.newOutputStream(Files.java:220)
    at java.base/java.nio.file.Files.write(Files.java:3425)
    at ai.starlake.tests.StarlakeTestResult$.html(StarlakeTestResult.scala:119)
    at ai.starlake.tests.StarlakeTestResult$.html(StarlakeTestResult.scala:103)
    at ai.starlake.workflow.IngestionWorkflow.test(IngestionWorkflow.scala:865)
    at ai.starlake.job.transform.TransformTestCmd.run(StarlakeTestCmd.scala:60)
    at ai.starlake.job.transform.TransformTestCmd.run$(StarlakeTestCmd.scala:57)
    at ai.starlake.job.transform.TransformTestCmd$.run(StarlakeTestCmd.scala:65)
    at ai.starlake.job.transform.TransformTestCmd$.run(StarlakeTestCmd.scala:65)
    at ai.starlake.job.Cmd.run(Cmd.scala:19)
    at ai.starlake.job.Cmd.run$(Cmd.scala:14)
    at ai.starlake.job.transform.TransformTestCmd$.run(StarlakeTestCmd.scala:65)
    at ai.starlake.job.Main.run(Main.scala:250)
    at ai.starlake.job.Main.$anonfun$run$1(Main.scala:175)
    at scala.runtime.java8.JFunction0$mcZ$sp.apply(JFunction0$mcZ$sp.java:23)
    at scala.util.DynamicVariable.withValue(DynamicVariable.scala:62)
    at scala.Console$.withErr(Console.scala:196)
    at ai.starlake.job.Main.run(Main.scala:172)
    at ai.starlake.job.Main$.main(Main.scala:62)
    at ai.starlake.job.Main.main(Main.scala)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.base/java.lang.reflect.Method.invoke(Method.java:566)
    at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:1029)
    at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:194)
    at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:217)
    at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:91)
    at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1120)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1129)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

I don't expect to have this exception while lauching test on load or when I restrict test to load.

hayssams commented 6 months ago

Solved by creating the target folder first.