search

Hydrospheredata / mist

Serverless proxy for Spark cluster
http://hydrosphere.io/mist/
Apache License 2.0
326 stars 68 forks source link

Mist-lib - incremental compilation fails on different spark versions #530

Closed palaiya closed 5 years ago

palaiya commented 5 years ago

I am getting the following error while building the mist (1.1.1) for spark 2.2.1.

[[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type DataFrameEncoding in package mist.api.encoding.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of mist.api.encoding.spark.
[error] missing or invalid dependency detected while loading class file 'package.class'.
[error] Could not access type DatasetEncoding in package mist.api.encoding.spark,
[error] because it (or its dependencies) are missing. Check your build definition for
[error] missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
[error] A full rebuild may help if 'package.class' was compiled against an incompatible version of mist.api.encoding.spark.
[error] two errors found
[error] (mistLib / Compile / compileIncremental) Compilation failed
dos65 commented 5 years ago

Running sbt clean helps with this. This happens only when mist-lib was compiled against different spark version previously.

dos65 commented 5 years ago

This package object leads to this problem. After its removing mist-lib compiles normally. Probably, we could just replace it with an ordinary object spark but this change will break binary compatibility.