Getting error- "Not a valid command: assembly" while running build.sh

[amit78del]

Hi,

While running build.sh, I am getting error - "not a valid command: assembly". I followed instructions from below page to run locally. https://github.com/scylladb/scylla-migrator

Installed sbt by using SDKMAN as mentioned in below doc. https://www.scala-sbt.org/release/docs/Installing-sbt-on-Linux.html

Java version- openjdk version "1.8.0_292" OpenJDK Runtime Environment (build 1.8.0_292-8u292-b10-0ubuntu1~18.04-b10) OpenJDK 64-Bit Server VM (build 25.292-b10, mixed mode)

Error logs-

• export TERM=xterm-color
• TERM=xterm-color
• git submodule update --init --recursive
• TMPDIR=/home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec
• mkdir -p /home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec
• trap 'rm -rf /home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec' EXIT
• pushd spark-cassandra-connector ~/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector ~/Desktop/scylla-migration/scylla-migrator
• sbt -Djava.io.tmpdir=/home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec ++2.11.12 assembly [info] Loading settings for project spark-cassandra-connector-build from plugins.sbt,updates.sbt ... [info] Loading project definition from /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/project [info] Loading settings for project root from build.sbt ... Using Some(releases: https://oss.sonatype.org/service/local/staging/deploy/maven2) for publishing Reading credentials from None Running 10 Parallel Tasks [info] Set current project to root (in build file:/home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/) [info] Setting Scala version to 2.11.12 on 3 projects. [info] Excluded 1 projects, run ++ 2.11.12 -v for more details. [info] Reapplying settings... [info] Set current project to root (in build file:/home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/) [warn] Credentials file /home/delhivery/.ivy2/.credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials.deploy does not exist [warn] Credentials file /home/delhivery/.ivy2/.credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials.deploy does not exist [warn] Credentials file /home/delhivery/.ivy2/.credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials.deploy does not exist [warn] Credentials file /home/delhivery/.ivy2/.credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials does not exist [warn] Credentials file /home/delhivery/.sbt/credentials.deploy does not exist [info] Strategy 'discard' was applied to a file (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/target/scala-2.12/root-assembly-2.3.1-637-g35e0c096.jar [info] Strategy 'discard' was applied to 2 files (Run the task at debug level to see details) [info] Strategy 'last' was applied to 7 files (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/test-support/target/scala-2.11/spark-cassandra-connector-test-support-assembly-2.3.1-637-g35e0c096.jar [info] Strategy 'discard' was applied to 2 files (Run the task at debug level to see details) [info] Strategy 'last' was applied to 7 files (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/driver/target/scala-2.11/spark-cassandra-connector-driver-assembly-2.3.1-637-g35e0c096.jar [info] Strategy 'discard' was applied to 2 files (Run the task at debug level to see details) [info] Strategy 'last' was applied to 7 files (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-cassandra-connector/connector/target/scala-2.11/spark-cassandra-connector-assembly-2.3.1-637-g35e0c096.jar [success] Total time: 3 s, completed 22 Nov, 2021 10:24:11 AM
• popd ~/Desktop/scylla-migration/scylla-migrator
• pushd spark-dynamodb ~/Desktop/scylla-migration/scylla-migrator/spark-dynamodb ~/Desktop/scylla-migration/scylla-migrator
• sbt assembly [info] Loading settings for project spark-dynamodb-build from plugins.sbt ... [info] Loading project definition from /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-dynamodb/project [info] Loading settings for project spark-dynamodb from build.sbt ... [info] Set current project to spark-dynamodb (in build file:/home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-dynamodb/) [info] Strategy 'discard' was applied to a file (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-dynamodb/target/scala-2.11/spark-dynamodb-assembly-1.0.5.jar [success] Total time: 1 s, completed 22 Nov, 2021 10:24:17 AM
• popd ~/Desktop/scylla-migration/scylla-migrator
• pushd spark-kinesis ~/Desktop/scylla-migration/scylla-migrator/spark-kinesis ~/Desktop/scylla-migration/scylla-migrator
• sbt assembly [info] welcome to sbt 1.3.13 (Private Build Java 1.8.0_292) [info] loading settings for project spark-kinesis-build from plugins.sbt ... [info] loading project definition from /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-kinesis/project [info] loading settings for project root from build.sbt ... [info] set current project to spark-streaming-kinesis-asl (in build file:/home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-kinesis/) [warn] Multiple main classes detected. Run 'show discoveredMainClasses' to see the list [info] Strategy 'discard' was applied to a file (Run the task at debug level to see details) [info] Assembly up to date: /home/delhivery/Desktop/scylla-migration/scylla-migrator/spark-kinesis/target/scala-2.11/spark-streaming-kinesis-asl-assembly-2.4.4.jar [success] Total time: 1 s, completed 22 Nov, 2021 10:24:23 AM
• popd ~/Desktop/scylla-migration/scylla-migrator
• '[' '!' -d ./lib ']'
• cp ./spark-cassandra-connector/connector/target/scala-2.11/spark-cassandra-connector-assembly-2.3.1-637-g35e0c096.jar ./lib
• cp ./spark-dynamodb/target/scala-2.11/spark-dynamodb-assembly-1.0.5.jar ./lib
• cp ./spark-kinesis/target/scala-2.11/spark-streaming-kinesis-asl-assembly-2.4.4.jar ./lib
• sbt -Djava.io.tmpdir=/home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec assembly [info] welcome to sbt 1.5.5 (Private Build Java 1.8.0_292) [info] loading project definition from /tmp/hsperfdata_delhivery/project [info] set current project to hsperfdata_delhivery (in build file:/tmp/hsperfdata_delhivery/) [error] Not a valid command: assembly [error] Not a valid project ID: assembly [error] Expected ':' [error] Not a valid key: assembly [error] assembly [error] ^
• rm -rf /home/delhivery/Desktop/scylla-migration/scylla-migrator/tmpexec

I tried all the solutions mentioned in stack overflow, some links given below. https://stackoverflow.com/questions/36186214/sbt-assembly-not-working-not-a-valid-command https://stackoverflow.com/questions/44106182/why-does-sbt-assembly-fail-with-not-a-valid-command-assembly https://stackoverflow.com/questions/29523694/sbt-assembly-command-not-found/29526644#29526644

My folder structure-

root folder has build.sbt and build.sh files. root/project folder has- build.sbt and build.properties files.

I didn't tune any parameter in these files and tried running after cloning project from below repo. https://github.com/scylladb/scylla-migrator

Any help is appreciated.

[amit78del]

@iravid, @tarzanek adding you guys here for faster inputs.

[tarzanek]

@amit78del this is definitely an env issue so we can go and debug your env (try to get env output to see what is wrong) OR ideal case is to spawn a new clean stock VM and try to build I am building this just fine on clean Ubuntu 21.10 or on Fedora 35 - java is openjdk packages, exported with JAVA_HOME sbt is installed from repo (so I don't use SDKMAN) I have a clean git clone of scylla-migrator and then inside it (with java exported in JAVA_HOME or set default using alternatives) I run ./build.sh

another option is to create a completely new user on your existing machine and try with user that has stock env (assuming your default env is not tainted) choose whatever is fastest for you (debug env, new user, new clean stock VM)

[amit78del]

Thanks @tarzanek after cleaning the env, the issue was resolved.