search

CODAIT / stocator

Stocator is high performing connector to object storage for Apache Spark, achieving performance by leveraging object storage semantics.
Apache License 2.0
113 stars 72 forks source link

Stocator fails with Spark Streaming #271

Open VidyasagarMSC opened 3 years ago

VidyasagarMSC commented 3 years ago

I tried reading and writing objects to COS using Stocator with Spark streaming but it fails with the below error 

java.lang.RuntimeException: java.lang.ClassNotFoundException: Class com.ibm.stocator.fs.ObjectStoreFileSystem not found

Here's what I tried to overcome the error and failed

  1. Built Stocator jar file and moved the jar to /opt/spark/jars with COS access key and secret key. Fails with jackson.databinding not found exception
  2. Built SDK jar file and used the IAM api key Fails with jackson.databinding not found exception

This is what worked as an alternative to Stocator I have to fall back to S3A by moving the required jar files to /opt/spark/jars folder and use s3a:// instead of cos://

mrmadira commented 3 years ago

Have you tried using the ready jar from https://mvnrepository.com/artifact/com.ibm.stocator/stocator ?