reading csv from s3 throwing this error using (sample-spark-script-csv-to-csv.py) file

aws-samples / spark-on-aws-lambda

Spark runtime on AWS Lambda

MIT No Attribution

92 stars 36 forks source link

reading csv from s3 throwing this error using (sample-spark-script-csv-to-csv.py) file #53

Open saiteja2420 opened 5 months ago

saiteja2420 commented 5 months ago

df=spark.read.option('header','true').csv(input_path) java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.hadoop.fs.s3a.S3AFileSystem not found please help how to resolve this issue .

ashishgupta-in commented 1 month ago

@saiteja2420 What is the input_path you are providing? Is it directly a S3 URL/URI?

JohnChe88 commented 1 month ago

@saiteja2420 we are using S3a://

JohnChe88 commented 1 month ago

@saiteja2420 if it csv to csv , then hadoop-aws.jar file version might be the problem, have you changed any version in the Dockerfile?