search

spark-redshift-community / spark-redshift

Performant Redshift data source for Apache Spark
Apache License 2.0
137 stars 63 forks source link

fixes #103: Wrong FS error while import data in RedShift #104

Closed conker84 closed 2 years ago

conker84 commented 2 years ago

This fixes #103

conker84 commented 2 years ago

@jsleight @lucagiovagnoli @88manpreet hi everybody, can someone review this?

conker84 commented 2 years ago

@jsleight @88manpreet ping

conker84 commented 2 years ago

@jsleight @88manpreet I updated the PR

aatallah2 commented 2 years ago

We have the exact same issue, and made a similar fix locally. I'm not a reviewer, but I can attest that this is an issue with the latest AWS library dependencies.

The problem is that the manifest path must be in s3a, s3n,... format, whereas the path inside the manifest must be in s3:// format.

conker84 commented 2 years ago

@jsleight please lemme know if you want to fix this

jsleight commented 2 years ago

@jsleight please lemme know if you want to fix this

Yeah fixing this issue definitely seems correct. I think I left a comment on the one main discussion thread for a slight tweak before we merge it.

jsleight commented 2 years ago

I just merged #110 which is a duplicate of this issue.