S3A and S3N path are not correctly handled

AbsaOSS / enceladus

Dynamic Conformance Engine

Apache License 2.0

29 stars 14 forks source link

Open benedeki opened 8 months ago

benedeki commented 8 months ago

While the s3a:// and s3n:// protocols are correctly recognized, they are replaced with hard-coded s3://

See FileSystemUtils class

Keep the protocol.

benedeki commented 7 months ago

Release notes:

when using s3 path, the protocol in the path description is used, instead of hard-coded s3:// prefix, thus supporting other s3 protocols too.
added --jceks-path option to run scripts that will add --conf spark.hadoop.fs.s3a.bucket.SOME_NAME=jceks:/path/to/file.jceks. Use it as --jceks-path spark.hadoop.fs.s3a.bucket.SOME_NAME=jceks:/path/to/file.jceks
added s3a_wrapper.sh script to auto-detect the jceks file and auto-clean-up temporary versions from ECS