search

ibm-watson-data-lab / ibmos2spark

Facilitates Data I/O between Spark and IBM Object Storage services.
10 stars 8 forks source link

CloudObjectStorage failed to connect to private COS #48

Open gilv opened 6 years ago

gilv commented 6 years ago

I tried to use CloudObjectStorage and connect to my private object storage, but nothing worked. For example, the following fails: var credentialsGilCOS = scala.collection.mutable.HashMapString, String

var configurationNameGil = "service"
var cosGil = new CloudObjectStorage(sc, credentialsGilCOS, configurationNameGil, "softlayer_cos")
val url = cosGil.url("gvernik", "test1812/a2/data.parquet")
peopleDF.write.format("parquet").save(url)

The following works: sc.hadoopConfiguration.set("fs.cos.gilv.access.key","AAA") sc.hadoopConfiguration.set("fs.cos.gilv.secret.key","AHA") sc.hadoopConfiguration.set("fs.cos.gilv.endpoint","https://s3-api.us-geo.objectstorage.softlayer.net") sc.hadoopConfiguration.set("fs.cos.gilv.v2.signer.type","false")

peopleDF.write.format("parquet").save("cos://gvernik.gilv/test1812/a2/data.parquet")
bassel-zeidan commented 6 years ago

Hi @gilv , Strange! The class CloudObjectStorage has been used by a lot of users and it works fine. Here is an example:

https://apsx-dev.stage1.ng.bluemix.net/analytics/notebooks/4709da6c-371e-4b0c-b58e-86b7e5b33aa9/view?access_token=9daf7b648595142e634a4344d2b2b9619dbd83d51bdcf85e9e080efb23354e37

Maybe you are passing wrong configs. Currently i am on vacation but i would like to take a closer look when i am back. Could you please invite me to your project? or give me access to your notebook?