search

snowch / biginsight-examples

Example projects to help you quickly get started with BigInsights
Apache License 2.0
7 stars 4 forks source link

Softlayer Objectstore issue : Unable to execute the HTTP call or to convert the HTTP Response #71

Closed snowch closed 8 years ago

snowch commented 8 years ago

For softlayer objectstore I'm using a url like this:

swift2d://lon02.objectstorage.softlayer.net/counts

However, the error I get is:

bicluster#7|py4j.protocol.Py4JJavaError: An error occurred while calling o86.saveAsTextFile.
bicluster#7|: org.javaswift.joss.exception.CommandException: Unable to execute the HTTP call or to convert the HTTP Response
bicluster#7|    at org.javaswift.joss.command.impl.core.AbstractCommand.call(AbstractCommand.java:58)
bicluster#7|    at org.javaswift.joss.command.impl.core.AbstractSecureCommand.call(AbstractSecureCommand.java:31)
bicluster#7|    at org.javaswift.joss.client.core.AbstractContainer.getInfo(AbstractContainer.java:235)
bicluster#7|    at org.javaswift.joss.client.core.AbstractObjectStoreEntity.checkForInfo(AbstractObjectStoreEntity.java:98)
bicluster#7|    at org.javaswift.joss.client.core.AbstractObjectStoreEntity.checkForInfoDisallowErrorLog(AbstractObjectStoreEntity.java:93)
bicluster#7|    at org.javaswift.joss.client.core.AbstractObjectStoreEntity.exists(AbstractObjectStoreEntity.java:110)
bicluster#7|    at com.ibm.stocator.fs.swift.SwiftAPIClient.<init>(SwiftAPIClient.java:209)
bicluster#7|    at com.ibm.stocator.fs.ObjectStoreVisitor.getStoreClient(ObjectStoreVisitor.java:37)
bicluster#7|    at com.ibm.stocator.fs.ObjectStoreFileSystem.initialize(ObjectStoreFileSystem.java:87)
bicluster#7|    at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2653)
bicluster#7|    at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:92)
bicluster#7|    at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2687)
bicluster#7|    at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2669)
bicluster#7|    at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:371)
bicluster#7|    at org.apache.hadoop.fs.Path.getFileSystem(Path.java:295)
bicluster#7|    at org.apache.spark.SparkHadoopWriter$.createPathFromString(SparkHadoopWriter.scala:170)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply$mcV$sp(PairRDDFunctions.scala:988)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:965)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$4.apply(PairRDDFunctions.scala:965)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
bicluster#7|    at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:965)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply$mcV$sp(PairRDDFunctions.scala:897)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:897)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsHadoopFile$1.apply(PairRDDFunctions.scala:897)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
bicluster#7|    at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
bicluster#7|    at org.apache.spark.rdd.PairRDDFunctions.saveAsHadoopFile(PairRDDFunctions.scala:896)
bicluster#7|    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply$mcV$sp(RDD.scala:1426)
bicluster#7|    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1405)
bicluster#7|    at org.apache.spark.rdd.RDD$$anonfun$saveAsTextFile$1.apply(RDD.scala:1405)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:147)
bicluster#7|    at org.apache.spark.rdd.RDDOperationScope$.withScope(RDDOperationScope.scala:108)
bicluster#7|    at org.apache.spark.rdd.RDD.withScope(RDD.scala:306)
bicluster#7|    at org.apache.spark.rdd.RDD.saveAsTextFile(RDD.scala:1405)
bicluster#7|    at org.apache.spark.api.java.JavaRDDLike$class.saveAsTextFile(JavaRDDLike.scala:522)
bicluster#7|    at org.apache.spark.api.java.AbstractJavaRDDLike.saveAsTextFile(JavaRDDLike.scala:47)
bicluster#7|    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
bicluster#7|    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
bicluster#7|    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
bicluster#7|    at java.lang.reflect.Method.invoke(Method.java:497)
bicluster#7|    at py4j.reflection.MethodInvoker.invoke(MethodInvoker.java:231)
bicluster#7|    at py4j.reflection.ReflectionEngine.invoke(ReflectionEngine.java:379)
bicluster#7|    at py4j.Gateway.invoke(Gateway.java:259)
bicluster#7|    at py4j.commands.AbstractCommand.invokeMethod(AbstractCommand.java:133)
bicluster#7|    at py4j.commands.CallCommand.execute(CallCommand.java:79)
bicluster#7|    at py4j.GatewayConnection.run(GatewayConnection.java:207)
bicluster#7|    at java.lang.Thread.run(Thread.java:745)
bicluster#7|Caused by: org.apache.http.client.ClientProtocolException: URI does not specify a valid host name: https:/%2Flon02.objectstorage.softlayer.net%2Fcounts
bicluster#7|    at org.apache.http.impl.client.CloseableHttpClient.determineTarget(CloseableHttpClient.java:94)
bicluster#7|    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
bicluster#7|    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:106)
bicluster#7|    at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:57)
bicluster#7|    at org.javaswift.joss.command.impl.core.AbstractCommand.call(AbstractCommand.java:49)
bicluster#7|    ... 49 more
snowch commented 8 years ago

Changing to:

    # destination url
    if (os_auth_method == 'swiftauth'):
        # E.g. SoftLayer
        url = os_auth_url.replace('https://', '')
        url = re.sub('\/$', '', url) # remove trailing forward slash if exists

        swift_file_url = "swift2d://{0}/counts".format(url)
    else:
        swift_file_url = "swift2d://{0}.{1}/counts".format(os_container, os_region)

   counts.saveAsTextFile(swift_file_url)

Results in HTTP Status code: 400 => UNKNOWN

Ref: https://github.com/SparkTC/stocator/issues/43