COSMOS remote file load

[kdempsey]

Hi, We have a COSMOS account on cosmos.lab.fi-ware.org and can load files locally onto the cluster. However, we are having trouble loading remotely, the instructions we followed on the guide site show the following: http://forge.fiware.org/plugins/mediawiki/wiki/fiware/index.php/BigData_Analysis_-_Quick_Start_for_Programmers "However, using the WebHDFS/HttpFS RESTful API will allow you to upload files existing outside the global instance of Cosmos in FI-LAB. The following example uses HttpFS instead of WebHDFS (uses the TCP/14000 port instead of TCP/50070), and curl is used as HTTP client (but your applications should implement your own HTTP client): [remote-vm]$ curl -i -X PUT "http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_data?op=MKDIRS&user.name=$COSMOS_USER" [remote-vm]$ curl -i -X PUT "http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_data/unstructured_data.txt?op=CREATE&user.name=$COSMOS_USER" [remote-vm]$ curl -i -X PUT -T unstructured_data.txt --header "content-type: application/octet-stream" http://cosmos.lab.fi-ware.org:14000/webhdfs/v1/user/$COSMOS_USER/input_data/unstructured_data.txt?op=CREATE&user.name=$COSMOS_USER&data=true As you can see, the data uploading is a two-step operation, as stated in the WebHDFS specification: the first invocation of the API talks directly with the Head Node, specifying the new file creation and its name; then the Head Node sends a temporary redirection response, specifying the Data Node among all the existing ones in the cluster where the data has to be stored, which is the endpoint of the second step. Nevertheless, the HttpFS gateway implements the same API but its internal behaviour changes, making the redirection to point to the Head Node itself."

However, when we run these commands we get server errors coming back, one example is: ~ kari$ -bash: user.name=kdempsey: command not found HTTP/1.1 100 Continue

HTTP/1.1 401 Unauthorized Server: Apache-Coyote/1.1 Set-Cookie: hadoop.auth=""; Expires=Thu, 01-Jan-1970 00:00:10 GMT; Path=/ Content-Type: text/html;charset=utf-8 Content-Length: 1275 Date: Fri, 05 Jun 2015 12:58:20 GMT

HTTP Status 401 - org.apache.hadoop.security.authentication.client.AuthenticationException: Anonymous requests are disallowed

---

type Status report

message org.apache.hadoop.security.authentication.client.AuthenticationException: Anonymous requests are disallowed

description This request requires HTTP authentication (org.apache.hadoop.security.authentication.client.AuthenticationException: Anonymous requests are disallowed).

---

Apache Tomcat/6.0.32

Another was a 500 server error. Could please provide the command for remotely loading a file into the COSMOS shared resource.

Many thanks, Kari

[JanWillemKruize]

Hi Kari,

I think this is a question particular for COSMOS. You can best ask this question at: stackoverflow.com (Tags: FIWARE, FIlab, fiware-cosmos)

You can also as francisco Romerobueno. For his e-mail check: http://catalogue.fiware.org/enablers/bigdata-analysis-cosmos

What

[rapw3k]

Hi! Indeed, stackoverflow would be the best thing to do. I forwarded your issue to Francisco on Friday but did not receive any info back yet. Regards, Raul

[kdempsey]

Thanks Raul, Some hints would be great, hopefully there is a simple REST command for loading the data.

[rapw3k]

Hi @kdempsey, I see that someone from Cosmos helped you in stackoverflow to solve this problem, is that right? http://stackoverflow.com/questions/30724933/remote-connection-to-fiware-cosmos-returning-authentication-error If so, can we close this issue?

[kdempsey]

Yes thanks Raul please close.

Best regards, Kari On 6 Jul 2015 09:14, "Raul Palma" notifications@github.com wrote:

Hi @kdempsey https://github.com/kdempsey, I see that someone from Cosmos helped you in stackoverflow to solve this problem, is that right? http://stackoverflow.com/questions/30724933/remote-connection-to-fiware-cosmos-returning-authentication-error If so, can we close this issue?

— Reply to this email directly or view it on GitHub https://github.com/SAF2/documentation/issues/27#issuecomment-118768716.