search

deanhiller / databus

time series data in cassandra with visualization(NREL's opensource databus project)
Other
53 stars 21 forks source link

"Oops, an error occured" server error 500 when registering table data #29

Open kbrems opened 10 years ago

kbrems commented 10 years ago

I am trying to register my table columns via java code, but I always get an internal server error 500 result with error message "Oops, an error occured". I am logging the json string I am sending to the registerV1 api and if I copy that string from the log and paste it into a file and send using curl, it works fine. If I then try again with my code, I get the error that the table exists, so I know that the register module is getting to the point of parsing my json. I am sending the json as a stream entity. This code works fine when I send data using the postdataV1 api.

Code looks like so: logger.info("post data to: " + registerUrl); HttpPost httpPost = new HttpPost(registerUrl); httpPost.setHeader("Content-Type", "application/json"); DefaultHttpClient httpClient = new DefaultHttpClient(); httpClient.getCredentialsProvider().setCredentials( new AuthScope(mServer, 9000), new UsernamePasswordCredentials("robot-robotTest2", "45580RCV5S.ALPHA1.1PT80JZFNKHUG"));

String androidID = Settings.Secure.getString(mContext.getContentResolver(), Settings.Secure.ANDROID_ID); String tableJson = "{\"datasetType\":\"RELATIONALTABLE\", \"modelName\":\"ANDROID4" + androidID + "\", \"schema\":\"sensorTest\", " + tableCols + "}"; try { httpPost.setEntity(new StringEntity(tableJson)); } catch (UnsupportedEncodingException e1) { logger.severe("encoding error in json string"); e1.printStackTrace(); return; }

try { // do the post logger.info("sending table data: " + tableJson); HttpResponse response = httpClient.execute(httpPost); StatusLine status = response.getStatusLine(); logger.info("http post response: " + status.getStatusCode() + " - " + status.getReasonPhrase());

log looks like this: 10-17 11:11:38.638: I/TableProvisioner(3089): post data to: http://15.185.92.96:9000/api/registerV1 10-17 11:11:38.658: I/TableProvisioner(3089): sending table data: {"datasetType":"RELATIONAL_TABLE", "modelName":"ANDROID4_910e2872a4e96f5e", "schema":"sensorTest", "columns":[{"name":"time","dataType":"BigInteger","isIndex":true,"isPrimaryKey":true},{"name":"ROTX","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTY","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTZ","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false}]} 10-17 11:11:38.848: I/TableProvisioner(3089): http post response: 500 - Internal Server Error 10-17 11:11:38.848: E/TableProvisioner(3089): http post error: 500 - <!DOCTYPE html> 10-17 11:11:38.848: E/TableProvisioner(3089): 10-17 11:11:38.848: E/TableProvisioner(3089): 10-17 11:11:38.848: E/TableProvisioner(3089): Application error 10-17 11:11:38.848: E/TableProvisioner(3089):
10-17 11:11:38.848: E/TableProvisioner(3089): 10-17 11:11:38.848: E/TableProvisioner(3089): 10-17 11:11:38.848: E/TableProvisioner(3089):

Oops, an error occured

10-17 11:11:38.848: E/TableProvisioner(3089):

10-17 11:11:38.848: E/TableProvisioner(3089): This exception has been logged with id 6fo4f81f7. 10-17 11:11:38.848: E/TableProvisioner(3089):

10-17 11:11:38.848: E/TableProvisioner(3089): 10-17 11:11:38.848: E/TableProvisioner(3089):

Copy/paste the table data from the log into a file and sending vial curl looks like this: kbrems@ubuntu:~/sensorTest$ curl -X POST -w "\nRESULT CODE:%{http_code}\n" --header "Content-Type:application/json" --user robot-robotTest2:45580RCV5S.ALPHA1.1PT80JZFNKHUG -d @dbTable4.txt http://15.185.92.96:9000/api/registerV1 {"modelName":"ANDROID4_910e2872a4e96f5e","groups":[]} RESULT CODE:200

kbrems@ubuntu:~/sensorTest$ cat dbTable4.txt {"datasetType":"RELATIONAL_TABLE", "modelName":"ANDROID4_910e2872a4e96f5e", "schema":"sensorTest", "columns":[{"name":"time","dataType":"BigInteger","isIndex":true,"isPrimaryKey":true},{"name":"ROTX","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTY","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTZ","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false}]}

kbrems commented 10 years ago

Looking at the server log I see this. I am setting username/key credentials exactly the same as when I use the postDataV1 url and that works fine. I am sending a column with isPrimaryKey set to true. Not sure what "username" is a primary key for.

Execution exception (In /app/models/EntityUser.java around line 201) UnsupportedOperationException occured : Primary key username cannot be null

play.exceptions.JavaExecutionException: Primary key username cannot be null at play.mvc.ActionInvoker.invoke(ActionInvoker.java:267) ~[play-1.2.x-1.1.0-2208.jar:na] at Invocation.HTTP Request(Play!) ~[na:na] Caused by: java.lang.UnsupportedOperationException: Primary key username cannot be null at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.processPrimaryKey(SpiIndexQueryImpl.java:262) ~[playorm.jar:na] at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.processRangeExpression(SpiIndexQueryImpl.java:211) ~[playorm.jar:na] at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.processExpressionTree(SpiIndexQueryImpl.java:139) ~[playorm.jar:na] at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultListImpl(SpiIndexQueryImpl.java:106) ~[playorm.jar:na] at com.alvazan.orm.layer5.query.SpiIndexQueryImpl.getResultList(SpiIndexQueryImpl.java:75) ~[playorm.jar:na] at com.alvazan.orm.layer0.base.QueryAdapter.getResults(QueryAdapter.java:126) ~[playorm.jar:na] at com.alvazan.orm.layer0.base.QueryAdapter.getResults(QueryAdapter.java:121) ~[playorm.jar:na] at com.alvazan.orm.layer0.base.QueryAdapter.getResults(QueryAdapter.java:111) ~[playorm.jar:na] at com.alvazan.orm.layer0.base.QueryAdapter.getSingleObject(QueryAdapter.java:100) ~[playorm.jar:na] at models.EntityUser.findByName(EntityUser.java:201) ~[webapp:na] at controllers.SecurityUtil.fetchUser(SecurityUtil.java:247) ~[webapp:na] at controllers.api.ApiRegistrationImpl.registerImpl(ApiRegistrationImpl.java:125) ~[webapp:na] at controllers.ApiRegistration.postRegisterTableImpl(ApiRegistration.java:136) ~[webapp:na] at controllers.ApiRegistration.postRegisterTableNew(ApiRegistration.java:79) ~[webapp:na] at play.mvc.ActionInvoker.invokeWithContinuation(ActionInvoker.java:587) ~[play-1.2.x-1.1.0-2208.jar:na] at play.mvc.ActionInvoker.invoke(ActionInvoker.java:538) ~[play-1.2.x-1.1.0-2208.jar:na] at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:514) ~[play-1.2.x-1.1.0-2208.jar:na] at play.mvc.ActionInvoker.invokeControllerMethod(ActionInvoker.java:509) ~[play-1.2.x-1.1.0-2208.jar:na] at play.mvc.ActionInvoker.invoke(ActionInvoker.java:187) ~[play-1.2.x-1.1.0-2208.jar:na] ... 1 common frames omitted

deanhiller commented 10 years ago

ok, a few things here...

  1. our TIME_SERIES table are much faster than RELATIONAL(just an FYI)....much much faster ;)
  2. it appears the basic auth user name is not getting through....I typically do a post to this url so the basic auth goes through with curl http://robot-robotTest2:45580RCV5S.ALPHA1.1PT80JZFNKHUG@15.185.92.96:9000/api/registerV1

Notice in 2 that the robot name and password are part of the url...ie. http://{username}:{password}@{host}

  1. I added a precondition higher in the code(so you can update your stuff from github and re-run it). It should now say "no username found in basic auth (or possibly the session if logged in via GUI)" in the logs and I think that is supposed to be sent back in the body of the post response as well with http code 500(I will check shortly).
  2. Which organization are you with? (we like to know our users and while many don't file tickets, we typically are talking to them offline to understand needs out in the industry).

thanks, Dean

kbrems commented 10 years ago

Hello Dean,

I tried putting the credentials in the url, but I still get the same 500 Oops error. I am using this url:

10-21 09:56:48.752: I/TableProvisioner(3986): post data to: http://robot-robotTest2:45580RCV5S.ALPHA1.1PT80JZFNKHUG@15.185.92.96:9000/api/registerV1

I have not yet installed your new code from github. I need to talk to the person who manages our databus installation about that.

I had tried using the credentials in the url for the postdata api before, but then then I always get the 401 – unauthorized error. This is the url I used:

10-21 09:45:13.250: I/DataTransmitter(2903): post data to: http://robot-robotTest2:45580RCV5S.ALPHA1.1PT80JZFNKHUG@15.185.92.96:9000/api/postdataV1

I work for HP along with Matt Hopcroft who has been in contact with you before. I am attempting to load sensor data from an Android device into databus. I am using the java libraries that come installed with the android SDK for DefaultHttpClient etc.

Karen

From: deanhiller [mailto:notifications@github.com] Sent: Monday, October 21, 2013 5:46 AM To: deanhiller/databus Cc: Brems, Karen (HP Labs) Subject: Re: [databus] "Oops, an error occured" server error 500 when registering table data (#29)

ok, a few things here...

  1. our TIME_SERIES table are much faster than RELATIONAL(just an FYI)....much much faster ;)
  2. it appears the basic auth user name is not getting through....I typically do a post to this url so the basic auth goes through with curl http://robot-robotTest2:45580RCV5S.ALPHA1.1PT80JZFNKHUG@15.185.92.96:9000/api/registerV1

Notice in 2 that the robot name and password are part of the url...ie. http://{username}:{password}@{host}

  1. I added a precondition higher in the code(so you can update your stuff from github and re-run it). It should now say "no username found in basic auth (or possibly the session if logged in via GUI)" in the logs and I think that is supposed to be sent back in the body of the post response as well with http code 500(I will check shortly).
  2. Which organization are you with? (we like to know our users and while many don't file tickets, we typically are talking to them offline to understand needs out in the industry).

thanks, Dean

— Reply to this email directly or view it on GitHubhttps://github.com/deanhiller/databus/issues/29#issuecomment-26713645.

deanhiller commented 10 years ago

I will run a quick local test myself and post my results

deanhiller commented 10 years ago

documenting my test (If you get a 401, make sure your robot is in the group and has write or admin access as 401 is permission error....ideally, we should never have 500...500 == our bug)

Here are my steps

  1. add database sensorTest
  2. create robot in my someusergroup
  3. go into database sensorTest, click Users and click "Add User" and type in robot-robotTest2 and change pulldown to have Admin permission to this database so he can write

Then I post json with this command(notice my generated robot key is different than yours). I also paste the command prompt and the result that printed

dhiller-23015s:~ dhiller2$ curl -X POST -w "\nRESULT CODE:%{http_code}\n" -d @myregister.txt http://robot-robotTest2:4BFQLAJ7KW.DHILLER-23015S.1TAPDSEZLOKDP@localhost:9000/api/registerV1 {"modelName":"ANDROID4_910e2872a4e96f5e","groups":[]} RESULT CODE:200

with this following json message from yours above(no changes)

{"datasetType":"RELATIONAL_TABLE", "modelName":"ANDROID4_910e2872a4e96f5e", "schema":"sensorTest", "columns":[{"name":"time","dataType":"BigInteger","isIndex":true,"isPrimaryKey":true},{"name":"ROTX","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTY","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTZ","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false}]}

Next, I post json with the following command and again with command prompt and what printed out

dhiller-23015s:~ dhiller2$ curl -X POST -w "\nRESULT CODE:%{http_code}\n" -d @data.json http://robot-robotTest2:4BFQLAJ7KW.DHILLER-23015S.1TAPDSEZLOKDP@localhost:9000/api/postdataV1 RESULT CODE:200 dhiller-23015s:~ dhiller2$

the json I sent in was

{"_dataset":[ {"time":"1371579942260","ROTX":"10100.0","ROTY":"10100.0","ROTZ":"10100.0","_tableName":"ANDROID4_910e2872a4e96f5e"}, {"time":"1371579942265","ROTX":"5470.0","ROTY":"10","ROTZ":"5470.0","_tableName":"ANDROID4_910e2872a4e96f5e"} ]}

so now

  1. let me know if any of your steps are different (especially if you gave a group permission to the database and the robot is in that group instead of giving the robot direct access as I could test that method too)
  2. Are you currently getting 500 or 401 right now with the post of data using "curl"?

thanks, Dean

kbrems commented 10 years ago

Everything works fine with curl. It is when I try to call the registerV1 api programmatically that I always get the error 500.

My robot is a Group Admin type – using that same robot and database via curl works fine. Robot-robotTest2 is a user in group sensorGroup, which is a user of database sensorTest. I could not create a database user without first creating a group.

I just tried also adding my robot-robotTest2 as a direct user (Admin) to my database and I still get the 500 error when running programmatically.

Karen

From: deanhiller [mailto:notifications@github.com] Sent: Monday, October 21, 2013 11:58 AM To: deanhiller/databus Cc: Brems, Karen (HP Labs) Subject: Re: [databus] "Oops, an error occured" server error 500 when registering table data (#29)

documenting my test (If you get a 401, make sure your robot is in the group and has write or admin access as 401 is permission error....ideally, we should never have 500...500 == our bug)

Here are my steps

  1. add database sensorTest
  2. create robot in my someusergroup
  3. go into database sensorTest, click Users and click "Add User" and type in robot-robotTest2 and change pulldown to have Admin permission to this database so he can write

Then I post json with this command(notice my generated robot key is different than yours). I also paste the command prompt and the result that printed

dhiller-23015s:~ dhiller2$ curl -X POST -w "\nRESULT CODE:%{http_code}\n" -d @myregister.txt http://robot-robotTest2:4BFQLAJ7KW.DHILLER-23015S.1TAPDSEZLOKDP@localhost:9000/api/registerV1 {"modelName":"ANDROID4_910e2872a4e96f5e","groups":[]} RESULT CODE:200

with this following json message from yours above(no changes)

{"datasetType":"RELATIONAL_TABLE", "modelName":"ANDROID4_910e2872a4e96f5e", "schema":"sensorTest", "columns":[{"name":"time","dataType":"BigInteger","isIndex":true,"isPrimaryKey":true},{"name":"ROTX","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTY","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false},{"name":"ROTZ","dataType":"BigDecimal","isIndex":false,"isPrimaryKey":false}]}

Next, I post json with the following command and again with command prompt and what printed out

dhiller-23015s:~ dhiller2$ curl -X POST -w "\nRESULT CODE:%{http_code}\n" -d @data.json http://robot-robotTest2:4BFQLAJ7KW.DHILLER-23015S.1TAPDSEZLOKDP@localhost:9000/api/postdataV1 RESULT CODE:200 dhiller-23015s:~ dhiller2$

the json I sent in was

{"dataset":[ {"time":"1371579942260","ROTX":"10100.0","ROTY":"10100.0","ROTZ":"10100.0","tableName":"ANDROID4_910e2872a4e96f5e"}, {"time":"1371579942265","ROTX":"5470.0","ROTY":"10","ROTZ":"5470.0","tableName":"ANDROID4910e2872a4e96f5e"} ]}

so now

  1. let me know if any of your steps are different (especially if you gave a group permission to the database and the robot is in that group instead of giving the robot direct access as I could test that method too)
  2. Are you currently getting 500 or 401 right now with the post of data using "curl"?

thanks, Dean

— Reply to this email directly or view it on GitHubhttps://github.com/deanhiller/databus/issues/29#issuecomment-26745555.