sourav-mazumder / Data-Science-Extensions

71 stars 63 forks source link

Watson PI - Invalid JSON input at line 1, sub_code='S00005' #2

Open kamalsky opened 5 years ago

kamalsky commented 5 years ago

I'm passing a temp table of book reviews for Watson PI API.

The content of the review has more than 600 words, which meets the minimum requirements as specified by the API, but I keep getting Invalid JSON input at line 1, sub_code='S00005' when loading the response on a Spark Dataframe.

It seems like it's authenticating but it's not passing the content-type parameters.

Code - Pyspark shell -- spark version 2.3.1, Debian 1.9

#temp table view
df.createOrReplaceTempView("api")

#watson pi uri
watsonuri = "https://gateway.watsonplatform.net/personality-insights/api/v3/profile?version=2017-10-13"

#passing parameters
prmsWatson = { 'url' : watsonuri, 'input' : 'api',  
    'userId' : '73b4a7f1-606a-4bb1-9320-61102575fe40',
    'userPassword' : 'NFfxWpFpEea8', 
    'readTimeout' : '10000', 
    'connectionTimeout' : '2000', 
    'partitions' : '10', 
    'callStrictlyOnce' : 'Y'}

#collecting response on a dataframe
watsonDf1 = spark.read.format("org.apache.dsext.spark.datasource.rest.RestDataSource").options(**prmsWatson).load()

watsonDf1.createOrReplaceTempView("watsonresult1")
watsonpersonality1 = spark.sql("select * from watsonresult1")
watsonpersonality1.take(1)

Reponse `[Row(content='this is a book i have been meaning to read since 1997 that s actually even before it was officially published which might seem weird if you didn t already know that wally lamb was teaching writing at my high school at the time he was working on this novel and if you didn t know that my freshman english class helped edit one of the first chapters back in 1994 or 1995 the novel tells the story of dominick and thomas birdsey identical twins dealing with very fraternal problems namely that thomas is schizophrenic and dominick is almost his last remaining caregiver the novel is about many things but most of all as told from dominick s perspective it is about forgiveness of the self of family of our pasts both personal and collective i know a number of the people named in the acknowledgements including several of my high school teachers and my high school gets a shout out as well i didn t know lamb very well but i always found him pretty favorable one of those teachers you like because they never actually have to give you a grade i never read this book until now because i also have a connection to one of those named in the dedication to sam deglin a high school friend who along with her younger brother randy was killed in a freak car accident in front of our high school in january of 1997 i knew she and randy were named in the dedication and for this reason i avoided the book i didn t want to know how their deaths connected wtih the story and i didn t want to be reminded at all of my hometown in southeastern connecticut where i assumed correctly it turns out the book was set i think enough time passed in order for me to feel ok about this book and in fact once i began reading i couldn t stop the references to people places and events of my hometown were only part of the allure this book is beautifully written and the structure of the novel which jumps back and forth between the 1920s 1940s 1960s and 1980s with refreshingly little unnecessary exposition contributes to the sense that like the mind of thomas birdsey who i guess you could call the antagonist things are never what they seem and that sometimes the answers to our biggest fears and questions were right in front of us all along i would recommend this book highly but it certainly helps to know a thing or two about growing up in norwich connecticut too', id='5162', output=Row(code=400, error='Invalid JSON input at line 1, column 8', sub_code='S00005'), reviewer_name='Heather', rn='1')]

`