Closed RoelantStegmann closed 4 years ago
I also have a similar issue - I am trying to get all the rows from executing a sql query, as opposed to just the 2500 rows.
%%sql -o bookratingslist -n 1000000 select a.userid, a.isbn, a.rating, b.booktitle, c.location, c.age from bookratings.bx_book_ratings_csv a join bookratings.bx_books_csv b on upper(rtrim(a.isbn)) = upper(rtrim(b.isbn)) join bookratings.bx_users_csv c on c.userid = a.userid
An error was encountered:
Invalid status code '400' from http://
Any thoughts on resolving this issue?
Same issue here. It would be nice to be able to configure Livy timeouts from sparkmagic %%configure command
Sparkmagic interacts with Livy via REST API as a client using requests library and only allow properties that are from /POST sessions
payload to be configurable.
https://livy.apache.org/docs/latest/rest-api.html
Sparkmagic creates the session by sending HTTP POST request on /sessions
endpoint.
Unfortunately, the livy server level properties cannot be overridden from a client request. Also, a change in server configuration requires a restart in most cases.
You need to modify your livy.conf file inorder to fix this https://github.com/apache/incubator-livy/blob/v0.7.0-incubating/conf/livy.conf.template#L51-L62
Same problem here, anyone figured out how to solve this?
Hi everyone, until Livy 0.6.0 there was an issue (https://issues.apache.org/jira/browse/LIVY-547) that even if the session was active the session timeout would kill the session anyway and this was only fixed on version 0.7.0 (which was released recently), so you need to upgrade to 0.7.0 or apply the issue patch to Livy in order to avoid this issue.
@RoelantStegmann @shaikmanu797 regarding retrieving Livy configuration through sparkmagic, Livy must enable this on their REST API, I would recommend searching Livy's Jira Board and see if they have something regarding this topic and if they eventually add it, we can add it on sparkmagic.
Going to close this issue for now since it is a problem with Livy, but any problem or extra comments fell free to add it here. \o/
Same problem, any solutions?
Hi,
After spending 1h on a cell, my livy ends with a 400 error.
After some digging I discovered that is the following config
livy.server.session.timeout=3600000
(or less likelylivy.server.session.state-retain.sec=3600000
). I can change it in the Ambari UI of my HDinsight cluster, but of course I would like to configure it with sparkmagic. Is that possible?A side question: how can I see the value of livy & spark parameters while running my kernel? For example, intuitively I expected one of this to work. It is nice to check if
config.json
settings are working, or to look up defaults.