Adding custom header for CSRF works for adding endpoint, but not for adding a session.

p1p3dream commented 4 years ago

I was getting the 400 error Missing Required Header for CSRF protection, when trying to add an endpoint. I added the X-Requested-By Header in livyreliablehttpclient, headers variable, and in reliablehttpclient too for good measure.

I am able to successfully add the endpoint now, but when I try to create session is throwing the same 400 CSRF error. I imagine its something I'm doing wrong, but not sure where to go from here.

~/.local/lib/python3.7/site-packages/sparkmagic/livyclientlib/reliablehttpclient.py in post(self, relative_url, accepted_status_codes, data)
     48         return self._send_request(relative_url, accepted_status_codes, requests.get)
     49 
---> 50     def post(self, relative_url, accepted_status_codes, data):
     51         """Sends a post request. Returns a response."""
     52         return self._send_request(relative_url, accepted_status_codes, requests.post, data)

~/.local/lib/python3.7/site-packages/sparkmagic/livyclientlib/reliablehttpclient.py in _send_request(self, relative_url, accepted_status_codes, function, data)
     55         """Sends a delete request. Returns a response."""
     56         return self._send_request(relative_url, accepted_status_codes, requests.delete)
---> 57 
     58     def _send_request(self, relative_url, accepted_status_codes, function, data=None):
     59         print(self.compose_url(relative_url))

~/.local/lib/python3.7/site-packages/sparkmagic/livyclientlib/reliablehttpclient.py in _send_request_helper(self, url, accepted_status_codes, function, data, retry_count)
     94                 if error:
     95                     raise HttpClientException(u"Error sending http request and maximum retry encountered.")
---> 96                 else:
     97                     raise HttpClientException(u"Invalid status code '{}' from {} with error payload: {}"
     98                                               .format(status, url, text))

HttpClientException: Invalid status code '400' from http://td2vspk0.travp.net:8999/sessions with error payload: <html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 400 </title>
</head>
<body>
<h2>HTTP ERROR: 400</h2>
<p>Problem accessing /sessions. Reason:
<pre>    Missing Required Header for CSRF protection.</pre></p>
<hr /><i><small>Powered by Jetty://</small></i>
</body>
</html>

bobloki commented 4 years ago

conf.custom_headers() is empty in the from_endpoint method even when populated. I'm looking at it to see if I can figure out a simple fix in a pull request.

Modifying the from_endpoint method in headers to include an X-Requested-By fixed it for me.

itamarst commented 4 years ago

@bobloki a PR would be great, thank you!

bobloki commented 4 years ago

After further investigation, my issue was caused by multiple .sparkmagic folders and multiple config.json files. I'm not sure how I got to have this many of them. Maybe after a long time of working with this project or different installations over time?

I had a sparkmagic folder at the C:\ and another in two different user folders, Three total.

This method in livyreliablehttpclient.py is the method that grabs the headers. For me this now works.

@staticmethod
    def from_endpoint(endpoint):
        headers = {"Content-Type": "application/json" }
        headers.update(conf.custom_headers())
        retry_policy = LivyReliableHttpClient._get_retry_policy()
        return LivyReliableHttpClient(ReliableHttpClient(endpoint, headers, retry_policy), endpoint)

conf.custom_headers() contained the correct headers with the following format after pulling them from the config.json

"custom_headers": {"Content-Type": "application/json", "X-Requested-By" : "blah.blah.com"},

jupyter-incubator / sparkmagic

Adding custom header for CSRF works for adding endpoint, but not for adding a session. #613