twosigma / beakerx

Beaker Extensions for Jupyter Notebook
http://BeakerX.com
Apache License 2.0
2.78k stars 382 forks source link

KDB autotranslation support #3493

Closed amaurydar closed 8 years ago

amaurydar commented 8 years ago

I have encountered issues using autotranslation to and from KDB q, both on Linux and OS X

.bk.s[foo;1e6]`works as expected

.bk.s[foo;1e7]` gives the following error

[qtp915500720-21] WARN org.eclipse.jetty.servlet.ServletHandler - 
javax.servlet.ServletException: org.codehaus.jackson.JsonParseException: Unexpected character (' ' (code 32)) in numeric value: Exponent indicator not followed by a digit
 at [Source: java.io.StringReader@6f11007; line: 1, column: 2]
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:420)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:538)
    at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:716)
    at javax.servlet.http.HttpServlet.service(HttpServlet.java:668)
    at com.google.inject.servlet.ServletDefinition.doService(ServletDefinition.java:263)
    at com.google.inject.servlet.ServletDefinition.service(ServletDefinition.java:178)
    at com.google.inject.servlet.ManagedServletPipeline.service(ManagedServletPipeline.java:91)
    at com.google.inject.servlet.FilterChainInvocation.doFilter(FilterChainInvocation.java:62)
    at com.google.inject.servlet.ManagedFilterPipeline.dispatch(ManagedFilterPipeline.java:118)
    at com.google.inject.servlet.GuiceFilter.doFilter(GuiceFilter.java:113)
    at org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1465)
    at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:499)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:137)
    at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:522)
    at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1088)
    at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:428)
    at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1020)
    at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
    at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
    at org.eclipse.jetty.server.Server.handle(Server.java:370)
    at org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
    at org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:960)
    at org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1021)
    at org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:865)
    at org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
    at org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:668)
    at org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
    at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
    at org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
    at java.lang.Thread.run(Thread.java:745)
Caused by: org.codehaus.jackson.JsonParseException: Unexpected character (' ' (code 32)) in numeric value: Exponent indicator not followed by a digit
 at [Source: java.io.StringReader@6f11007; line: 1, column: 2]
    at org.codehaus.jackson.JsonParser._constructError(JsonParser.java:1433)
    at org.codehaus.jackson.impl.JsonParserMinimalBase._reportError(JsonParserMinimalBase.java:521)
    at org.codehaus.jackson.impl.JsonParserBase.reportUnexpectedNumberChar(JsonParserBase.java:969)
    at org.codehaus.jackson.impl.ReaderBasedParser.parseNumberText(ReaderBasedParser.java:754)
    at org.codehaus.jackson.impl.ReaderBasedParser.nextToken(ReaderBasedParser.java:482)
    at org.codehaus.jackson.map.ObjectMapper._initForReading(ObjectMapper.java:2770)
    at org.codehaus.jackson.map.ObjectMapper._readMapAndClose(ObjectMapper.java:2718)
    at org.codehaus.jackson.map.ObjectMapper.readValue(ObjectMapper.java:1863)
    at com.twosigma.beaker.core.rest.NamespaceRest.set(NamespaceRest.java:71)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:497)
    at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
    at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
    at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
    at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:302)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
    at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
    at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1511)
    at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1442)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1391)
    at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1381)
    at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
    ... 30 more

It seems the bug comes from JSON generation : q).j.j 1e6 "1000000" q).j.j 1e7 "1e+07" In the HTTP POST request to store the beaker object, the + is misinterpreted. It could be encoded as %2B instead.

I was able to fix the bug by replacing in beaker-notebook/core/config/plugins/eval/kdb/q.q

s: {[name;val]
    j:.j.j raze 4_"\n" vs bc_post["/rest/namespace/set"; (
        "sync=true";
        "&name="; string name;
        "&value="; .j.j val
    )];
    $[j~"\"ok\"";`ok;`$j]
    }

with

s: {[name;val]
    j:.j.j raze 4_"\n" vs bc_post["/rest/namespace/set"; (
        "sync=true";
        "&name="; string name;
        "&value="; ssr[.j.j val;"+";"%2B"]
    )];
    $[j~"\"ok\"";`ok;`$j]
    }

There might be other conflicts between JSON generation with .j.j and HTTP POST requests that I haven't encountered though

scottdraves commented 8 years ago

thanks for the report, will look into it and get back to you.

scottdraves commented 8 years ago

Thanks again. The original author (Michael Pymm) provided a more general fix. Should be merged shortly and will appear in the next release.