search

marklogic-community / roxy

Deployment tool for MarkLogic applications. Also provides optional unit test and XQuery MVC structure
Other
87 stars 66 forks source link

JSON sending in the request body is not being converted into map:map in ML8 #418

Closed jeet1212 closed 9 years ago

jeet1212 commented 9 years ago

We are upgrading ML6 to ML8 and using roxy as well. The roxy is customized based on the customer requirement.

For one of the REST request "/setcache", we are sending below data to cache and that is being converted into map (key value pair)

ML6-

Request in Body-

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 PM":{"user":"<user-profile wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\"><docType>Users</docType><username type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39</username><userId type=\"string\">lrunner91@wk.com</userId><email type=\"string\">lrunner91@wk.com</email><userStatus type=\"string\">Active</userStatus><defaultContentCollection>ALL</defaultContentCollection><role type=\"string\">user</role><trialEndDate type=\"string\"/><activationDate type=\"string\">2013-12-25</activationDate><phoneNumber type=\"string\"/><sendWelcomeEmail type=\"string\"/><additionalEmail type=\"string\"/><filingContent type=\"string\">Allowed</filingContent><firstName type=\"string\">John</firstName><lastName type=\"string\">Doe</lastName><orgCode type=\"string\">WK001</orgCode><orgName type=\"string\">LoadRunner</orgName><salesTerritory type=\"string\"/><password>5f4dcc3b5aa765d61d8327deb882cf99</password><pw>password</pw><countOfLoggedIn>11</countOfLoggedIn><created>2014-07-11T11:05:11.589-05:00</created><createdBy type=\"string\">System</createdBy><modified>2014-07-11T11:05:11.589-05:00</modified><modifiedBy>yun.yang@wolterskluwer.com</modifiedBy></user-profile>", "session":"<session wk-pid=\"a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 PM\"><username>USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39</username><created>2015-04-28T17:48:14.661+05:30</created><expiration>2015-04-28T21:48:14.661+05:30</expiration><currenttime>2015-04-28T17:48:14.661+05:30</currenttime><valid>false</valid></session>"}}

Converted into below map using ML6-

map:map(<map:map xmlns:xs="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xmlns:map="http://marklogic.com/xdmp/map"><map:entry key="USERSESSION:97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24 PM"><map:value><map:map><map:entry key="user"><map:value><user-profile wk-pid="USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa"><username>USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa</username><docType>Users</docType><userId type="string">sjauhari@innodata.com</userId><firstName type="string">shruti</firstName><lastName type="string">jauhari</lastName><orgCode type="string">FORD-101</orgCode><orgName type="string">Ford Motor Company</orgName><filingContent type="string">Allowed</filingContent><userStatus type="string">Active</userStatus><trialEndDate type="string">2018-12-31</trialEndDate><role type="string">user</role><telephoneNumber type="string">212-555-1212</telephoneNumber><salesTerritory type="string">Northeast</salesTerritory><sendWelcomeEmail type="string">False</sendWelcomeEmail><defaultContentCollection type="string">default</defaultContentCollection><additionalEmail type="string">skhan2@innodata.com</additionalEmail><password>5f4dcc3b5aa765d61d8327deb882cf99</password><pw>password</pw><countOfLoggedIn>11</countOfLoggedIn><created>2015-04-22T15:37:32.151+05:30</created><createdBy>USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00</createdBy><modified>2015-04-22T15:37:32.151+05:30</modified><modifiedBy>USR-2fe944cc-013e-5b5a-0dba-7e98c3ffab00</modifiedBy></user-profile></map:value></map:entry><map:entry key="session"><map:value><session wk-pid="97f7d8ec0b5bba928cb025bd1d4d28d4|4/28/2015 9:57:24 PM"><username>USR-59fb4f7d-de03-1e37-a23c-5c3d71552eaa</username><created>2015-04-28T17:57:24.34+05:30</created><expiration>2015-04-28T21:57:24.34+05:30</expiration><currenttime>2015-04-28T17:57:24.34+05:30</currenttime><valid>false</valid></session></map:value></map:entry></map:map></map:value></map:entry></map:map>)

ML8 -

Input and converted output is same. i.e. without map:

{"USERSESSION:a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 PM":{"user":"<user-profile wk-pid=\"USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39\"><docType>Users</docType><username type=\"string\">USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39</username><userId type=\"string\">lrunner91@wk.com</userId><email type=\"string\">lrunner91@wk.com</email><userStatus type=\"string\">Active</userStatus><defaultContentCollection>ALL</defaultContentCollection><role type=\"string\">user</role><trialEndDate type=\"string\"/><activationDate type=\"string\">2013-12-25</activationDate><phoneNumber type=\"string\"/><sendWelcomeEmail type=\"string\"/><additionalEmail type=\"string\"/><filingContent type=\"string\">Allowed</filingContent><firstName type=\"string\">John</firstName><lastName type=\"string\">Doe</lastName><orgCode type=\"string\">WK001</orgCode><orgName type=\"string\">LoadRunner</orgName><salesTerritory type=\"string\"/><password>5f4dcc3b5aa765d61d8327deb882cf99</password><pw>password</pw><countOfLoggedIn>11</countOfLoggedIn><created>2014-07-11T11:05:11.589-05:00</created><createdBy type=\"string\">System</createdBy><modified>2014-07-11T11:05:11.589-05:00</modified><modifiedBy>yun.yang@wolterskluwer.com</modifiedBy></user-profile>", "session":"<session wk-pid=\"a57e7e98cae8a7857285cd9a66de813f|4/28/2015 9:48:14 PM\"><username>USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39</username><created>2015-04-28T17:48:14.661+05:30</created><expiration>2015-04-28T21:48:14.661+05:30</expiration><currenttime>2015-04-28T17:48:14.661+05:30</currenttime><valid>false</valid></session>"}}

I suspect roxy router.xqy and request.xqy are creating some issues however not able to identify the issue yet.

Please suggest the fix for the same.

Other errors in Errorlog.txt-

2015-04-28 17:11:18.302 Notice: FilingsTool: Expecting map (INVALID-PARAMS):
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/controllers/setcache.xqy, at 29:22,
2015-04-28 17:11:18.302 Notice: FilingsTool: in xdmp:function(fn:QName("http://marklogic.com/roxy/controller/setcache","main"), "/app/controllers/setcache.xqy")() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $params = document{text{"{"USERSESSION:3aa2eb094d15d973e355d54812cf5720|4/28/2015 8:..."}}
2015-04-28 17:11:18.302 Notice: FilingsTool: $e = <error:error xsi:schemaLocation="http://marklogic.com/xdmp/error error.xsd" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:error="http://marklogic.com/xdmp/error"><error:code>XDMP-UNEXPECTED</error:code><error:name>err:XPST0003...</error:error>
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/lib/router.xqy, at 97:16,
2015-04-28 17:11:18.302 Notice: FilingsTool: in router:route() [1.0-ml]
2015-04-28 17:11:18.302 Notice: FilingsTool: $x = ()
2015-04-28 17:11:18.302 Notice: FilingsTool: $valid-request = fn:true()
2015-04-28 17:11:18.302 Notice: FilingsTool: in /app/restful-router.xqy, at 20:0 [1.0-ml]

Regards, Indrajeet

grtjn commented 9 years ago

My first question would be, how was it converting the JSON into map:map before. I don't think that is default Roxy behavior. I am afraid you will have to share some bits of your custom code. Can you start with the main function of the setcache controller, and tell us whether /app/lib/router.xqy is in any way different from https://github.com/marklogic/roxy/blob/master/src/roxy/router.xqy? It might also help to show relevant lines from restful-router.xqy..

jeet1212 commented 9 years ago

After more research, I think this is some misbehavior due to key/value pair handling in ML8. I might be wrong. Please ignore if this is not the roxy framework issue however I am struggling to identify/understand this misbehavior.

roxy/router.xqy has been overridden. 

$user = &lt;user-profile wk-pid="USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39"&gt;&lt;docType&gt;Users&lt;/docType&gt;&lt;username type="string"&gt;USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39&lt;/username&gt;&lt;userId type="string"&gt;lrunner91@wk.com&lt;/userId&gt;&lt;email type="string"&gt;lrunner91@wk.com&lt;/email&gt;&lt;userStatus type="string"&gt;Active&lt;/userStatus&gt;&lt;defaultContentCollection&gt;ALL&lt;/defaultContentCollection&gt;&lt;role type="string"&gt;user&lt;/role&gt;&lt;trialEndDate type="string"/&gt;&lt;activationDate type="string"&gt;2013-12-25&lt;/activationDate&gt;&lt;phoneNumber type="string"/&gt;&lt;sendWelcomeEmail type="string"/&gt;&lt;additionalEmail type="string"/&gt;&lt;filingContent type="string"&gt;Allowed&lt;/filingContent&gt;&lt;firstName type="string"&gt;John&lt;/firstName&gt;&lt;lastName type="string"&gt;Doe&lt;/lastName&gt;&lt;orgCode type="string"&gt;WK001&lt;/orgCode&gt;&lt;orgName type="string"&gt;LoadRunner&lt;/orgName&gt;&lt;salesTerritory type="string"/&gt;&lt;password&gt;5f4dcc3b5aa765d61d8327deb882cf99&lt;/password&gt;&lt;pw&gt;password&lt;/pw&gt;&lt;countOfLoggedIn&gt;11&lt;/countOfLoggedIn&gt;&lt;created&gt;2014-07-11T11:05:11.589-05:00&lt;/created&gt;&lt;createdBy type="string"&gt;System&lt;/createdBy&gt;&lt;modified&gt;2014-07-11T11:05:11.589-05:00&lt;/modified&gt;&lt;modifiedBy&gt;yun.yang@wolterskluwer.com&lt;/modifiedBy&gt;&lt;/user-profile&gt;

$session = &lt;session wk-pid="2f27109f160ff5f2e6993023086a5cdf|4/29/2015 4:2:0 AM"&gt;&lt;username&gt;USR-b740cbc5-5e27-8b1c-d97a-1329625f6b39&lt;/username&gt;&lt;created&gt;2015-04-29T00:02:00.372+05:30&lt;/created&gt;&lt;expiration&gt;2015-04-29T04:02:00.372+05:30&lt;/expiration&gt;&lt;currenttime&gt;2015-04-29T00:02:00.372+05:30&lt;/currenttime&gt;&lt;valid&gt;false&lt;/valid&gt;&lt;/session&gt;

$auth:CACHE-PREFIX = "USERSESSION"

auth:login-User($user) --->auth:cache-user($user, $session) -->ch:set-cache($name as xs:string, $value) -->ch:set-cache($hosts as xs:string*, $port as xs:string, $map) -->/setcache

    declare function auth:cache-user($user, $session)
    {
      if($auth:ENABLE-CACHING = fn:true() ) then (
        let $name := fn:concat($auth:CACHE-PREFIX,$session/@wk-pid)
        let $map := map:map()
        let $_ := map:put( $map, "user", $user)
        let $_ := map:put( $map, "session", $session)
        return ch:set-cache($name, $map)   
      ) else ()
    };

    declare function ch:set-cache($name as xs:string, $value)
    {
      let $server := xdmp:server-status(xdmp:host(), xdmp:server())
      let $port := $server/s:port/text()
      let $params := map:map()
      let $_ := map:put($params, $name, $value)
      let $hosts := ch:server-hosts()
      return ch:set-cache($hosts, $port, $params)

    };

    declare function ch:set-cache($hosts as xs:string*, $port as xs:string, $map)
    {
      for $host in $hosts
      let $url := fn:concat($ch:PROTOCOL,"://",$host,":",$port,"/setcache")
      let $_ := xdmp:log("Invoking setCache on - " || $url,$ch:LOG-LEVEL)
      return 

        xdmp:http-post($url,
          <options xmlns="xdmp:http">
            <data>{xdmp:quote($map}</data>
          </options>)    
    };

setcache.xqy

    declare function c:main() as item()*
    {

      let $params := xdmp:get-request-body()

      let $params := try {
          xdmp:eval($params/node())
        } catch($e) {
          xdmp:log($e),
          fn:error(xs:QName("INVALID-PARAMS"),"Expecting map")
        }              
      let $name := map:keys($params)[1]
      let $value := try {
          xdmp:unquote(map:get($params, $name))/node()
        } catch($e) {
          map:get($params, $name)
        }
      let $value := if(map:keys($value)) then $value else ()             
      let $_ := cache:set-cache-host($name, $value)

      let $result   := 
        <json:object type="object">
          <json:responseCode>200</json:responseCode>
          <json:message>Cache set successful</json:message>
        </json:object>

      return
        (
          ch:add-value("res-code", xs:int($result/json:responseCode) ),
          ch:add-value("res-message", xs:string($result/json:message) ),
          ch:add-value("result",  $result),
          ch:use-view("/worklists/render"),

          ch:add-value(
            "res-header", 
            element header {
              element Date {fn:current-dateTime()},
              element Content-Type {req:get("req-header")/content-type/fn:string()}
            }
          )
        )

    };

app/lib/router.xqy:

declare function router:route()
{
  (: add HTTP header information to the request parameter object :)
    let $x := (
          map:put($req:request, "req-header",
              element header {
                for $header in xdmp:get-request-header-names()
                 return
                  element {fn:lower-case($header)} {
                    xdmp:get-request-header($header)
                  }
              }
          ),
          if (fn:exists(xdmp:get-request-body())) then
              map:put($req:request, "req-body",
                      if (xdmp:get-request-body()/node() instance of element()) then
                        xdmp:get-request-body()/node()
                      else if (xdmp:get-request-body()/node() instance of text()) then
                        xdmp:quote(xdmp:get-request-body()/node())
                      else ()
                  ) 
          else (),
          router:set-format()
     )

    (: PRawal : Checks authentication before executing the request :)   
    let $valid-request := 
          if(fn:not($config:SESSION-AUTHENTICATE)) then fn:true() 
          else if(xs:string($controller) = ("ping")) then fn:true()
          else if(xs:string($controller) = ("login")) then fn:true()
          else if(xs:string($controller) = ("logout")) then fn:true()
          else if(xs:string($controller) = ("verify")) then fn:true()
          else if(xs:string($controller) = ("setcache")) then fn:true()
          else if(xs:string($controller) = ("browseCalibration")) then fn:true()
          else if(xs:string($controller) = ("sessions")) then fn:true()
          else if(xs:string($controller) = ("forgetpassword")) then fn:true()
          else if(xs:string($controller) = ("resetpassword")) then fn:true()
          else auth:authenticate()

  (: run the controller. errors bubble up to the error module :)
     let $data := 
       if($valid-request) then 
       (
            if(fn:string($valid-request)="true") then
                xdmp:apply(xdmp:function(
                    fn:QName(fn:concat("http://marklogic.com/roxy/controller/", $controller), $func),
                    $controller-path)) 
             else if(fn:string($valid-request) = "timeout") then (
                   xdmp:apply(xdmp:function(
                        fn:QName(fn:concat("http://marklogic.com/roxy/controller/login"), "login-timeout"),
                        $login-controller-path))                
             ) else (
                   xdmp:apply(xdmp:function(
                        fn:QName(fn:concat("http://marklogic.com/roxy/controller/login"), "login-error"),
                        $login-controller-path))

             )
        )                                    
        else 
        (            
           xdmp:apply(xdmp:function(
                fn:QName(fn:concat("http://marklogic.com/roxy/controller/login"), "login-error"),
                $login-controller-path))
        )

    (: Roxy options :)
    let $options :=
      for $key in map:keys($ch:map)
      where fn:starts-with($key, "ch:config-")
      return
        map:get($ch:map, $key)

    (: remove options from the data :)
    let $_ :=
        for $key in map:keys($ch:map)
        where fn:starts-with($key, "ch:config-")
        return
           map:delete($ch:map, $key)

    let $format as xs:string := ($options[self::ch:config-format][ch:formats/ch:format = $format]/ch:format, $format)[1]
    let $_ := (
          for $header in map:get($ch:map, "res-header")/* 
          return 
               xdmp:add-response-header(fn:local-name($header), $header/fn:string()),
               xdmp:set-response-code((map:get($ch:map, "res-code"), 200)[1],(map:get($ch:map, "res-message"), "OK")[1]) 
          )

    (: controller override of the view :)
    let $view := ($options[self::ch:config-view][ch:formats/ch:format = $format]/ch:view, $default-view)[1][. ne ""]

    (: controller override of the layout :)
    let $layout :=
      if (fn:exists($options[self::ch:config-layout][ch:formats/ch:format = $format])) then
        $options[self::ch:config-layout][ch:formats/ch:format = $format]/ch:layout[. ne ""]
      else
        $default-layout

    (: if the view return something other than the map or () then bypass the view and layout :)
    let $bypass as xs:boolean := fn:exists($data) and fn:not($data instance of map:map)

    return
      if (fn:not($bypass) and (fn:exists($view) or fn:exists($layout))) then
        let $view-result :=
          if (fn:exists($ch:map) and fn:exists($view)) then
            rh:render-view($view, $format, $ch:map)
          else
            ()
        return
          if (fn:not($bypass) and fn:exists($layout)) then
            let $_ :=
              if (fn:exists($view-result) and
                  fn:not($view-result instance of map:map) and
                  fn:not(fn:deep-equal(document {$ch:map}, document {$view-result}))) then
                map:put($ch:map, "view", $view-result)
              else
                map:put( $ch:map, "view",
                  for $key in map:keys($ch:map)
                  return
                    map:get($ch:map, $key) )
            return
              rh:render-layout($layout, $format, $ch:map)
          else
            $view-result
      else if (fn:not($bypass)) then
        for $key in map:keys($ch:map)
        return
          map:get($ch:map, $key)
      else
        $data,
        xdmp:log( fn:concat("TEST#1 - ", xdmp:get-original-url()," - Time - ",xdmp:elapsed-time()),"info")
};
joemfb commented 9 years ago

Hi @jeet1212,

To get up and running on MarkLogic 8, try replacing xdmp:eval($params/node()) with xdmp:from-json(xdmp:unquote($params/node())) in the c:main() function of setcache.xqy.

xdmp:unquote will parse your JSON string into an object-node(), and then xdmp:from-json() will convert the object-node() into a json:object, which is compatible with the map:* functions.

I'm still not sure how this would've worked in MarkLogic 6; I wouldn't have thought that xdmp:eval() would parse JSON strings ...

jeet1212 commented 9 years ago

Hi Joseph,

It worked with your suggested code. Thank you so much for your help!

However i was trying doing following ways and it worked but it was carrying two changes,

1) I was adding $map inside a <m></m> and passing <map:map/> to the setcache.xqy <data>{xdmp:quote(<m>{$map}</m>/*)}</data>

declare function ch:set-cache($hosts as xs:string*, $port as xs:string, $map)
{
    for $host in $hosts
    let $url := fn:concat($ch:PROTOCOL,"://",$host,":",$port,"/setcache")
    let $_ := xdmp:log("Invoking setCache on - " || $url,$ch:LOG-LEVEL)
    return 

    xdmp:http-post($url,
     <options xmlns="xdmp:http">
            <data>{xdmp:quote(<m>{$map}</m>/*)}</data>
     </options>)    
};

2) In the setcache.xqy, I was creating map that was getting from xdmp:eval($params/node()) inside c:main() function.

let $params := xdmp:eval($params/node())
let $params := map:map($params)
......

As far as this code is working in ML6, I am also not sure however some ML expert consultants made it workable :)

Thank you again for your time and help!

Regards, Indy

dmcassel commented 9 years ago

@jeet1212, it sounds like we can close this ticket, right?

jeet1212 commented 9 years ago

Yes, Please close the ticket however we are seeing a lot of issues in our API code due to JSON/MAP handling in ML8. We are trying to investigate one by one and fixing those.

if possible, please take a look at my approach to fix the issue and provide your thoughts, will be great as when I tried solution provided by Joseph, It worked in one of the API where I was testing yesterday however failed in other cases.

Though my proposed solution is working in all the places so far.

Thank you for your time and help!

Regards, Indrajeet

grtjn commented 9 years ago

Joe provided a small change, which sounds about right, but might not cover all your code. I am afraid you will have to check everywhere where you were parsing or serializing JSON.

If backwards-compatibility in the code would be important to you, you could consider checking result of eval, and if that is not an element, then apply xdmp:from-json to change object-node() into <jbasic:json>. You could also check xdmp:version(), and if that starts with '8' do something different. But be careful not to use functions that won't compile on ML7. There are a few new function in ML8, like xdmp:from-json-string..