script_fields not work after upgrade to kuzzle version 2

[cklinx]

After upgrade kuzzle from 1 to version 2, script_fields stop to work. To reproduce if enough execute this query

{
    "script_fields": {
        "test1": {
            "script": {
                "lang": "painless",
                "source": "doc['Your Variable'].value"
            }
        }
    },
    "query": {}
}

Cannot get "test1" in the result but only the normal query.

{
    "requestId": "c020782d-1a56-49c3-b585-36f74e01ab4e",
    "status": 200,
    "error": null,
    "controller": "document",
    "action": "search",
    "collection": "asd",
    "index": "asd",
    "volatile": null,
    "result": {
        "hits": [{
                "_id": "yTRD1G4BC56TnFTJHsph",
                "_score": 1,
                "_source": {
                    "asd": "ppippo",
                    "_kuzzle_info": {
                        "author": "-1",
                        "createdAt": 1575519198812,
                        "updatedAt": null,
                        "updater": null
                    }
                }
            }
        ],
        "total": 1
    }
}

Elasticsearch is 7.4
kuzzle v2

In kuzzle v1 it works fine, any idea?

[Aschen]

Hi @cklinx,

I tried to reproduce your issue with the following script without success:

const {
  Kuzzle,
  WebSocket
} = require('kuzzle-sdk')

const kuzzle = new Kuzzle(new WebSocket('localhost'))

kuzzle.on('networkError', err => console.log(err));

(async () => {
  try {
    await kuzzle.connect();
    const query = {
      "script_fields": {
        "test1": {
          "script": {
            "lang": "painless",
            "source": "doc['age'].value"
          }
        }
      },
      "query": {}
    };

    if (! await kuzzle.collection.exists('index', 'collection')) {
      await kuzzle.index.create('index');
      await kuzzle.collection.create('index', 'collection');
      await kuzzle.document.create('index', 'collection', { age: 21 });
      await kuzzle.document.create('index', 'collection', { age: 42 });
      await kuzzle.collection.refresh('index', 'collection');
    }

    let res = await kuzzle.document.search('index', 'collection', query);
    console.log(JSON.stringify({ with_script: res.hits }, null, 2));
    res = await kuzzle.document.search('index', 'collection', { query: {} });
    console.log(JSON.stringify({ without_script: res.hits }, null, 2));
  }
  catch (error) {
    console.log(error);
  }
  finally {
    kuzzle.disconnect();
  }
})();

Can you provide me a way to reproduce your issue so I can investigate?

[cklinx]

Hi @Aschen, did you get "test1" variable in the result?

I tried to do a fresh install and just make a curl request with postman but i dont see the "fields" where script_fields are located.

[cklinx]

I downloaded elastic search without kuzzle and script_fields is visible. Query:

{
    "query" : {
        "match" : { "user": "kimchy" }
    },
    "script_fields": {
        "number-of-actors": {
            "script": {
                "inline": "params['_source']['user']"
            }
        }
    }
}

Result:

{
  "took": 3,
  "timed_out": false,
  "_shards": {
    "total": 1,
    "successful": 1,
    "skipped": 0,
    "failed": 0
  },
  "hits": {
    "total": {
      "value": 1,
      "relation": "eq"
    },
    "max_score": 0.2876821,
    "hits": [
      {
        "_index": "twitter",
        "_type": "_doc",
        "_id": "1",
        "_score": 0.2876821,
        "fields": {
          "number-of-actors": [
            "kimchy"
          ]
        }
      }
    ]
  }
}

[Aschen]

Ok I have identified the issue, it's because of the way we are returning the search results.
In the v2, we try to be more independent from ES and not just return the raw ES queries to the user. CF: https://github.com/kuzzleio/kuzzle/blob/master/lib/services/elasticsearch.js#L252 I will propose a fix soon

[cklinx]

Hi @Aschen, Thanks! Let me know when you solve the problem!

[Aschen]

~I opened a PR, you should have this fix in the next release~ We are currently discussing about this feature

[scottinet]

Or maybe not :expressionless:

[cklinx]

ok, i get your mind. Ok disable script_fields for performance but how get, for example, variables as geo_distance? It is very simple to get with script.

"script_fields" : {
          "distance" : {
            "script" : {
              "lang": "painless",
              "inline": "doc['centroid'].planeDistance("+currentPosition.location.lat+","+currentPosition.location.lon+")",

            }
          }

This variable is calculated if you sort for distance also, so doesn't impact in performance.

Any idea? Thanks

[scottinet]

Hi @cklinx

You can do this by using a small plugin adding a new API route: https://docs.kuzzle.io/core/2/plugins/guides/controllers/

If you do that, you shouldn't allow scripted queries either, instead your new route should perform the scripted query on behalf of the requesting user. That way, you can safely provide an otherwise dangerous feature.

We don't want to allow this ourselves, because the only way to do that would be to allow any and all scripts, and that makes it impossible to secure by administrators.

[cklinx]

Thanks @scottinet! I'm going to study the plugin.

[cklinx]

Hi @Aschen, @scottinet, i successfully installed my plugin in the docker using this example. I got info() function response:

return `Hello from example:info. Current user id: ${currentUser._id}`;

Actually i don't understand how create a function that allow me to execute a query and get script_fields. Can you help me please? Thanks in advance.

[scottinet]

Hi,

In the context provided to your plugin's init function, you get a pre-configured ES client that you can instantiate: https://docs.kuzzle.io/core/2/plugins/plugin-context/constructors/esclient/

From there, you can easily add new API routes that will use that ES client to execute scripts in a secure way.

[Aschen]

You also have a complete example of a controller route using the ESClient in the functional tests:

• https://github.com/kuzzleio/kuzzle/blob/master/plugins/available/functional-test-plugin/index.js#L11
• https://github.com/kuzzleio/kuzzle/blob/master/plugins/available/functional-test-plugin/index.js#L52

[cklinx]

Really thanks guys! With your examples its works!