opensearch-project / opensearch-spark

Spark Accelerator framework ; It enables secondary indices to remote data stores.
Apache License 2.0
14 stars 23 forks source link

[FEATURE]Add `ispresent` PPL command #518

Open YANG-DB opened 1 month ago

YANG-DB commented 1 month ago

Is your feature request related to a problem? Add ispresent command for PPL spark based execution driver .

Since spark sql doesnt offer such expression a possible option would be:

Do you have any additional context?

lukasz-soszynski-eliatra commented 3 weeks ago

I created a working PoC (proof of concept). The source code can be found https://github.com/eliatra/opensearch-spark/pull/3.

The usage of the commands looks as follows.

search source=people | is_present name;
search source=people | is_present age;
search source=people |  is_present age, name;

Please let me know if such an approach meets your expectations.

BTW, the MR is not yet ready. Documentation and tests need to be created.

salyh commented 2 weeks ago

We need a final decision wether this should be a command or a function like asked in https://github.com/opensearch-project/opensearch-spark/issues/512#issuecomment-2331451920

lukasz-soszynski-eliatra commented 2 weeks ago

ispresent implemented as a function. @YANG-DB the MR is ready for review.

Example: search source=people | eval p=ispresent(age) | where ispresent(age);