finos / perspective

A data visualization and analytics component, especially well-suited for large and/or streaming datasets.
https://perspective.finos.org/
Apache License 2.0
8.48k stars 1.18k forks source link

Better string functions in expressions #1527

Open sc1f opened 3 years ago

sc1f commented 3 years ago

Feature Request

Description of Problem:

Some common string functions for parsing and creating new strings should be added to the expressions engine using ExprTk:

texodus commented 3 years ago

I'm going to leave some use cases I've found for these in comments, starting with this one.

I have a column in a CSV which is parsed as a string, in the format "${currency} ${currency_symbol}${value}", e.g. "USD $1000". Ideally I'd like 2 columns, Currency and Value, and they should be tolerant of failure to parse. This could be better accomplished with split(), but until there is vector support for Perspective+ExprTK, we could use

// Currency (string)
var idx = find("Bad Column", " ");
if (idx != -1) {
    float(substring("Bad Column", 0, idx))
} else {
    null()
}
// Value (integer)
var idx = find("Bad Column", " ");
if (idx != -1) {
    float(substring("Bad Column", idx, strlen("Bad Column")))
} else {
    null()
}
texodus commented 3 years ago

I have longitude/latitude coordinates in the form of a string, e.g. (123, 456), and I'd like to plot it via perspective-viewer-mapbox plugin.

// Longitude
var idx = find("Bad Column", ",");
if (idx != -1) {
    float(substring("Bad Column", 1, idx)))
} else {
    null()
}
// Latitude
var idx = find("Bad Column", ",");
if (idx != -1) {
    float(substring("Bad Column", idx + 1, strlen("Bad Column") - 1))
} else {
    null()
}
texodus commented 3 years ago

I have various manually-entered fields with (mis/alt)-spellings I would like to pivot on, for example "NC", "N.C.", "N. C." can be normalized to "NC" with

replace_all("State", '[\.\s]', '')