Open philrz opened 1 month ago
Building further off what was shown previously, in the absence of the proposed purpose-built search_replace
function written in Go as a core part of the Zed language, it could be written as a user-defined function such as this search-replace-func.zed
.
func search_replace (s, change_map, default): (
coalesce((over change_map with s | switch (case regexp(key, s) != null => yield value | head 1)), default)
)
I've also broken out the map of pairs for search/replace in this changes.zed
.
const changes = |{
"foo|bar": "The Foobar category",
"a.*\\-z.*": "The A-to-Z category"
}|
Putting it all together at the command line:
$ zq -I search-replace-func.zed -I changes.zed 'category := search_replace(desc, changes, "The default category")' descriptions.zson
{desc:"It's foo time",category:"The Foobar category"}
{desc:"a is the first letter-z is last",category:"The A-to-Z category"}
{desc:"Something else",category:"The default category"}
This shows simple modularity at the CLI. We've also discussed the idea of more formally introducing "Zed modules" (#2599) and related concepts like having a standard library of functionality written as Zed to augment the rest of the core functionality implemented in Go, community users being able to share their own modules/libraries easily such as via GitHub, and so forth. This opens the door to being able to deliver enhancements in the short term as Zed that might be hacky at times but can be used like a black box. If perf problems or functional gaps in such an implementation limit usability, usage in its initial form would help validate the use cases for more ambitious core functionality later written in Go.
Also related: The original user later expressed an interest in being able to pull the search/replace pairings from a pool rather than defining them in a const
. I was able to hack something together that does this using existing building blocks, but it's super ugly. A better approach would be if we had what's described in #3201.
Separately, @mattnibs has looked over everything shown here and had the impression that a "cross-product join" would be the way he'd approach this problem.
tl;dr
A community zync user had created a
switch
-based Zed program for a large search-and-replace task in their data transformation pipeline. They asked for some assistance in simplifying it for maintainability. We came up with some improvements using existing Zed building blocks, but in the end @mccanne had the following thought for something new we could write to nail this directly:Details
At the time this issue is being filed, Zed it at commit 9766d17.
The original question from the community zync user was posed as:
Indeed, this is not currently possible in Zed. There's other examples of places where a user might expect to be able to use regexps but can't (#4917).
To illustrate the use case, the user shared their program, but it's confidential and can't be pasted here. However, here's a simplified program
switch-grep.zed
that uses their approach:With input data
descriptions.zson
:Running it:
The user's program actually had about 70
case
statements.In addition to its sheer size, a couple other challenges with maintainability are evident here:
case (grep(..., desc)) => category := ...
const
, they could be more easily re-used in other contexts (e.g., define them in a separate file that's included withzq -I ...
to be invoked in many programs)After a little hacking, I found this could be improved to this
switch-regexp.zed
:Running it gives the same output we saw before.
Things to highlight:
over
we're able to avoid the repeat of all thecase
clausesregexp
function instead ofgrep
we're able to leverage the former's unique ability to take a regular expression that's defined as a string, and strings can be defined viaconst
. However, this does come with one caveat: Some escape sequences that aren't valid for a string but are for regex (such as the\-
in the second pattern) now need a double backslash.The user was satisfied with this improvement. However, having watched this unfold, @mccanne had the idea quoted above for a purpose-built
search_replace
function that would allow the user to avoid needing to know or look up thecoalesce
/over
/switch
/regexp
combination shown here. If regexps in their/ /
form also became first class concepts at the same time that would surely be convenient to the user as well since they'd be able to avoid the "double backslash" overhead when creating the map (and hence more easily re-use regexps from other tools without modification), but this seems orthogonal.