Collection of ideas for new functions

sharkdp commented 6 years ago

[ ] More string utilities (e.g. regex functions)
[x] Formatting functions (echo 3.1415 | map format "{:.2f}")

leeexyz commented 6 years ago

String

[ ] search and match based on regex
[ ] Add regex support for replace input
[x] Add function endwith a pattern

Arithmetic

[ ] Add mod, pow, div and log etc. functions

Filesystem

[ ] Add function mv to move and rename files
[ ] More functions to filter files based on file mode bits and ownship

sharkdp commented 6 years ago

@leeleeee Sounds great. Especially the mv function, which will be really useful (we should add cp as well)

leeexyz commented 6 years ago

@sharkdp What do you think if we can add du or something like this?

sharkdp commented 6 years ago

@sharkdp What do you think if we can add du or something like this?

In which sense? Like a filesize :: Path -> Int function?

By the way, I just realized that the coreutils du commands works very nicely with functools:

> du | filter -c1 greater_than 8000 | map -c2 abspath
8340    /tmp/demo/folder/subdirectory
8408    /tmp/demo/folder
8480    /tmp/demo

leeexyz commented 6 years ago

Yeah, more precisely it is filesize :: Path -> String.

$ find . | map basename | filter startswith test_ | map filesize mb 
test_file1 1MB 
test_file2 5.2MB

It is fantastic that linux commands are work very well with these functions. 👍

sharkdp commented 6 years ago

filesize is now supported.

sharkdp commented 6 years ago

[ ] fst and snd for array-index access (alias for index 1 and index 2)
[ ] replicate
[ ] reverse

ghost commented 6 years ago

Does split currently work with variable size of whitespace between words? if not, does it make sense to add it? Something like below by default if no separator is provided, then it would be a nice replacement for awk

➜  ~ echo "a b         ccc " | ~/repos/shell-functools/ft/map split | map at 2
ccc

sharkdp commented 6 years ago

Does split currently work with variable size of whitespace between words?

It does not (yet). We could either provide a split_whitespace function which would simply call .split() without argument and/or provide a split_regex function which could be called with map split_regex '\s+'.

vegarsti commented 6 years ago

The reverse function, do you mean on array level, or on string level?

sharkdp commented 6 years ago

I think array-reverse can be useful. Not sure if we need a string-reverse function.

guilhermeleobas commented 6 years ago

Hey @sharkdp, what about creating a max as well as min function in shell-functools? I can work on something if there is interest.

max :: Array -> Int
find . -name cnt.txt | xargs cat | max

sharkdp commented 6 years ago

@guilhermeleobas Sounds great!

Jake6329 commented 5 years ago

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

sharkdp commented 5 years ago

These days, new unix/linux CLI tools are coming out supporting JSON outputs. Would it make sense to create a json function that can parse JSON outputs?

I've been thinking about JSON support in the past, so this sounds interesting. What would this function do, exactly?

Jake6329 commented 5 years ago

I can imagine there would a command named json which could be used like this:

$ curl http://echo.jsontest.com/key1/value1 | json key1

The curl command outputs {"key1": "value1"}. The whole command would output value1. The argument to json command would specify the path to the value you want. It could support the full json path spec or just a small part of it.

And then there would also a function called json which could be used like this:

$ curl <url that returns json in each line> | map json <json path>

In this case, the assumption is that the curl outputs a json-encoded string in each line, which is also a proposed standard format called jsonlines. And the map will transform each line with the json function.

$ <command that outputs json> | json <path to an array of objects> | map json <path in each object>

In the above command, the json command narrows data down to an array of objects, and then map json transforms each json into something. I think the path you specify could be some kind of template so that you can transform json into anything, which sounds a bit too ambitious.

sharkdp commented 5 years ago

Do you know jq? I think this could probably play well in combination with shell-functools.

Jake6329 commented 5 years ago

@sharkdp jq seems to cover my use cases. Cool, thanks.

sharkdp commented 5 years ago

More functions that should be implemented:

[ ] div/divide
[ ] sum/product (working on arrays)
[ ] matches => similar to starts_width/ends_with

jdhedden commented 2 years ago

Offering

For README.md:

### Usage of `lambda`

The `lambda` *function* allows you to create custom functions using a
definition string to Python's *lambda* function.
``` bash
> seq 3 | map lambda 'x : x+x'
2
4
6

> echo -e 'a:b\n2:4' | map split ':' | map lambda 'x, y : x+y'
ab
6

> seq -3 3 | map lambda 'x : abs(x)'
3
2
1
0
1
2
3

> echo -e "uppercase\nLOWERCASE\ncAMELcASE" | map lambda "x : x.swapcase()"
UPPERCASE
lowercase
CamelCase

For ft/ft/functions.py:

@register("lambda")
@typed(None, None)
def ft_lambda(definition, inp):
    _lambda = 'lambda ' + definition.value
    if type(inp) == list:
        args = list(map(lambda v: v.value, inp))
        return [TypedValue(str((eval(_lambda))(*args)), T_STRING)]
    else:
        return (eval(_lambda))(inp)

jdhedden commented 2 years ago

Offering

For README.md:

### Usage of `shell`

The `shell` *function* allows you to create custom functions using arbitrary
*shell* commands.  If the command string contains pairs curly braces `{}`, the
arguments to the function are formatted into it, otherwise they are appended
to it.
``` bash
> /usr/bin/ls -1 ft/ft/*.py | map shell 'grep -H -c def' | map split ':' | map format '{} has {} functions'
ft/ft/command.py has 14 functions
ft/ft/error.py has 1 functions
ft/ft/functions.py has 60 functions
ft/ft/__init__.py has 0 functions
ft/ft/internal.py has 4 functions
ft/ft/termcolor.py has 2 functions
ft/ft/test_command.py has 5 functions
ft/ft/types.py has 21 functions
ft/ft/version.py has 0 functions

> echo -e 'a\tb\n2\t4' | map shell 'echo "{} and {}"'
a and b
2 and 4

Total lines of text

find | filter is_file | map shell 'cat {} | wc -l' | foldl1 add
5040

For ft/ft/functions.py:

@register("shell")
@typed(None, None)
def shell(command, inp):
    if type(inp) == list:
        args = list(map(lambda v: str(v.value), inp))
    else:
        args = [str(inp)]

    command = command.value
    if '{}' in command:
        command = command.format(*args)
    else:
        command += ' '+' '.join(args)

    result = os.popen(command).read().rstrip('\r\n')

    if type(inp) == list:
        return [TypedValue(result, T_STRING)]
    else:
        return result

jdhedden commented 2 years ago

Offering

For README.md:

### Usage of `regex`

The `regex` *function* permits filtering based on Python supported regular
expressions.  For case-insensitive filtering, use the `(?i:...)` syntax:
``` bash
> /usr/bin/ls | filter regex '(?i:[a-c]+)'
LICENSE
make
path
README.md

For ft/ft/functions.py:

import re

@register("regex")
@typed(T_STRING, T_BOOL)
def contains(pattern, inp):
    pattern = dynamic_cast(T_STRING, pattern).value
    return re.search(pattern, inp)

jdhedden commented 2 years ago

Offering of new command `array` to create or flatten arrays

For README.md:

### Usage of `array`

The `array` command is used to take *lines of input*, and convert them into an
`array`.
``` bash
> seq 3 | array
1   2   3

Given an numerical argument, array will make arrays with that number of columns:

> seq 12 | array 5
1   2   3   4   5
6   7   8   9   10
11  12

Given an argument of '0' (or a non-numerical argument), array will convert arrays back into lines of input:

> echo -e 'aa\tbb\tcc\tdd' | array 0
aa
bb
cc
dd

For ft/array (new):

#!/usr/bin/bash

if [[ -z $1 || $1 -gt 0 ]]; then
    if [[ -n $1 ]]; then
        max=$1
    else
        max=1000000000000000
    fi

    cnt=0
    while read; do
        if [[ $cnt -gt 0 ]]; then
            result+=$'\t'
        fi
        result+=$REPLY

        (( cnt++ ))
        if [[ $cnt -eq $max ]]; then
            echo "$result"
            result=
            cnt=0
        fi
    done

    # Remaining partial line
    if [[ -n $result ]]; then
        echo "$result"
    fi

else
    # array 0
    sed 's@\t@\n@g'
fi

# EOF

goyalyashpal commented 9 months ago

hi! can u expand on that? like what would these do or what would be their input be like.

[ ] div/divide

[ ] sum/product (working on arrays) - @ sharkdp at https://github.com/sharkdp/shell-functools/issues/7#issuecomment-493706062

sharkdp / shell-functools