Bash vars

[saleyn]

Extend support for advanced variable replacement in bash

This update extends PR #427 by supporting variable replacement syntax of bash to be used in vm.args and sys.config. If the current shell is bash, then variables can contain things like ${NAME,,}, ${NAME^^}, ${NAME:-${DEF_NAME}}, etc.

The changes are backward-compatible with non-bash shells.

[lrascao]

i guess this should now be rebased onto master without 7cef84e9b80843c6bd63753a6d2eb4e2a3310805 in it

[lrascao]

any ideas on a test case for this functionality?

[saleyn]

E.g. create a vm.args with:

-sname ${NON_EXISTING_VAR:-test}

Then when a node is started, ensure it's equal to test@....

[lrascao]

I've written a test case for this, please squash commit 07a0fe5e7ad1219141e143b653c65a84646170a1 onto your own, the test right now is failing, from what i could gather it's related with the awk version running on Ubuntu 12.04 (which is what Travis is using), could you please take a look? ideally this should run in the most number of awk versions possible

[saleyn]

For some reason git pull origin master or git fetch origin 07a0fe5:refs/remotes/origin/lrescao-commit don't pull/fetch that commit. What's the way of fetching it?

[lrascao]

i think you must first add my remote (git remote add lrascao https://github.com/lrascao/relx.git) then do a fetch and then you'd be able to cherry-pick the commit

[saleyn]

Still can't find it:

$ git remote -v
lrescao https://github.com/lrascao/relx.git (fetch)
lrescao https://github.com/lrascao/relx.git (push)
origin  https://github.com/erlware/relx.git (fetch)
origin  https://github.com/erlware/relx.git (push)
upstream        git@github.com:saleyn/relx.git (fetch)
upstream        git@github.com:saleyn/relx.git (push)
$ git fetch lrescao 07a0fe:refs/remotes/origin/lrescao-commit
fatal: Couldn't find remote ref 07a0fe

[lrascao]

that's weird, what if you do the cherry-pick? same thing?

[saleyn]

fatal: bad revision '07a0fe'

[lrascao]

i guess one other way is to clone the repo and then use git format-patch to extract the commit and git am to apply it

[lrascao]

Just pushed a commit to your branch, forgot there's this new feature on github that allows to do that

[saleyn]

Interesting. I thought you need to be a member of the project in order to commit to the owner's branch. How do you do that?

Your test cases pass on ArchLinux:

$ rebar3 ct
===> Verifying dependencies...
===> Compiling relx
===> Running Common Test suites...
...
%%% rlx_extended_bin_SUITE ==> remote_console: OK
%%% rlx_extended_bin_SUITE ==> replace_os_vars: OK
%%% rlx_extended_bin_SUITE ==> replace_os_vars_custom_location: OK
%%% rlx_extended_bin_SUITE ==> replace_os_vars_dev_mode: OK
%%% rlx_extended_bin_SUITE ==> replace_os_vars_twice: OK
%%% rlx_extended_bin_SUITE ==> replace_os_bash_vars: OK
...
All 61 tests passed.

I looked briefly at the test cases, but understanding how you set up the environment requires a bit more time than I currently have. Possibly I'll be able to examine it more thoroughly the following week. Meanwhile you can possibly setup Travis to evaluate a simple script calling awk with:

        awk '{
            while(match($0,/\$[{(][^{}()]+?[})]/)) {
                s=RSTART; n=RLENGTH;
                v0=substr($0,0,s-1); v1=substr($0,s,n); v2=substr($0,n+s,length($0)-s-n+1)
                "echo "v1|getline v; $0=sprintf("%s%s%s", v0, v, v2)
            }
        }1' < "$1" > "$2"

and possibly replacing awk with gawk and checking if that makes a difference on Debian.

Another idea is that perhaps when echo ... | getline... gets evaluated it starts the default shell which on Debian is not bash (i.e. dash). Hence, maybe we need to change that to "bash -c echo "v1|getline v", or something in that vein.

[lrascao]

Interesting. I thought you need to be a member of the project in order to commit to the owner's branch. How do you do that?

It's this new github feature, when you open a pr there's a checkbox that allows commits from maintainers

and possibly replacing awk with gawk and checking if that makes a difference on Debian. Another idea is that perhaps when echo ... | getline... gets evaluated it starts the default shell which on Debian is not bash (i.e. dash). Hence, maybe we need to change that to "bash -c echo "v1|getline v", or something in that vein.

tried both ideas, same result, i can repro it on a vps debian machine, unfortunately my awk is not very good but it does seem related with that getline invocation you mentioned

[saleyn]

Can you check if the following code works in your environment? I have a Ubuntu VPS install with dash, and it seems to produce correct result:

$ uname -a
Linux ubu 4.4.0-31-generic #50-Ubuntu SMP Wed Jul 13 00:07:12 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux
$ ls -al /bin/sh
lrwxrwxrwx 1 root root 9 Nov 22 18:58 /bin/sh -> /bin/dash
$ echo $SHELL
/bin/sh
$ cat test.txt
Text=${A:-default}
$ cat t.sh
#!/bin/dash
echo "Shell=$SHELL"

/usr/bin/awk '{
    while(match($0,/\$[{(][^{}()]+?[})]/)) {
        s=RSTART; n=RLENGTH;
        v0=substr($0,0,s-1); v1=substr($0,s,n); v2=substr($0,n+s,length($0)-s-n+1)
        vv=sprintf("/bin/bash -c \"/bin/echo %s\"", v1)
        vv|getline v; $0=sprintf("%s%s%s", v0, v, v2)
    }
}1' < $1
$ export A=123
$ ./t.sh test.txt
Shell=/bin/sh
Text=123

[lrascao]

that seems to work, to get it right i added a space in the last sprintf

        awk '{
            while(match($0,/\$[{(][^{}()]+?[})]/)) {
                s=RSTART; n=RLENGTH;
                v0=substr($0,0,s-1); v1=substr($0,s,n); v2=substr($0,n+s,length($0)-s-n+1)
                vv=sprintf("/bin/bash -c \"/bin/echo %s\"", v1)
                vv|getline v; $0=sprintf("%s %s%s", v0, v, v2)
            }
        }1' < $1

[saleyn]

Great. Updated the branch

[lrascao]

closing via #586