SCX installation fails on Ubuntu 18.04.4

[Hermann99]

Hi

The installation of SCX fails on Ubuntu 18.04.4

azure# dpkg -i ./scx-1.6.4-7.ssl_110.universal.x64.deb (Reading database ... 297782 files and directories currently installed.) Preparing to unpack .../scx-1.6.4-7.ssl_110.universal.x64.deb ... Unpacking scx (1.6.4.7) over (1.6.4.7) ... Trying to stop omi with systemctl omi is stopped. Trying to start omi with systemctl omi is started. Setting up scx (1.6.4.7) ... /var/lib/dpkg/info/scx.postinst: 26: /usr/share/bash-completion/bash_completion: Syntax error: "(" unexpected dpkg: error processing package scx (--install): installed scx package post-installation script subprocess returned error exit status 2 Errors were encountered while processing: scx

[JumpingYang001]

@Hermann99 it should be error here: https://github.com/microsoft/SCXcore/blob/dacfe2f69ed1f7ed131ee0496750146e52dafef8/installer/datafiles/Base_SCXCore.data#L249

Fix solution: Do some fixes on /etc/profile to let [ -f /etc/profile ] && . /etc/profile to run pass, then scx will install pass.

[MG92]

Can you explain what you mean by 'some fixes on /etc/profile'? I'm having the same problem

[JumpingYang001]

@MG92 can you share me your [ -f /etc/profile ] && . /etc/profile output?

[MG92]

It doesn't output anything, it just activates the default conda environment

[JumpingYang001]

it seems conda conflicts with scx install script for ( set +e; [ -f /etc/profile ] && . /etc/profile; set -e; ${{SCX_SSL_CONFIG}} ) , we have setup conda on our local but not reproduce the issue.

[JumpingYang001]

@MG92 can you share us your OS version and conda version? so we can have a try to repro it on local. If you have any other products version required to repro the issue, please also provide, thanks.

[grwann]

Hi @JumpingYang001

I'm also running into the same issue where it seems like conda conflicts with scx install script. I was able to mitigate the issue manually by commenting out conda activate py37_default in the /etc/profile.d/conda_env.sh file, but I'd like this to work automatically since many of our customers use Data Science Ubuntu 18.04 image and this error is preventing the use of a common feature our customers would like to use. Not sure if I also have to reach out to conda team, but I figured I'd add details here:

Image: Data Science Ubuntu 18.04 Conda version: conda 4.8.3

[JumpingYang001]

@grwann , thanks for sharing it! we are trying to get a Data Science Ubuntu 18.04 vm on Azure now, will investigate the issue on it.

[JumpingYang001]

conda + data science libraries will append (py37_default) before root@test-data-sci-ubuntu:~#. Also I tried call . /etc/profile in shell script, it just repro the issue, so I think it is a problem of the Azure image, and not scx issue, you can check below tests.

It doesn't make sense to fix on scx, as scx use shell script to call . /etc/profile, and the problem is on the conda environment with data science libraries.

Mitigate steps:

echo ''>/etc/profile.d/conda_env.sh
sh omsagent-1.13.9-0.universal.x64.sh --install
echo 'conda activate py37_default'>/etc/profile.d/conda_env.sh

Some repro tests:

(py37_default) root@test-data-sci-ubuntu:~# ./test2.sh
./test2.sh: 5: /anaconda/envs/py37_default/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected
(py37_default) root@test-data-sci-ubuntu:~# cat test2.sh
#!/bin/sh

test() {
    [ -f /etc/profile ] && . /etc/profile;
}
test
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~# vi test2.sh
(py37_default) root@test-data-sci-ubuntu:~# cat test2.sh
#!/bin/sh

[ -f /etc/profile ] && . /etc/profile;
(py37_default) root@test-data-sci-ubuntu:~# ./test2.sh
./test2.sh: 5: /anaconda/envs/py37_default/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~#
(py37_default) root@test-data-sci-ubuntu:~# vi test2.sh
(py37_default) root@test-data-sci-ubuntu:~# cat test2.sh
#!/bin/sh

. /etc/profile;
(py37_default) root@test-data-sci-ubuntu:~# ./test2.sh
./test2.sh: 5: /anaconda/envs/py37_default/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh: Syntax error: "(" unexpected
(py37_default) root@test-data-sci-ubuntu:~#

[JumpingYang001]

that say the fixes should be on the Azure image, and not scx side, thanks.

[grwann]

@JumpingYang001 thank you for looking into it!

[JumpingYang001]

Hi @grwann , another better solution,

1.echo 'conda activate py37_default &'>/etc/profile.d/conda_env.sh
2.do anything others like oms installation/uninstallation.

above conda activate py37_default & command will make conda activate py37_default as daemon process. that can pass installation/usage for both conda and scx even it still show 1 same message inner omi restart script part of omsagent installation, I think.

[VariableDeclared]

WORKAROUND:

Edit /var/lib/dpkg/info/scx.postinst shbang to use #!/bin/bash rather than #!/bin/sh

[JumpingYang001]

@VariableDeclared do you know why replacing sh with bash fix the issue?

BTW, it is a variable for all Linux and Unix at present: https://github.com/microsoft/SCXcore/blob/6a1cdc0434ce67add62698a631ec531c7124168d/installer/datafiles/Base_SCXCore.data#L2 https://github.com/microsoft/SCXcore/blob/6a1cdc0434ce67add62698a631ec531c7124168d/installer/datafiles/Base_SCXCore.data#L339

[phvalguima]

Hi @JumpingYang001, this is a difference between distros, such as Centos and Ubuntu, for example.

What @VariableDeclared is referring is the fact that, in Ubuntu, /bin/sh will default to dash: $ ll /bin/sh lrwxrwxrwx 1 root root 4 <DATE> /bin/sh -> dash*

Whereas, Centos 7 for example, will default sh to bash: [centos@<hostname> ~]$ ll /bin/sh lrwxrwxrwx. 1 root root 4 <DATE> /bin/sh -> bash

Dash will give a different interpretation to "(" than bash. Given you are expecting bash to read your postinst script, I'd recommend, like @VariableDeclared did, to define it directly rather than using /bin/sh.

[JumpingYang001]

@phvalguima thanks for picking it up! I will check with team.

[JumpingYang001]

After doing some experience in an Azure vm, the root cause is conda calls function isn't dash compatible, and we cannot control conda or dash, I would suggest to use provided workaround echo 'conda activate py37_default &'>/etc/profile.d/conda_env.sh.

$ cat /etc/profile.d/conda_env.sh
conda activate py38_default
$ cat /anaconda/envs/py38_default/etc/conda/deactivate.d/deactivate-gxx_linux-64.sh
#!/bin/bash

# This function takes no arguments
# It tries to determine the name of this file in a programatic way.
function _get_sourced_filename() {
    if [ -n "${BASH_SOURCE[0]}" ]; then
        basename "${BASH_SOURCE[0]}"
    elif [ -n "${(%):-%x}" ]; then
        # in zsh use prompt-style expansion to introspect the same information
        # see http://stackoverflow.com/questions/9901210/bash-source0-equivalent-in-zsh
        basename "${(%):-%x}"
    else
        echo "UNKNOWN FILE"
    fi
}

sample root cause script:

(py38_default) azureuser@ubuntu-dsvm-1804:~$ dash
$ function a(){ echo test;};a
dash: 1: Syntax error: "(" unexpected
$
(py38_default) azureuser@ubuntu-dsvm-1804:~$ bash
azureuser@ubuntu-dsvm-1804:~$ function a(){ echo test;};a
test
azureuser@ubuntu-dsvm-1804:~$ dash ./b.sh
./b.sh: 3: ./b.sh: Syntax error: "(" unexpected
azureuser@ubuntu-dsvm-1804:~$ sh ./b.sh
./b.sh: 3: ./b.sh: Syntax error: "(" unexpected
azureuser@ubuntu-dsvm-1804:~$ cat b.sh
#!/bin/bash

function a(){ echo test;};a
azureuser@ubuntu-dsvm-1804:~$ bash ./b.sh
test
azureuser@ubuntu-dsvm-1804:~$

[george-andrei]

I had a similar issue but with Ubuntu 20.04 LTS:

Setting up scx (1.6.9.2) ...
dpkg: error processing package scx (--configure):
 installed scx package post-installation script subprocess returned error exit status 2
Errors were encountered while processing:
 scx
E: Sub-process /usr/bin/dpkg returned an error code (1)

I tried to upgrade to latest SCXcore, but had this:

Preparing to unpack scx-1.6.11-0.ssl_110.universal.x64.deb ...
Unpacking scx (1.6.11.0) over (1.6.9.2) ...
Trying to stop omi with systemctl
omi is stopped.
Trying to start omi with systemctl
omi is started.
Setting up scx (1.6.11.0) ...
dpkg: error processing package scx (--install):
 installed scx package post-installation script subprocess returned error exit status 2
Errors were encountered while processing:
 scx

In the end, thanks to @VariableDeclared workaround, I managed to fix the issue

WORKAROUND:

Edit /var/lib/dpkg/info/scx.postinst shbang to use #!/bin/bash rather than #!/bin/sh

[mohag]

The set -e and then sourcing the profile is likely the issue... It breaks with some bash-specific stuff in a script in /etc/profile.d for me...

Adding a -x to the shebang line in the postinst script gave me this:

+ [ -r /etc/profile.d/functions-and-aliases.sh ]
+ . /etc/profile.d/functions-and-aliases.sh
/var/lib/dpkg/info/scx.postinst: 21: /etc/profile.d/functions-and-aliases.sh: Syntax error: "(" unexpected (expecting "fi")

The lazy fix was to change the postinst script to use bash instead...

Scripts should likely not process /etc/profile... It is meant for login shells...