Syntax error reported in `deactivate.d/deactivate-gcc_linux-64.sh` when using tox-conda

[ctb]

tl;dr see https://github.com/ctb/2021-conda-tox-wat/ :).

Hi all,

thanks so much for conda-forge!!

I was pointed here in re this error output:

rm -fr .tox UNKNOWN.egg-info
tox -e py39
.package create: /home/ctbrown/2021-conda-tox-wat/.tox/.package
.package installdeps: setuptools >= 48
ERROR: invocation failed (exit code 2), logfile: /home/ctbrown/2021-conda-tox-wat/.tox/.package/log/.package-1.log
================================== log start ===================================
/tmp/tmpp10ke8je: 146: /home/ctbrown/miniconda3/envs/failme2/etc/conda/deactivate.d/deactivate-gcc_linux-64.sh: Syntax error: "(" unexpected (expecting "fi")

=================================== log end ====================================
ERROR: could not install deps [setuptools >= 48]; v = InvocationError("/home/ctbrown/2021-conda-tox-wat/.tox/.package/bin/python -m pip install 'setuptools >= 48'", 2)
py39 create: /home/ctbrown/2021-conda-tox-wat/.tox/py39
py39 inst: /home/ctbrown/2021-conda-tox-wat/.tox/.tmp/package/1/UNKNOWN-0.0.0.tar.gz
py39 installed: UNKNOWN @ file:///home/ctbrown/2021-conda-tox-wat/.tox/.tmp/package/1/UNKNOWN-0.0.0.tar.gz
py39 run-test-pre: PYTHONHASHSEED='2953931048'
___________________________________ summary ____________________________________
  py39: commands succeeded
  congratulations :)

which is generated by the github repo https://github.com/ctb/2021-conda-tox-wat/.

I first ran across this in the sourmash project; see https://github.com/sourmash-bio/sourmash/issues/1778. It looks similar to https://github.com/conda/conda/issues/9959. 2021-conda-tox-wat is my simplest-possible recapitulation of the sourmash problem.

---

It looks very much like the problem is somewhere at the intersection of tox and tox-conda. The code works fine if I omit tox-conda from the conda environment, in which case tox uses pip instead of conda environments. But I don't yet know where the problem lies specifically - with tox, or with the conda package.

If I had to guess, it's that there is some variable that is present in the environment that's causing a syntax error in the deactivate script, but there are too many moving parts for me to isolate it just yet.

My next debugging steps will be:

• use set -x to see exactly what's happening in the deactivation script
• figure out how to get gory details out of tox in terms of exactly what commands are being run

and I will update here if I get anywhere with them.

Any help is much appreciated! This is not a showstopper but it sure would be nice to get the fly out of the ointment :)

[isuruf]

cc @ktlim, @beckermr

[beckermr]

Thanks all!

I cannot reproduce this in a container locally in a variety of scenarios. So the example you sent is not enough to reproduce this and I think it is something local to your system.

Can you manually edit the deactivate file above to add set -x to the top and send any extra output?

Also what shell are you using?

[ctb]

thanks @beckermr! Note that it occurs both on my local Linux system and in GitHub actions (in sourmash repo), which is why I'm reasonably confident it's not just me, and it's probably not dependent on my local conda install.

I'll do the set -x in a bit.

Shell is bash.

[beckermr]

I am a tox n00b. What am I doing wrong here?

(failme) [conda@b258f61abe32 2021-conda-tox-wat]$ conda info

     active environment : failme
    active env location : /opt/conda/envs/failme
            shell level : 2
       user config file : /home/conda/.condarc
 populated config files : /home/conda/.condarc
          conda version : 4.11.0
    conda-build version : 3.21.7
         python version : 3.8.12.final.0
       virtual packages : __linux=5.10.76=0
                          __glibc=2.17=0
                          __unix=0=0
                          __archspec=1=x86_64
       base environment : /opt/conda  (writable)
      conda av data dir : /opt/conda/etc/conda
  conda av metadata url : None
           channel URLs : https://conda.anaconda.org/conda-forge/linux-64
                          https://conda.anaconda.org/conda-forge/noarch
                          https://repo.anaconda.com/pkgs/main/linux-64
                          https://repo.anaconda.com/pkgs/main/noarch
                          https://repo.anaconda.com/pkgs/r/linux-64
                          https://repo.anaconda.com/pkgs/r/noarch
          package cache : /opt/conda/pkgs
                          /home/conda/.conda/pkgs
       envs directories : /opt/conda/envs
                          /home/conda/.conda/envs
               platform : linux-64
             user-agent : conda/4.11.0 requests/2.26.0 CPython/3.8.12 Linux/5.10.76-linuxkit centos/7.9.2009 glibc/2.17
                UID:GID : 9001:9001
             netrc file : None
           offline mode : False

(failme) [conda@b258f61abe32 2021-conda-tox-wat]$ mamba list --explicit | grep gxx
https://conda.anaconda.org/conda-forge/linux-64/gxx_impl_linux-64-9.4.0-h03d3576_11.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx-9.4.0-h192d537_3.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx_linux-64-9.4.0-h0316aca_3.tar.bz2
(failme) [conda@b258f61abe32 2021-conda-tox-wat]$ rm -rf UNKNOWN.egg-info .tox
(failme) [conda@b258f61abe32 2021-conda-tox-wat]$ tox -e py39
.package create: /home/conda/2021-conda-tox-wat/.tox/.package
.package installdeps: setuptools >= 48
py39 create: /home/conda/2021-conda-tox-wat/.tox/py39
py39 inst: /home/conda/2021-conda-tox-wat/.tox/.tmp/package/1/UNKNOWN-0.0.0.tar.gz
py39 installed: UNKNOWN @ file:///home/conda/2021-conda-tox-wat/.tox/.tmp/package/1/UNKNOWN-0.0.0.tar.gz
py39 run-test-pre: PYTHONHASHSEED='2624061568'
________________________________________________________________________________ summary _________________________________________________________________________________
  py39: commands succeeded
  congratulations :)

[ctb]

(here's a link to one of the failing GitHub Action reports)

[ctb]

I am also not a tox expert, but I am reasonably Python knowledgeable and I'm 99% sure it's just creating the environment, activating it, and then running pip install :).

I check you -- same actual gxx packages installed:

% mamba list --explicit | grep gxx
https://conda.anaconda.org/conda-forge/linux-64/gxx_impl_linux-64-9.4.0-h03d3576
_11.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx-9.4.0-h192d537_3.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx_linux-64-9.4.0-h0316aca_3.ta
r.bz2

Here's the full output of tox -e py39 with set -x added at the top of deactivate script; as well as conda info. Thanks again! tox-fail.txt conda-info.txt

[ktlim]

This really does look like the error you get when trying to source the deactivate script under dash:

/usr/bin/dash: 146: ./deactivate-gcc.sh: Syntax error: "(" unexpected (expecting "fi")

[ktlim]

Is /tmp/tmpp10ke8je an explicitly sh (or at least non-bash) shell script?

[beckermr]

I thought the POSIX stuff was fixed? I guess we need better tests.

Ahh maybe we need this PR: https://github.com/conda-forge/ctng-compiler-activation-feedstock/pull/41

[ktlim]

That PR is great, but it still doesn't solve this problem AFAICT. dash won't execute the commands because of the zsh-only if, but it still needs to read them to skip them, and their array syntax is strict-POSIX non-compliant.

[ctb]

I would be happy to test/run #41 on my system - would it be as simple as copying recipe/deactivate-gcc.sh over my deactivate script? "No" is a fine answer 😆

I attach the output of tox -e py39 -vvv, which seems to be the only way to get lots of tox output. Not sure it'll help; I'm lost in a soup of twisty command line execution messages.

tox-fail-vv.txt

[beckermr]

hmmmm - maybe we need to generate the commands as a string and then eval them?

This should be safe I think?

[beckermr]

My local sh interpreter seems to understand the ( and so is not generating an error. Maybe I need to be in an ubuntu container with dash to see it.

[ktlim]

Yes, string plus eval is the way I was thinking, and it does look like it will work. Unfortunately it then reveals another problem in line 8 (with #41). I think that needs to be:

    elif [ -n "$ZSH_NAME" ] && [ -n "${(%):-%x}" ]; then

[isuruf]

@ctb, did this work before 9 days ago?

[ctb]

oooh, excellent question. No, 15 days ago was the first time we noticed it - https://github.com/sourmash-bio/sourmash/issues/1772#issuecomment-998020418. So that's ~December 20th, 2021. We did not have a lot of activity before that, so the last time I can clearly see that it was working was Nov 21, 2021 (see https://github.com/sourmash-bio/sourmash/pull/1765).

Digging a bit more, it looks like the last successful action run was maybe 17 days ago (December 16th? 17th?) link.

So, sometime between Dec 16th and Dec 20th was where any correlated package changes might have occurred...

[beckermr]

OK. I think the changes in #61 will do the trick. Does anyone have a local container with dash or otherwise where they can build this PR and check it out? I can try later but don't have the time this moment.

[beckermr]

I just merged #61. Give it a few hours for the builds to show up on the CDN and then try again.

[ctb]

preliminary results suggest that it works 🎉 - confirming with github actions now. Thank you VERY MUCH!

[ctb]

confirmed - github actions worked as well.

for posterity, this is what's working --

% mamba list --explicit | grep gxx
https://conda.anaconda.org/conda-forge/linux-64/gxx_impl_linux-64-9.4.0-h03d3576_11.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx-9.4.0-h192d537_4.tar.bz2
https://conda.anaconda.org/conda-forge/linux-64/gxx_linux-64-9.4.0-h0316aca_4.tar.bz2

so the _4 update for gxx_linux-64 and gxx-9.4.0 seems to be the key one.

Again, thanks very much for jumping on this, even when you couldn't replicate it yourselves! I'll close this now.