ImportError on jplm test

[jirivrany]

Hello, when I try to run all the test locally using

jlpm run build:test
jlpm test

the run fails with ImportError

jlpm test
yarn run v1.6.0
$ lerna run test:firefox --scope "@jupyterlab/test-*" --concurrency 1 --stream
lerna info version 2.9.0
lerna info versioning independent
lerna info scope @jupyterlab/test-*
@jupyterlab/test-application: $ python run-test.py --browsers=Firefox karma.conf.js
@jupyterlab/test-application: Traceback (most recent call last):\n  File "run-test.py", line 5, in <module>\n    from jupyterlab.tests.test_app import run_karma
@jupyterlab/test-application: ImportError: No module named jupyterlab.tests.test_app
@jupyterlab/test-application: info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
@jupyterlab/test-application: error Command failed with exit code 1.
lerna ERR! execute callback with error
lerna ERR! Error: Command failed: yarn run test:firefox
lerna ERR! Traceback (most recent call last):
lerna ERR!   File "run-test.py", line 5, in <module>
lerna ERR!     from jupyterlab.tests.test_app import run_karma
lerna ERR! ImportError: No module named jupyterlab.tests.test_app
lerna ERR! error Command failed with exit code 1.
lerna ERR! 
lerna ERR! $ python run-test.py --browsers=Firefox karma.conf.js
lerna ERR! info Visit https://yarnpkg.com/en/docs/cli/run for documentation about this command.
lerna ERR! 
lerna ERR!     at Promise.all.then.arr (/home/albert/work/python/jupyterlab/node_modules/lerna/node_modules/execa/index.js:236:11)
lerna ERR!     at <anonymous>
{ Error: Command failed: yarn run test:firefox
Traceback (most recent call last):
  File "run-test.py", line 5, in <module>
    from jupyterlab.tests.test_app import run_karma
ImportError: No module named jupyterlab.tests.test_app
error Command failed with exit code 1.

Specific tests eg. test-console are passing without any issue. What I'm missing?

[vidartf]

Did you perform a develop install? pip install -e .

[jirivrany]

Yes, I did everything as written in the docs. Even the optional builds of assets.

python --version
Python 3.6.5 :: Anaconda custom (64-bit)
jupyter notebook --version
5.5.0

[vidartf]

Does this command give you what you would expect?

python -c "import jupyterlab.tests.test_app as ta; print(ta.__file__)"

[jirivrany]

Yes, it gives the name of the file /home/albert/work/python/jupyterlab/jupyterlab/tests/test_app.py.

[vidartf]

If you are not on Windows, you could try

npm install -g npx
npx --no-install -c 'python -c "import jupyterlab.tests.test_app as ta; print(ta.__file__)"'

This will try to run the same command as through an npm run command. Maybe something strange is happening to the env?

[jirivrany]

Thanks, but this did not help. The result is same as if it runs as a standalone command. I'm using Ubuntu 16.04 LTS. I also did a try with a clean conda enviroment, but there is still the error. I will try to clone the repository once again and rebuild everything.

[vidartf]

Hmm. You can try to open up tests/test-application/tun-test.py, and at the very start add some debugging lines, e.g.:

import sys
import os
from pprint import pprint
print(sys.executable)
pprint(os.environ)

[jirivrany]

Thanks. The difference is in the executable. When I run jlpm test in the root the system /usr/bin/python/ but when I run it from the directory tests/test-application it uses my default interpret correctly /home/albert/anaconda3/bin/python. This is also a result of

$ which python
/home/albert/anaconda3/bin/python

Now how to tell lerna to use my default not the system one?

[jirivrany]

OK with the /usr/bin/python symlinked to my default conda enviroment everything works as expected. Thanks again @vidartf for help.

[vidartf]

Glad it was resolved for you! For future reference, and to see if there is an underlying problem, was the PATH variable from pprint(os.environ) as you would expect, or was this also the system default?

[jirivrany]

Maybe there is some problem in the path. It seems that Lerna grabs the first python in the search path, which is the python in /usr/bin in case of run-test.py

As you can see in the following output of print(os.environ['PATH']) in the run-test.py, there is first /usr/bin and only after it is my default python path set by .bashrc file.

/home/albert/work/python/jupyterlab/tests/test-application/node_modules/.bin
/home/albert/.config/yarn/link/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/work/python/jupyterlab/tests/test-application/node_modules/.bin
/home/albert/.config/yarn/link/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/usr/libexec/lib/node_modules/npm/bin/node-gyp-bin
/usr/lib/node_modules/npm/bin/node-gyp-bin
/usr/bin/node_modules/npm/bin/node-gyp-bin
/home/albert/work/python/jupyterlab/tests/test-application/node_modules/.bin
/home/albert/work/python/jupyterlab/tests/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/work/python/node_modules/.bin
/home/albert/work/node_modules/.bin
/home/albert/node_modules/.bin
/home/node_modules/.bin
/node_modules/.bin
/usr/bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/.config/yarn/link/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/.config/yarn/link/node_modules/.bin
/home/albert/work/python/jupyterlab/node_modules/.bin
/home/albert/.yarn/node-gyp-bin
/usr/libexec/lib/node_modules/npm/bin/node-gyp-bin
/usr/lib/node_modules/npm/bin/node-gyp-bin
/usr/bin/node_modules/npm/bin/node-gyp-bin
/home/albert/anaconda3/bin
/home/albert/bin
/home/albert/.local/bin
/usr/local/sbin
/usr/local/bin
/usr/sbin
/usr/bin
/sbin
/bin
/usr/games
/usr/local/games
/snap/bin

My localpath is:

echo $PATH
/home/albert/anaconda3/bin:/home/albert/bin:/home/albert/.local/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games:/usr/local/games:/snap/bin

[vidartf]

Since the anaconda parts of the path does not get propagated, I'm assuming that it loses the correct environment variables somewhere along the line.

1. jlpm test
2. lerna run test:firefox --scope "@jupyterlab/test-*" --concurrency 1 --stream
3. python run-test.py --browsers=Firefox karma.conf.js

By some local tests, it seems as if yarn drops it at 1. Steps to reproduce:

• Add a new entry to "scripts" in root package.json: "check": "python -c \"import os; print(os.environ['PATH'])\""
• Add something non-standard to PATH.
• Run jlpm run check in repo root.

[vidartf]

Possibly related: https://github.com/yarnpkg/yarn/issues/5381

[vidartf]

Update: It actually works here, but from that link above it seems to be an issue on linux only?

[jirivrany]

Yes looks like Linux only problem. I'm using Ubuntu - currently 18.04 LTS. I reopen the issue as you suggest. Now I'm going to do some more tests to explore the behavior.

[vidartf]

Are you able to reproduce the issue on the yarn tracker? If you export another environment variable, does that get lost?

[jirivrany]

No, I think if I use the correct CASE then everything works as expected. I was not able to reproduce protractor issue because of missing spec files etc. But for Yarn, it is ok.

albert@sela:~/work/python/jupyter-work/yarn-bug$ export FOO=bar
albert@sela:~/work/python/jupyter-work/yarn-bug$ echo $FOO
bar
albert@sela:~/work/python/jupyter-work/yarn-bug$ yarn run foo
yarn run v1.7.0
$ node conf.js
foo =  undefined
FOO =  bar
Done in 0.10s.
albert@sela:~/work/python/jupyter-work/yarn-bug$ cat package.json 
{
  "name": "yarn-bug",
  "version": "1.0.0",
  "description": "reproduce a bug",
  "main": "index.js",
  "license": "MIT",
  "engines": {
    "node": "8.11.2",
    "yarn": ">= 1.0.0"
  },
  "scripts": {
    "foo": "node conf.js"
  }
}
albert@sela:~/work/python/jupyter-work/yarn-bug$ cat conf.js 
console.log('foo = ', process.env.foo);
console.log('FOO = ', process.env.FOO);

[jirivrany]

I did a test for protractor (used spec from protractor docs) and it passes. The enviroment variable did not dissapear.

conf.js

exports.config = {
  seleniumAddress: 'http://localhost:4444/wd/hub',
  onPrepare: () => console.log(process.env.FOO),
  specs: ['todo-spec.js']
};

package.json

{
  "name": "yarn-bug",
  "version": "1.0.0",
  "description": "reproduce a bug",
  "main": "index.js",
  "license": "MIT",
  "engines": {
    "node": "8.11.2",
    "yarn": ">= 1.0.0"
  },
  "scripts": {
    "foo": "node simple.js",
    "test": "protractor conf.js"
  },
  "dependencies": {
    "protractor": "^5.3.2"
  }
}

execution

albert@sela:~/work/python/jupyter-work/yarn-bug$ yarn run test
yarn run v1.7.0
$ protractor conf.js
(node:6069) [DEP0022] DeprecationWarning: os.tmpDir() is deprecated. Use os.tmpdir() instead.
[18:45:49] I/launcher - Running 1 instances of WebDriver
[18:45:49] I/hosted - Using the selenium server at http://localhost:4444/wd/hub
bar
Started
.

1 spec, 0 failures
Finished in 3.561 seconds

[18:45:54] I/launcher - 0 instance(s) of WebDriver still running
[18:45:54] I/launcher - chrome #01 passed
Done in 4.96s.