search

n-riesco / ijavascript

IJavascript is a javascript kernel for the Jupyter notebook
Other
2.19k stars 185 forks source link

Set up working directory #4

Closed itkach closed 9 years ago

itkach commented 9 years ago

Importing one of Node's built-in modules, such as fs or http, works in a notebook. Importing a module installed via NPM (with or without -g) fails like this:

In [5]:

require('lodash');
module.js:324
    throw err;
          ^
Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:322:15)
    at Function.Module._load (module.js:264:25)
    at Module.require (module.js:351:17)
    at require (module.js:370:17)
    at evalmachine.<anonymous>:1:1
    at Object.exports.runInThisContext (vm.js:54:17)
    at run ([eval]:110:19)
    at onMessage ([eval]:57:41)
    at process.emit (events.js:100:17)
    at handleMessage (child_process.js:305:10)

iojs v1.1.0, ijavascript 4.1.4, ipython 3.0.0

n-riesco commented 9 years ago

I cannot reproduce this issue.

Please, note that neither node.js nor io.js allow the use of require with packages globally installed. See here.

I've noticed in your report in issue #5 that ijavascript is installed in folder /home/itkach/lib/node_modules/. My understanding is that node.js would look for local packages in folder /home/itkach/node_modules/. Please, could you check whether require("lodash") works in the REPL:

$ node
> require("lodash")
itkach commented 9 years ago

Yes, require in REPL works.

$ mkdir ~/test-ijs-require
$ cd ~/test-ijs-require
$ npm install lodash
lodash@3.3.1 node_modules/lodash
$ iojs                                            
> require('lodash')
{ [Function: lodash]
  support: 
   { funcDecomp: true,
...
~/test-ijs-require% ls -l `which iojs`
lrwxrwxrwx 1 itkach itkach 43 Feb  6 12:21 /home/itkach/bin/iojs -> /home/itkach/iojs-v1.1.0-linux-x64/bin/iojs

Starting ijavascript like this:

~/lib/node_modules/ijavascript/bin/ijavascript.js

or

cd ~/test-ijs-require
npm install ijavascript
./node_modules/ijavascript/bin/ijavascript.js

makes no difference, require('lodash') still fails

n-riesco commented 9 years ago

I'd like to reproduce the issue in my machine. Please, could you describe what steps you followed to setup io.js in you machine? And especially, what do I need to do to get npm to install local modules in ~/lib/node_modules rather than ~/node_modules.

itkach commented 9 years ago

what steps you followed to setup io.js

I downloaded distribution tarball from https://iojs.org/en/index.html (was 1.1.0 at that time), unpacked in my home directory

tar -xvf ~/Downloads/iojs-v1.1.0-linux-x64.tar.xz

and created symlinks for iojs, node and npm in ~/bin pointing to corresponding binaries in ~/iojs-v1.1.0-linux-x64/bin/

what do I need to do to get npm to install local modules in ~/lib/node_modules rather than ~/node_modules

I didn't do anything for this. I think none of the node versions I used tried putting modules into ~/node_mofules. An old version (v0.10.21) I had compiled from source configured with --prefix=$HOME/node-install was using ~/node-install/lib/node_modules/

n-riesco commented 9 years ago

Since require("lodash") works for you in the REPL, you could try the following as a temporary workaround:

n-riesco commented 9 years ago

I have followed the instructions in your post to install iojs-v1.1.0-linux-x64.tar.gz, and I can't reproduce the issue (all the local packages are installed in ~/node_modules).

This makes me think npm in your machine doesn't have the default configuration. Please, could you post the output of:

npm config ls -l | grep global

If global is set to true, then npm install lodash would install the package in ~/lib/node_modules/.

itkach commented 9 years ago

could you post the output

$ npm config ls -l | grep global                                                                                                                  git:[977c469626d9  master ]
global = false
globalconfig = "/home/itkach/etc/npmrc"
globalignorefile = "/home/itkach/etc/npmignore"
itkach commented 9 years ago

npm install installs modules in current directory, ./node_modules, npm install -g installs in ~/lib/node_modules

itkach commented 9 years ago

run require.resolve("lodash") in the REPL (I imagine it will return `/home/itkach/lib/node_modules/lodash')

It doesn't. 

$ cd ~/test-ijs-require
$ iojs -e "console.log(require.resolve('lodash'))"
/home/itkach/test-ijs-require/node_modules/lodash/index.js

If I'm not in a directory with locally installed module require cannot find module - which appears to be expected and is fine with me.

itkach commented 9 years ago

require("/home/itkach/test-ijs-require/node_modules/lodash/index.js") does work

n-riesco commented 9 years ago

By any chance, is there a package.json in folder /home/itkach/test-ijs-require?

itkach commented 9 years ago

It should've been:

require("/home/itkach/test-ijs-require/node_modules/lodash/")

that works too. But require("./node_modules/lodash/") doesn't. Basically, it looks like relative paths or module names that are supposed to be resolvable via NODE_PATH don't work, only absolute paths do.

itkach commented 9 years ago

By any chance, is there a package.json in folder /home/itkach/test-ijs-require?

no, there's no package.json, only node_modules produced by npm install lodash

itkach commented 9 years ago

btw, ipython runs in a virtualenv. I don't see how this can affect node module resolution, but thought I'd mention it

n-riesco commented 9 years ago

My understanding of npm is that:

whereas:

iJavascript only sees the packages local to itself and those local to the user (in your case folder /home/itkach/node_modules).

To install a package local to a user, I would expect the following should work:

cd ~ # To ensure we are not in a package folder
npm install lodash
itkach commented 9 years ago

iJavascript only sees the packages local to itself and those local to the user

That explains it, thank you.

I think it would be useful if iJavascript could find package local modules too. Matching node's module resoltion rules (http://nodejs.org/api/modules.html#modules_loading_from_node_modules_folders and http://nodejs.org/api/modules.html#modules_loading_from_the_global_folders) would be ideal.

n-riesco commented 9 years ago

I believe it already does. In fact, iJavascript doesn't change the definition of require. You should be able to reproduce the following results in your machine:

user@host:~$ npm uninstall lodash
npm WARN uninstall not installed in /home/user/node_modules: "lodash"
user@host:~$ npm uninstall -g lodash
npm WARN uninstall not installed in /home/user/lib/node_modules: "lodash"
user@host:~$ iojs -e "console.log(require('lodash').name)"
module.js:324
    throw err;
          ^
Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:322:15)
    at Function.Module._load (module.js:264:25)
    at Module.require (module.js:351:17)
    at require (module.js:370:17)
    at [eval]:1:13
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:446:26)
    at evalScript (node.js:413:25)
    at startup (node.js:72:7)
user@host:~$ npm install -g lodash
lodash@3.3.1 /home/user/lib/node_modules/lodash
user@host:~$ iojs -e "console.log(require('lodash').name)"
module.js:324
    throw err;
          ^
Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:322:15)
    at Function.Module._load (module.js:264:25)
    at Module.require (module.js:351:17)
    at require (module.js:370:17)
    at [eval]:1:13
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:446:26)
    at evalScript (node.js:413:25)
    at startup (node.js:72:7)
user@host:~$ npm install lodash
lodash@3.3.1 node_modules/lodash
user@host:~$ iojs -e "console.log(require('lodash').name)"
lodash

The results above show that packages globally installed can't be loaded using require.

itkach commented 9 years ago

iJavascript doesn't change the definition of require.

and yet package local modules and relative paths don't work, and it looks like NODE_PATH is also not honored, while node REPL is able to import all these modules, so something is different

n-riesco commented 9 years ago

I'm not sure I understand. Please, could you give me an example that "package local modules" and "relative paths" don't work?

Here's an example you can run to confirm NODE_PATH is honoured:

user@host:~$ npm uninstall lodash
unbuild lodash@3.3.1
user@host:~$ iojs -e "console.log(require('lodash').name)"
module.js:324
    throw err;
          ^
Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:322:15)
    at Function.Module._load (module.js:264:25)
    at Module.require (module.js:351:17)
    at require (module.js:370:17)
    at [eval]:1:13
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:446:26)
    at evalScript (node.js:413:25)
    at startup (node.js:72:7)
user@host:~$ npm install -g lodash
lodash@3.3.1 /home/user/lib/node_modules/lodash
user@host:~$ iojs -e "console.log(require('lodash').name)"
module.js:324
    throw err;
          ^
Error: Cannot find module 'lodash'
    at Function.Module._resolveFilename (module.js:322:15)
    at Function.Module._load (module.js:264:25)
    at Module.require (module.js:351:17)
    at require (module.js:370:17)
    at [eval]:1:13
    at Object.exports.runInThisContext (vm.js:54:17)
    at Object.<anonymous> ([eval]-wrapper:6:22)
    at Module._compile (module.js:446:26)
    at evalScript (node.js:413:25)
    at startup (node.js:72:7)
user@host:~$ NODE_PATH=/home/user/lib/node_modules ijs
[...]
itkach commented 9 years ago 
#Let's go $HOME
cd

# and simply start ijs
ijs
# require.resolve('./test-ijs-require/node_modules/lodash') in notebook works, as expected

# lets start with relative NODE_PATH
NODE_PATH=./test-ijs-require/node_modules ijs
#require.resolve('lodash') in notebook now works, as expected

#Now let's go to ~/test-ijs-require directory

cd ~/test-ijs-require
npm install lodash
#simply start ijs
ijs
# require.resolve("lodash") in ipython notebook doesn't work
# but works in node REPL, results in /home/itkach/test-ijs-require/node_modules/lodash/index.js
# same for require.resolve("./node_modules/lodash")

#start ijs with relative NODE_PATH

NODE_PATH=./node_modules ijs
# require.resolve("lodash") in ipython notebook doesn't work

# Works in REPL
NODE_PATH=./node_modules node -e "console.log(require.resolve('lodash'))"
/home/itkach/test-ijs-require/node_modules/lodash/index.js

#start ijs with absolute NODE_PATH
NODE_PATH=/home/itkach/test-ijs-require/node_modules ijs
# require.resolve("lodash") in ipython notebook works

#Let's create a custom module now (we are in ~/test-ijs-require)
mkdir src
echo "module.exports = {}" >! src/mymodule.js
#Let's see if node finds it:
node -e "console.log(require.resolve('./src/mymodule'))"
#yep, it's /home/itkach/test-ijs-require/src/mymodule.js
#Let's see if node finds it like this:
NODE_PATH=./src node -e "console.log(require.resolve('mymodule'))"
#yep, it's /home/itkach/test-ijs-require/src/mymodule.js
#Let's see if ijs finds it:
ijs
#require.resolve('./src/mymodule') in notebook gives us 'cannot find module' error
#Let's see if ijs finds it like this:
NODE_PATH=./src ijs
#require.resolve('mymodule') in notebook gives us 'cannot find module'
error
#What about absolute path?:
NODE_PATH=$PWD/src ijs
#yes, require.resolve('mymodule') works now
n-riesco commented 9 years ago

On 03/03/15 21:50, itkach wrote:

Now let's go to ~/test-ijs-require directory

cd ~/test-ijs-require npm install lodash

simply start ijs

ijs

require.resolve("lodash") in ipython notebook doesn't work

but works in node REPL, results in /home/itkach/test-ijs-require/node_modules/lodash/index.js

same for require.resolve("./node_modules/lodash")

OK, I got it now. It works in the REPL because the current working directory is ~/test-ijs-require.

At the moment, iJavascript's working directory is always set to the user's home folder.

Ideally, I would set iJavascript's working directory to the location of the notebook. But, in general, this is not possible, because the kernel may not even be running on the same machine.

I could add an option to ijs to set the working directory to something else. Something like:

$ ijs --ijs-working-dir=/path/to/folder

Please, let me know what you think.

start ijs with relative NODE_PATH

NODE_PATH=./node_modules ijs

require.resolve("lodash") in ipython notebook doesn't work

Works in REPL

NODE_PATH=./node_modules node -e "console.log(require.resolve('lodash'))" /home/itkach/test-ijs-require/node_modules/lodash/index.js

Same reason as above: . (i.e., the working directory) in iJavascript is set to the user's home folder.

Let's create a custom module now (we are in ~/test-ijs-require)

mkdir src echo "module.exports = {}" >! src/mymodule.js

Let's see if node finds it:

node -e "console.log(require.resolve('./src/mymodule'))"

yep, it's /home/itkach/test-ijs-require/src/mymodule.js

Let's see if node finds it like this:

NODE_PATH=./src node -e "console.log(require.resolve('mymodule'))"

yep, it's /home/itkach/test-ijs-require/src/mymodule.js

Let's see if ijs finds it:

ijs

require.resolve('./src/mymodule') in notebook gives us 'cannot find module' error

Let's see if ijs finds it like this:

NODE_PATH=./src ijs

require.resolve('mymodule') in notebook gives us 'cannot find module'

error

Same reason.

Thank you for all the examples (and your time).

Nico

itkach commented 9 years ago

option to ijs to set the working directory to something else

That would be useful. I think current directory is also a more useful default than user directory - that would result in the least surprising behavior out of the box, at least for "REPL replacement" use case.

n-riesco commented 9 years ago

On 03/03/15 23:43, itkach wrote:

option to |ijs| to set the working directory to something else

That would be useful. I think current directory is also a more useful default than user directory - that would result in the least surprising behavior out of the box, at least for "REPL replacement" use case.

I was concerned about using the current working directory and calling ijs from a script, but combining both ideas may actually serve both needs:

and

Again, thanks for bringing up this issue.

n-riesco commented 9 years ago

The last release (v4.1.5) already implements this change.

itkach commented 9 years ago

Nice, thank you :+1:

jkroso commented 9 years ago

At the moment, iJavascript's working directory is always set to the user's home folder.

Ideally, I would set iJavascript's working directory to the location of the notebook. But, in general, this is not possible, because the kernel may not even be running on the same machine.

Why not use process.cwd()

n-riesco commented 9 years ago

@jkroso Version 4.1.5 and later have exactly that behaviour (i.e. the kernel uses process.cwd() by default). In this way, as itkach pointed out, IJavascript could be used as an REPL replacement.

I also added the flag --ijs-working-dir, in case setting another working directory is needed. For example:

ijs --ijs-working-dir=/tmp

I'm not entirely happy with this flag, because I find confusing when the Jupyter dashboard and the IJavascript kernel run on different folders.

jkroso commented 9 years ago

I'm not sure what you mean. It doesn't seem use process.cwd() for me. I've edited this line in my version to get the behaviour I want. And process.cwd() isn't used anywhere in that file

n-riesco commented 9 years ago

How do you run the kernel?

For example, this is what I get:

$ cd /tmp
$ ijs
In [1]: process.cwd()
Out[1]: '/tmp'
jkroso commented 9 years ago

I run it through Jupyter. Which executes the kernel.js file directly

On Tue, Jun 30, 2015 at 9:15 PM, Nicolas Riesco notifications@github.com wrote:

How do you run the kernel? For example, this is what I get:

$ cd /tmp
$ ijs
In [1]: process.cwd()
Out[1]: '/tmp'

Reply to this email directly or view it on GitHub: https://github.com/n-riesco/ijavascript/issues/4#issuecomment-117068510

n-riesco commented 9 years ago

Note that the IJavascript kernel spec is regenerated every time ijs is run. Unfortunately this means that the kernel working folder is set to whatever the working directory was when ijs was run.

I've open issue #36 to track this bug.

$ cd /tmp
$ ijs
In [1]: process.cwd()
Out[1]: '/tmp'
$ mkdir /tmp/ijs
$ cd /tmp/ijs
$ ipython notebook
In [1]: process.cwd()
Out[1]: '/tmp'
jkroso commented 9 years ago

it should probably just be generated on npm install. Would you like me to make a PR for that issue or are you working on it?

n-riesco commented 9 years ago

See #36

fasiha commented 8 years ago

I was starting ijs from any old directory, not realizing it had to be an npm project directory with a package.json. I was seeing the error Error: ENOENT: no such file or directory, open 'package.json' but this thread helped me see that I needed to either be in an npm project directory or run npm -init -y in my ~/notebooks directory and start ijs from there.

This makes perfect sense in hindsight, but it’s not what I expected coming from Python (since they use global or centralized-user directories for dependencies), so might be worth putting on the README:

Start ijs from an npm project directory, or run npm -init -y in the directory you start ijs from.

n-riesco commented 8 years ago

@fasiha Could you open a new issue and post the steps to reproduce the error?

I'm guessing that what you'd like to see documented are the steps to ensure that require("some-npm-package") works. If that's the case, npm init won't be enough.