Support vectorized numpy operations

[talwrii]

Example

seq 10 | pyline -N 'd**2'
seq 100 | pyline -N 'sum(d)'
seq 100 | pyline -N 'sum(d)'

This will likely be contentious:

• numpy dependecy, interaction of numpy with virtualenvs, testing if we decide to only import numpy when used
• Different mode of functionality: data has to be read ahead of time to work with numpy, this is potentially confusing since the behaviour is different with and without the numpy option
• To be useable one might end up adding magic (special cases for different types of data), this may offend your sensibilties.

That said, all things being equal, the world might only want one command line tool to quickly run python from the command line.

[talwrii]

See https://github.com/talwrii/pyline/tree/talwrii--numpy--2016-09-21--WIP--issue-27 for a reference implementation.

I wanted these features to exist, so I coded them; if this is deemed an ill fit for pyline then I can pull this into a separate tool without too much work.

Note that this is a work in progress. There is an unresolved question of the numpy dependency:

• Are we happy for this tool to have a numpy dependency?
• If not are we happy to have certain functionality to have a numpy dependency? If so how should tests work.

Opinions?

[westurner]

Are we happy for this tool to have a numpy dependency?

You could add it as an extras_require in setup.py; though I think expecting users to have numpy installed is a fair assumption.

If not are we happy to have certain functionality to have a numpy dependency? If so how should tests work.

try:
    import numpy as np
except ImportError:
    np = None

# ...

skipif(np is None, "NumPy is not installed")

• unittest has skipif: https://docs.python.org/2/library/unittest.html#skipping-tests-and-expected-failures
• pytest has skipif: http://doc.pytest.org/en/latest/skipping.html#id1
• nose appears to not work w/ unittest.skipif: http://stackoverflow.com/questions/21936292/conditional-skip-testcase-decorator-in-nosetests

So, probably pytest

import pytest
@pytest.mark.skipif
@unittest.skipif(np is None,
                    reason= "NumPy is not installed")
def testfuncname():
    # ...

[westurner]

I'm sort of partial to using full imports in the command (for the sake of making it easier to copy and paste to a program), so, personally, I'd prefer e.g. np.sum() over just sum(); but there may be a justifiable reason to be sage-y?

[westurner]

(I am open to a (first!) PR)

[westurner]

I wanted these features to exist, so I coded them; if this is deemed an ill fit for pyline then I can pull this into a separate tool without too much work.

In terms of scope; IDK about other functions which consume the whole file

[westurner]

You could add it as an extras_require in setup.py; though I think expecting users to have numpy installed is a fair assumption.

If there's an ImportError in the pyline function, $ pyline should return a nonzero error code

[talwrii]

Cool cool. It seems like you're open to the idea of this feature.

Numpy dependency while running

No one's going to use the extra_require :) . Yep just erroring out if numpy is missing and --numpy is used seems reasonable. Perhaps together with instructions to install numpy.

Numpy dependency while testing

So my concern about skipping tests is that it's an invitation for tests to not be run when they need to be. There appears to be no way to automatically have different requirements for python setup.py test. test_requires would appear to exist solely to mislead! (http://stackoverflow.com/questions/9607565/how-do-i-force-setup-py-test-to-install-dependencies-into-my-virtualenv).

Some options:

• Just have the tests fail if numpy is not installed (people will be able to work out how to fix these easily enough)
• Do the above and add a special extra_requires for testing. (but people probably won't remember to use it)
• Suck it up and add a numpy dependency (But when you add a numpy dependency to a virtualenv numpy gets built from source, which takes ages...)

Continuous integration

There seems to be some sort of continuous integration (CI) for this, but a brief inspection of the source code failed to tell me how this worked. Is there anything I should bear in mind here?

What numpy things to put into the namespace

Some comments

• One could still refer to numpy.func even if we flood the namespace with the contents of numpy. But the argument is presumably to force / encourage to make a decision that results in more reusable code.
• I don't know how confusing the alias np is for people new to numpy, though it does seems standard in a number of examples
• I feel that long function names in formulas can make them difficult to read. Consider: numpy.sqrt(numpy.sum(d[,0]**2)) versus sqrt(sum(d[,0]**2)). But np.sqrt(np.sum(d[,0])) isn't too bad.

Opinions? I don't have strong opinions other writing out numpy.blah is a little wordy.

Actions

If you give me some judgement calls then I'll finish off my branch and give you a pull request.

P.S What does sage-y mean :) ? Do you mean like the computer algebra system or like the accountancy system. I don't use either extensively!

[talwrii]

In terms of scope; IDK about other functions which consume the whole file

More details please!

[westurner]

In terms of scope; IDK about other functions which consume the whole file

More details please!

As-is, pyline iterates through the input without reading the whole file into RAM (thus avoiding issues with memory consumption).

The proposed changes build a list for each line of the whole input file (consumes the whole file) and then copies those into a numpy array.

I'm hesitant to expand the scope of this utility; though I do recognize the usefulness of a --numpy option.

[westurner]

[CI]

• https://github.com/audreyr/cookiecutter-pypackage
• https://github.com/westurner/pyline/blob/master/.travis.yml
• https://github.com/westurner/pyline/blob/master/tox.ini

[long function names]

• import numpy; np = numpy
• from numpy import * could potentially cause namespace collision

numpy dependency

As-is, pyline requires zero dependencies. The pyline.py file can be copied to sys.path and runs fine.

There is obviously a tradeoff between {zero dependencies, fast-tests} and {third-party library features}.

[westurner]

• (TBH, I don't need this functionality)
• A fork/workalike would be cool
• Otherwise
  • The optparse help text for --numpy should carry a warning like NOTE: This method consumes the whole input file
  • travis.yml could run a different tox env which includes numpy
  • Because I need/want to be able to run the tests without installing numpy
    • (Though I recognize that conda install numpy is really easy)

[talwrii]

TBH, I don't need this functionality

Cool cool. You can have it if you want though :) (the "otherwise" approach seems reasonable).

Okay, unless you tell me otherwise I'm going to extract out a a tool called npcli which

• Runs like npcli sum(d), possibly doing magic things to work out whether it needs input.
• References this tool and includes you as a second author (feel free to request otherwise), but clearly not a maintainer. ( I wonder if this counts as endorsing according to the license)
• Uses the same BSD license

If it becomes apparent that other people want this feature then I imagine we can easily finish off this branch and merge.

[westurner]

Cool cool. You can have it if you want though :) (the "otherwise" approach seems reasonable).

:) cool

• Pyed piper may also be worth a look. It has some fancy/magical command parsing.
  • https://code.google.com/archive/p/pyp/
• ... https://github.com/westurner/pyline/issues/27#issuecomment-250038384
• [ ] there could be a requirements-numpy.txt (with a -r requirements.txt) here https://github.com/westurner/pyline/blob/master/requirements.txt https://github.com/westurner/pyline/blob/master/requirements-numpy.txt
  • [ ] install: pip install requirements-numpy.txt here https://github.com/westurner/pyline/blob/master/.travis.yml#L12

I suppose, since the tests already require third party packages, requiring numpy for tests and --numpy would be fine

[westurner]

AFAIK, there's still not yet a numpy/__main__.py (for python -m numpy or python -m numpy.cli.

Now that I've added other optional dependencies to pyline, it's unlikely that it would be accepted for inclusion as a numpy CLI module