Results from running 20/predict.py

[mscappini]

Retrieving data 5036 records retrieved for equity 'dd' 5036 records retrieved for equity 'f' 5036 records retrieved for equity 'ge' 5036 records retrieved for equity 'ibm' 5036 records retrieved for equity 'jnj' 5036 records retrieved for equity 'jpm' 5036 records retrieved for equity 'xom' Traceback (most recent call last): File "D:\Source\Projects\investor-bot\predictors\linreg\20\predict.py", line 119, in results = run(_predrange_begin, _predrange_end) File "D:\Source\Projects\investor-bot\predictors\linreg\20\predict.py", line 98, in run _features, _labels = get_alldata(predrange_begin, predrange_end) File "D:\Source\Projects\investor-bot\predictors\linreg\20\predict.py", line 52, in get_alldata features, labels = data.aggregate(equities, startdate, enddate, featurefn, labelfn) File "../common\data.py", line 137, in aggregate _features = np.append(_features, _featdf.values, axis=0) File "C:\Python27\lib\site-packages\numpy\lib\function_base.py", line 3884, in append return concatenate((arr, values), axis=axis) MemoryError

[aisthesis]

~/Workspace/investor-bot/predictors/linreg/20$ python3 predict.py
Retrieving data
Verifying data integrity
Features and labels validated
Evaluating model
...........
~/Workspace/investor-bot/predictors/linreg/20$ python predict.py
Retrieving data
Verifying data integrity
Features and labels validated
Evaluating model
...........
$

We're probably going to have to go through and compare each package version since I can't repro the error. I was running these unfortunately before I had virtualenv installed, which allows you to do easy version comparisons. What versions of numpy, pandas and pynance are you running?

[mscappini]

Looks like some of my modules are more up to date. Latest pynance (doesn't export version attribute), pandas 0.16.0, numpy 1.9.2. I could try earlier versions, but maybe investor-bot should use the latest?

[aisthesis]

are you using python 3 or 2?

[mscappini]

Python 2.7.9.

[aisthesis]

It should use the latest, but what the latest is changes over time. When I started the linreg part, those versions didn't exist yet. For now, you might see if you have better results using numpy 1.9.1 and pandas 0.15.1. Those are the ones I just tested it on.

Better still: Set up a virtualenv with those versions and run the code in that virtualenv. That's the correct way to do it.

[aisthesis]

Also, with virtualenv and autoenv, you can put a .env and/or requirements.txt file at the root of a directory to specify which versions you were using at the time and automatically load a virtualenv with exactly those versions.

[mscappini]

I successfully set up a virtual environment with the versions required by investor-bot as stated in the readme, but I still get the error.

Would this be at all related? http://stackoverflow.com/questions/15815854/how-to-add-column-to-numpy-array

[aisthesis]

Judging from your error messages, it has something to do with appending. But the line where the error occurs doesn't append, although it does assign in place, and it works on my Mac, as the above output shows.

Could you try running the pynance unit tests on your machine?

[aisthesis]

In general, since Rob and I were both using Linux and there were complications associated with Windows support in various spots, we decided that we wouldn't support Windows. That being said, if you were able to get the repo installed, I don't see why this particular part would work on Mac/Linux but not on Windows. But that is a distinct possibility.

[mscappini]

Yes, the pynance tests run fine.

[aisthesis]

Just out of curiosity, is there a reason why "cat" isn't included in the equities in the line "{} records retrieved for equity '{}'"? https://github.com/aisthesis/investor-bot/blob/master/predictors/linreg/common/data.py#L135

[aisthesis]

The weird thing is that, given the output, the loop is working every time, even including 'xom', which should be the last equity tried.

[aisthesis]

This is where the list of equities comes from: https://github.com/aisthesis/investor-bot/blob/master/predictors/data/adjclose/equities.csv

[aisthesis]

Just created a new branch for the issue: https://github.com/aisthesis/investor-bot/tree/iss12

The only change is just a guarantee that the print() statement goes to stdout immediately so that we know for sure whether or not the loop where the issue is happening has been successful. It really has to be for the print statements to be executed at all. But I don't understand why "cat" wasn't getting picked up nor why it would fail on its very last run.

You might try checking out the issue branch (iss12) and playing around with it in that spot to see what's going on. It's hard for me, since I can't repro the issue without going through a complete Windows install.

[mscappini]

Yeah, I saw how the equities were resolved from the settings.py. I am getting dependencies set up again to try and reproduce the error. But I don't know if I'm familiar enough to resolve it.

[Irialad]

Want to get together and look at it? On Mar 28, 2015 12:44 PM, "Mark Scappini" notifications@github.com wrote:

Yeah, I saw how the equities were resolved from the settings.py. I am getting dependencies set up again to try and reproduce the error. But I don't know if I'm familiar enough to resolve it.

— Reply to this email directly or view it on GitHub https://github.com/aisthesis/investor-bot/issues/12#issuecomment-87291217 .

[mscappini]

I was using the direct numpy package instead of scipy. I'm trying to get scipy installed now to see if that resolves it.

[aisthesis]

@Rob sounds good to meet. When and where? I have plans this evening but am available this afternoon or tomorrow

[mscappini]

Cool, got it figured out. After installing a Windows binary for scipy and got that working, it was giving another error. @aisthesis correctly suggested that pylearn wasn't being properly resolved on Windows because of the symlink in that directory. Since it's virtually impossible to create and commit Windows symlinks to the repo, I will make a branch that (for now) will use a relative path to resolve pylearn from the local repo.