search

kmcd / clearwing

Basket trading with machine learning
1 stars 0 forks source link

select model fails with TypeError #9

Open kmcd opened 11 years ago

kmcd commented 11 years ago

In [6]: %run select_model_index.py -it 1 test_01-01-2010_31-12-2012.h5 processing 2010-06-16 09:30:00 processing 2010-10-06 09:30:00 processing 2010-06-04 09:30:00 processing 2011-07-11 09:30:00 processing 2010-06-15 09:30:00 processing 2010-08-11 09:30:00 processing 2012-02-22 09:30:00 processing 2010-08-23 09:30:00 processing 2010-01-14 09:30:00 processing 2012-05-02 09:30:00 processing 2012-01-31 09:30:00 processing 2011-12-16 09:30:00 processing 2012-10-19 09:30:00 (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 10, 19, 9, 30),), <traceback object at 0x110cb28c0>) no record found, maybe a holiday processing 2012-11-15 09:30:00 (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 11, 15, 9, 30),), <traceback object at 0x110cb7e60>) no record found, maybe a holiday processing 2011-07-13 09:30:00 processing 2012-08-17 09:30:00 (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 8, 17, 9, 30),), <traceback object at 0x110cb27a0>) no record found, maybe a holiday processing 2012-01-24 09:30:00 processing 2010-08-19 09:30:00 processing 2011-01-04 09:30:00 processing 2010-11-02 09:30:00 iter 1 today = 2012-01-24 09:30:00 top10 nasdaq components : array([aapl, goog, msft, amzn, intc, nflx, csco, orcl, stx, qcom], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137ce998>) today = 2010-08-11 09:30:00 top10 nasdaq components : array([aapl, intc, msft, goog, nflx, csco, amzn, qcom, brcm, wynn], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137cef80>) today = 2011-01-04 09:30:00 top10 nasdaq components : array([aapl, nflx, goog, intc, amzn, csco, orcl, ebay, pcln, siri], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137c2b48>) today = 2010-06-16 09:30:00 top10 nasdaq components : array([aapl, pcln, intc, goog, csco, nflx, nwsa, ntap, msft, brcm], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137bd128>) today = 2011-07-13 09:30:00 top10 nasdaq components : array([aapl, nflx, goog, nwsa, amzn, pcln, orcl, intc, msft, qcom], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137bd950>) today = 2010-01-14 09:30:00 top10 nasdaq components : array([goog, orcl, aapl, amzn, qcom, msft, genz, intc, dell, ebay], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137b2248>) today = 2010-11-02 09:30:00 top10 nasdaq components : array([aapl, msft, nflx, goog, ffiv, amzn, csco, orcl, qcom, wynn], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137bd368>) today = 2010-10-06 09:30:00 top10 nasdaq components : array([aapl, goog, qcom, nflx, cost, ctxs, msft, ffiv, intc, adbe], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137bd518>) today = 2011-12-16 09:30:00 top10 nasdaq components : array([aapl, goog, msft, intc, csco, orcl, qcom, amzn, amgn, cmcsa], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137bdb48>) today = 2010-08-19 09:30:00 top10 nasdaq components : array([intc, aapl, csco, goog, ntap, symc, nflx, amzn, qcom, sndk], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137c22d8>) today = 2011-07-11 09:30:00 top10 nasdaq components : array([aapl, msft, goog, nflx, amzn, intc, nwsa, orcl, csco, siri], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137b2a28>) today = 2012-05-02 09:30:00 top10 nasdaq components : array([aapl, intc, amzn, msft, pcln, goog, wynn, brcm, cmcsa, csco], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1144bca70>) today = 2010-08-23 09:30:00 top10 nasdaq components : array([aapl, msft, goog, intu, nflx, intc, amzn, csco, pcln, akam], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1144da3b0>) today = 2010-06-15 09:30:00 top10 nasdaq components : array([aapl, goog, nflx, amzn, nwsa, msft, intc, csco, sndk, apol], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1144bc710>) (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 8, 17, 9, 30),), <traceback object at 0x1144bcb48>) (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 11, 15, 9, 30),), <traceback object at 0x1144bc710>) today = 2010-06-04 09:30:00 top10 nasdaq components : array([aapl, goog, msft, csco, orcl, ebay, intc, amzn, dtv, qcom], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1144bcb48>) today = 2012-02-22 09:30:00 top10 nasdaq components : array([aapl, nflx, goog, dell, grmn, msft, gild, csco, amzn, joy], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137ce998>) (<type 'exceptions.KeyError'>, KeyError(datetime.datetime(2012, 10, 19, 9, 30),), <traceback object at 0x1144bcb48>) today = 2012-01-31 09:30:00 top10 nasdaq components : array([aapl, goog, amzn, nflx, msft, qcom, intc, pcar, csco, orcl], dtype=object) (<type 'exceptions.TypeError'>, TypeError("sum() got an unexpected keyword argument 'dtype'",), <traceback object at 0x1137b2638>) dataset: test_01-01-2010_31-12-2012.h5 k range: [5, 6, 7, 8, 9] iterations: 1 days: 20 5 6 7 8 9 std dev error: NaN NaN NaN NaN NaN 5 6 7 8 9 avg error: NaN NaN NaN NaN NaN

kmcd commented 11 years ago

Doesn't matter if I run it on test/train/validate sets, same error.

kmcd commented 11 years ago

The last example is from an ipython session.

When I run from the command line, I get the following error:

$ python select_model_index.py -it 1 test_01-01-2010_31-12-2012.h5 Traceback (most recent call last): File "select_model_index.py", line 29, in store = HDFStore(args.in_dir+'/'+args.dataset) File "/Users/kmcd/src/clearwing/venv/lib/python2.7/site-packages/pandas/io/pytables.py", line 180, in init self.open(mode=mode, warn=False) File "/Users/kmcd/src/clearwing/venv/lib/python2.7/site-packages/pandas/io/pytables.py", line 248, in open self.handle = _tables().openFile(self.path, self.mode) File "/Users/kmcd/src/clearwing/venv/lib/python2.7/site-packages/tables/file.py", line 230, in openFile return File(filename, mode, title, rootUEP, filters, kwargs) File "/Users/kmcd/src/clearwing/venv/lib/python2.7/site-packages/tables/file.py", line 495, in init self._g_new(filename, mode, params) File "hdf5Extension.pyx", line 317, in tables.hdf5Extension.File._g_new (tables/hdf5Extension.c:3039) tables.exceptions.HDF5ExtError: HDF5 error back trace

File "H5F.c", line 1522, in H5Fopen unable to open file File "H5F.c", line 1313, in H5F_open unable to read superblock File "H5Fsuper.c", line 334, in H5F_super_read unable to find file signature File "H5Fsuper.c", line 155, in H5F_locate_signature unable to find a valid file signature

End of HDF5 error back trace

Unable to open/create file 'data/training/test_01-01-2010_31-12-2012.h5'

karlosmid commented 11 years ago

I run command with exact parameters on my dev machine and I did not got that exception. Two questions:

  1. Could you give me access to Ipython server, so I can try to replicate problem.
  2. For same command, you received two different exceptions. I think that there was a problem on the server with Ipython session and that problem is not related with clearwing scripts.
kmcd commented 11 years ago
  1. No - I'm running iPython on my laptop
  2. The different exception is because the first command was run in iPython, the second from the command line; ie without iPython.
kmcd commented 11 years ago

Aslo, I'm using venv & pip to run my development environment. Maybe that's causing the issue?

https://devcenter.heroku.com/articles/python

karlosmid commented 11 years ago
  1. I am also using virtualenv python module. I successfully installed all required packages using pip in virtualenv.
  2. Have you ever successively run any script on your laptop?
  3. Could you please observe my comment in clearwing timesheet about the installation issues that I had and successfully resolved?