blue-yonder / tsfresh

Automatic extraction of relevant features from time series:
http://tsfresh.readthedocs.io
MIT License
8.43k stars 1.21k forks source link

Deprecated boolean operation #405

Closed hassanshah7866 closed 6 years ago

hassanshah7866 commented 6 years ago
  1. Operating system: Linux 18.04 LTS

  1. Python version: 3.6.2

  1. tsfresh version: 0.11.0

  1. Extracted row from data: test.zip

  1. Code snippet:

data = pd.read_csv('test.csv', encoding='latin1') extracted_features = extract_features(data, column_id="reservation_number", column_value="no_show", column_sort="arrival_date")

Using extract_features() generates the problem.


  1. Traceback:

Feature Extraction: 0%| | 0/10 [00:00<?, ?it/s]multiprocessing.pool.RemoteTraceback: """ Traceback (most recent call last): File "/home/hassan/.conda/envs/bookings/lib/python3.6/multiprocessing/pool.py", line 119, in worker result = (True, func(*args, kwds)) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/utilities/distribution.py", line 38, in _function_with_partly_reduce results = list(itertools.chain.from_iterable(results)) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/utilities/distribution.py", line 37, in results = (map_function(chunk, kwargs) for chunk in chunk_list) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/feature_extraction/extraction.py", line 293, in _do_extraction_on_chunk return list(_f()) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/feature_extraction/extraction.py", line 280, in _f result = func(data, param=parameter_list) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/feature_extraction/feature_calculators.py", line 205, in symmetry_looking max_min_difference = max(x) - min(x) TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead. """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "", line 1, in File "/snap/pycharm-professional/80/helpers/pydev/_pydev_bundle/pydev_umd.py", line 194, in runfile pydev_imports.execfile(filename, global_vars, local_vars) # execute the script File "/snap/pycharm-professional/80/helpers/pydev/_pydev_imps/_pydev_execfile.py", line 18, in execfile exec(compile(contents+"\n", file, 'exec'), glob, loc) File "/home/hassan/infomgmt/projects/bookings/src/main/python/train_with_feature_selection.py", line 29, in extracted_features = extract_features(data, column_id="reservation_number", column_value="no_show", column_sort="arrival_date") File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/feature_extraction/extraction.py", line 152, in extract_features distributor=distributor) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/feature_extraction/extraction.py", line 233, in _do_extraction function_kwargs=kwargs) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tsfresh/utilities/distribution.py", line 149, in map_reduce result = list(itertools.chain.from_iterable(result)) File "/home/hassan/.conda/envs/bookings/lib/python3.6/site-packages/tqdm/_tqdm.py", line 931, in iter for obj in iterable: File "/home/hassan/.conda/envs/bookings/lib/python3.6/multiprocessing/pool.py", line 735, in next raise value TypeError: numpy boolean subtract, the - operator, is deprecated, use the bitwise_xor, the ^ operator, or the logical_xor function instead.


  1. Bug fix:

Replace

max(x) - min(x)

with

np.logical_xor(max(x), min(x))

for all occurences.

earthgecko commented 6 years ago

Hi @hassanshah7866

Would it be possible for you to add the versions of all your pip dependencies to aid in debugging the issue.


pip list

It is probably just enough to know the numpy version, however knowing all the version of the packages in https://github.com/blue-yonder/tsfresh/blob/master/requirements.txt may be useful.

To fix this issue immediately for you, you may just have to install the lower version of numpy. I am pointing this out as a possible workaround to the problem, however please assess how applicable this is for your environment. If you are running tsfresh in a virtualenv or you are confident that your are not going to break something in some other aspect of numpy use in your environment you can do ths:


pip install numpy==1.10.4

That will provide a version of numpy that has been tested and is known to work. I hope this helps.

hassanshah7866 commented 6 years ago

Hello, I am using a conda virtual environment, so maybe that is more relevant?

`

packages in environment at /home/hassan/anaconda3:

#

Name Version Build Channel

_ipyw_jlab_nb_ext_conf 0.1.0 py36he11e457_0
absl-py 0.1.13 alabaster 0.7.10 py36h306e16b_0
anaconda custom py36hbbc8b67_0
anaconda-client 1.6.5 py36h19c0dcd_0
anaconda-navigator 1.6.8 py36h672ccc7_0
anaconda-project 0.8.0 py36h29abdf5_0
asn1crypto 0.22.0 py36h265ca7c_1
astor 0.6.2 astroid 1.5.3 py36hbdb9df2_0
astropy 2.0.2 py36ha51211e_4
babel 2.5.0 py36h7d14adf_0
backports 1.0 py36hfa02d7e_1
backports.shutil_get_terminal_size 1.0.0 py36hfea85ff_2
beautifulsoup4 4.6.0 py36h49b8c8c_1
bitarray 0.8.1 py36h5834eb8_0
bkcharts 0.2 py36h735825a_0
blaze 0.11.3 py36h4e06776_0
bleach 2.0.0 py36h688b259_0
bleach 1.5.0 bokeh 0.12.7 py36h169c5fd_1
boto 2.48.0 py36h6e4cd66_1
bottleneck 1.2.1 py36haac1ea0_0
ca-certificates 2018.4.16 0 conda-forge cairo 1.14.10 h58b644b_4
certifi 2018.8.13 py36_0 conda-forge cffi 1.10.0 py36had8d393_1
chardet 3.0.4 py36h0f667ec_1
click 6.7 py36h5253387_0
cloudpickle 0.4.0 py36h30f8c20_0
clyent 1.2.2 py36h7e57e65_1
colorama 0.3.9 py36h489cec4_0
conda 4.5.10 py36_0 conda-forge conda-build 3.0.23 py36h1da9bb0_0
conda-env 2.6.0 h36134e3_1
conda-verify 2.0.0 py36h98955d8_0
contextlib2 0.5.5 py36h6c84a62_0
cryptography 2.0.3 py36ha225213_1
curl 7.55.1 hcb0b314_2
cycler 0.10.0 py36h93f1223_0
cython 0.26.1 py36h21c49d0_0
cytoolz 0.8.2 py36h708bfd4_0
dask 0.15.2 py36h9b48dc4_0
dask-core 0.15.2 py36h0f988a8_0
datashape 0.5.4 py36h3ad6b5c_0
dbus 1.10.22 h3b5a359_0
decorator 4.1.2 py36hd076ac8_0
distributed 1.18.3 py36h73cd4ae_0
docutils 0.14 py36hb0f60f5_0
entrypoints 0.2.3 py36h1aec115_2
et_xmlfile 1.0.1 py36hd6bccc3_0
expat 2.2.4 hc00ebd1_1
fastcache 1.0.2 py36h5b0c431_0
filelock 2.0.12 py36hacfa1f5_0
flask 0.12.2 py36hb24657c_0
flask-cors 3.0.3 py36h2d857d3_0
fontconfig 2.12.4 h88586e7_1
freetype 2.8 h52ed37b_0
future 0.16.0 py36_0 conda-forge gast 0.2.0 get_terminal_size 1.0.0 haa9412d_0
gevent 1.2.2 py36h2fe25dc_0
glib 2.53.6 hc861d11_1
glob2 0.5 py36h2c1b292_1
gmp 6.1.2 hb3b607b_0
gmpy2 2.0.8 py36h55090d7_1
google-api-python-client 1.6.6 graphite2 1.3.10 hc526e54_0
greenlet 0.4.12 py36h2d503a6_0
grpcio 1.11.0 gst-plugins-base 1.12.2 he3457e5_0
gstreamer 1.12.2 h4f93127_0
h5py 2.7.0 py36he81ebca_1
harfbuzz 1.5.0 h2545bd6_0
hdf5 1.10.1 hb0523eb_0
heapdict 1.0.0 py36h79797d7_0
html5lib 0.9999999 html5lib 0.999999999 py36h2cfc398_0
httplib2 0.11.3 icu 58.2 h211956c_0
idna 2.6 py36h82fb2a8_1
imageio 2.2.0 py36he555465_0
imagesize 0.7.1 py36h52d8127_0
intel-openmp 2018.0.0 h15fc484_7
ipykernel 4.6.1 py36hbf841aa_0
ipython 6.1.0 py36hc72a948_1
ipython_genutils 0.2.0 py36hb52b0d5_0
ipywidgets 7.0.0 py36h7b55c3a_0
isort 4.2.15 py36had401c0_0
itsdangerous 0.24 py36h93cc618_1
jbig 2.1 hdba287a_0
jdcal 1.3 py36h4c697fb_0
jedi 0.10.2 py36h552def0_0
jinja2 2.9.6 py36h489bce4_1
jpeg 9b habf39ab_1
jsonschema 2.6.0 py36h006f8b5_0
jupyter 1.0.0 py36h9896ce5_0
jupyter_client 5.1.0 py36h614e9ea_0
jupyter_console 5.2.0 py36he59e554_1
jupyter_core 4.3.0 py36h357a921_0
jupyterlab 0.32.0 py36_1 conda-forge jupyterlab_launcher 0.10.5 py36_0 conda-forge Keras 2.1.5 lazy-object-proxy 1.3.1 py36h10fcdad_0
libedit 3.1 heed3624_0
libffi 3.2.1 h4deb6c0_3
libgcc-ng 7.2.0 hcbc56d2_1
libgfortran-ng 7.2.0 h6fcbd8e_1
libpng 1.6.32 hda9c8bc_2
libsodium 1.0.13 h31c71d8_2
libssh2 1.8.0 h8c220ad_2
libstdcxx-ng 7.2.0 h24385c6_1
libtiff 4.0.8 h90200ff_9
libtool 2.4.6 hd50d1a6_0
libxcb 1.12 he6ee5dd_2
libxml2 2.9.4 h6b072ca_5
libxslt 1.1.29 hcf9102b_5
llvmlite 0.20.0 py36_0
locket 0.2.0 py36h787c0ad_1
lxml 3.8.0 py36h6c6e760_0
lzo 2.10 hc0eb8fc_0
Markdown 2.6.11 markupsafe 1.0 py36hd9260cd_1
matplotlib 2.0.2 py36h2acb4ad_1
mccabe 0.6.1 py36h5ad9710_1
mistune 0.7.4 py36hbab8784_0
mkl 2018.0.2 1
mkl-service 1.1.2 py36h17a0993_4
mpc 1.0.3 hf803216_4
mpfr 3.1.5 h12ff648_1
mpmath 0.19 py36h8cc018b_2
msgpack-python 0.4.8 py36hec4c5d1_0
multipledispatch 0.4.9 py36h41da3fb_0
navigator-updater 0.1.0 py36h14770f7_0
nbconvert 5.3.1 py36hb41ffb7_0
nbformat 4.4.0 py36h31c9010_0
ncurses 6.0 h06874d7_1
networkx 1.11 py36hfb3574a_0
nltk 3.2.4 py36h1a0979f_0
nose 1.3.7 py36hcdf7029_2
notebook 5.0.0 py36h0b20546_2
numba 0.35.0 np113py36_10
numexpr 2.6.2 py36hdd3393f_1
numpy 1.13.1 py36h5bc529a_2
numpy 1.14.2 numpydoc 0.7.0 py36h18f165f_0
oauth2client 4.1.2 odo 0.5.1 py36h90ed295_0
olefile 0.44 py36h79f9f78_0
openpyxl 2.4.8 py36h41dd2a8_1
openssl 1.0.2o h470a237_1 conda-forge packaging 16.8 py36ha668100_1
pandas 0.22.0 py36hf484d3e_0
pandoc 1.19.2.1 hea2e7c5_1
pandocfilters 1.4.2 py36ha6701b7_1
pango 1.40.11 hedb6d6b_0
partd 0.3.8 py36h36fd896_0
patchelf 0.9 hf79760b_2
path.py 10.3.1 py36he0c6f6d_0
pathlib2 2.3.0 py36h49efa8e_0
patsy 0.4.1 py36ha3be15e_0
pcre 8.41 hc71a17e_0
pep8 1.7.0 py36h26ade29_0
pexpect 4.2.1 py36h3b9d41b_0
pickleshare 0.7.4 py36h63277f8_0
pillow 4.2.1 py36h9119f52_0
pip 9.0.1 py36h30f8307_2
pip 18.0 pixman 0.34.0 ha72d70b_1
pkginfo 1.4.1 py36h215d178_1
ply 3.10 py36hed35086_0
prompt_toolkit 1.0.15 py36h17d85b1_0
protobuf 3.5.2.post1 psutil 5.2.2 py36h74c8701_0
ptyprocess 0.5.2 py36h69acd42_0
py 1.4.34 py36h0712aa3_1
pyasn1 0.4.2 pyasn1-modules 0.2.1 pycodestyle 2.3.1 py36hf609f19_0
pycosat 0.6.3 py36h0a5515d_0
pycparser 2.18 py36hf9f622e_1
pycrypto 2.6.1 py36h6998063_1
pycurl 7.43.0 py36h5e72054_3
pyflakes 1.5.0 py36h5510808_1
pygments 2.2.0 py36h0d3125c_0
pylint 1.7.2 py36h484ab97_0
pyodbc 4.0.17 py36h999153c_0
pyopenssl 17.2.0 py36h5cc804b_0
pyparsing 2.2.0 py36hee85983_1
pyqt 5.6.0 py36h0386399_5
pysocks 1.6.7 py36hd97a5b1_1
pytables 3.4.2 py36hdce54c9_1
pytest 3.2.1 py36h11ad3bb_1
python 3.6.2 hdfe5801_15
python-dateutil 2.6.1 py36h88d3b88_1
pytz 2017.2 py36hc2ccc2a_1
pywavelets 0.5.2 py36he602eb0_0
pyyaml 3.12 py36hafb9ca4_1
pyzmq 16.0.2 py36h3b0cf96_2
qt 5.6.2 h974d657_12
qtawesome 0.4.4 py36h609ed8c_0
qtconsole 4.3.1 py36h8f73b5b_0
qtpy 1.3.1 py36h3691cc8_0
readline 7.0 hac23ff0_3
requests 2.18.4 py36he2e5f8d_1
rope 0.10.5 py36h1f8c17e_0
rsa 3.4.2 ruamel_yaml 0.11.14 py36ha2fb22d_2
scikit-image 0.13.0 py36had3c07a_1
scikit-learn 0.19.1 py36h7aa7ec6_0
scipy 1.0.1 py36hfc37229_0
seaborn 0.8.1 py36hfad7ec4_0
setuptools 36.5.0 py36he42e2e1_0
setuptools 39.0.1 simplegeneric 0.8.1 py36h2cb9092_0
singledispatch 3.4.0.3 py36h7a266c3_0
sip 4.18.1 py36h51ed4ed_2
six 1.11.0 six 1.10.0 py36hcac75e4_1
snowballstemmer 1.2.1 py36h6febd40_0
sortedcollections 0.5.3 py36h3c761f9_0
sortedcontainers 1.5.7 py36hdf89491_0
sphinx 1.6.3 py36he5f0bdb_0
sphinxcontrib 1.0 py36h6d0f590_1
sphinxcontrib-websupport 1.0.1 py36hb5cb234_1
spyder 3.2.3 py36he38cbf7_1
sqlalchemy 1.1.13 py36hfb5efd7_0
sqlite 3.20.1 h6d8b0f3_1
statsmodels 0.8.0 py36h8533d0b_0
sympy 1.1.1 py36hc6d1c1c_0
tblib 1.3.2 py36h34cf8b6_0
tensorboard 1.7.0 tensorflow 1.7.0 termcolor 1.1.0 terminado 0.6 py36ha25a19f_0
testpath 0.3.1 py36h8cadb63_0
tk 8.6.7 h5979e9b_1
toolz 0.8.2 py36h81f2dff_0
tornado 4.5.2 py36h1283b2a_0
tqdm 4.23.4 py_0 conda-forge traitlets 4.3.2 py36h674d592_0
tsfresh 0.11.0 py_0 conda-forge typing 3.6.2 py36h7da032a_0
unicodecsv 0.14.1 py36ha668878_0
unixodbc 2.3.4 hc36303a_1
uritemplate 3.0.0 urllib3 1.22 py36hbe7ace6_0
wcwidth 0.1.7 py36hdf4376a_0
webencodings 0.5.1 py36h800622e_1
Werkzeug 0.14.1 werkzeug 0.12.2 py36hc703753_0
wheel 0.29.0 py36he7f4e38_1
wheel 0.31.0 widgetsnbextension 3.0.2 py36hd01bb71_1
wrapt 1.10.11 py36h28b7045_0
xgboost 0.71 py36_0 conda-forge xlrd 1.1.0 py36h1db9f0c_1
xlsxwriter 0.9.8 py36hf41c223_0
xlwt 1.3.0 py36h7b00a1f_0
xz 5.2.3 h2bcbf08_1
yaml 0.1.7 h96e3832_1
zeromq 4.2.2 hb0b69da_1
zict 0.1.2 py36ha0d441b_0
zlib 1.2.11 hfbfcf68_1
`

earthgecko commented 6 years ago

Hi @hassanshah7866

Seeing as you have a lot installed in that conda environment, I would recommend if you want to try and get tfresh running asap without having to make changes in your exisitng conda environment, you could set it up in its own virtualenv. Then the installation of numpy==1.10.4 will not have any effect on your existing conda environment and packages as changing the numpy version in your current conda environment looks like it would impact a lot of things.

There seems to be a fairly good howto at https://uoa-eresearch.github.io/eresearch-cookbook/recipe/2014/11/20/conda/

After you had set up a conda virtualenv named tsfresh you could do:


conda install -n tsfresh numpy==1.10.4
conda install -n tsfresh tsfresh

That would provide the requirements.txt numpy version for tsfresh and tsfresh all totally isolated from your existing conda environment, which is a much cleaner option given your environment. Once again, I hope this helps you.

earthgecko commented 6 years ago

I was just looking at the code and there has been a change in master relating to his function in master at: https://github.com/blue-yonder/tsfresh/commit/48226bade8f5dd65ca021a4977f22ad5dd60769b#diff-6d5c97a0f5c5242f0f8d14214ec0a84eR209

max_min_difference = max(x) - min(x) was changed to:

max_min_difference = np.max(x) - np.min(x)

This change was not related to this numpy deprecation that you have encounter, but related to General performance improvements. (#339)

However it appears as if numpy have now deprecated this in somewhere between 1.10.4 and 1.14.2 so the replacement with np.logical_xor(max(x), min(x)) any replacements will all need to be tested and any other similar functions too.

Thanks for the reporting the issue @hassanshah7866 , perhaps there can be some testing done this weekend.

hassanshah7866 commented 6 years ago

Many thanks for your help and the replies, @earthgecko. Much appreciated!

MaxBenChrist commented 6 years ago

The new 0.11.1 release will contain that fix