search

slaypni / fastdtw

A Python implementation of FastDTW
MIT License
785 stars 122 forks source link

Inconsistent results between different python environment. #46

Open taroushirani opened 4 years ago

taroushirani commented 4 years ago

Hello, I got inconsistent results from the same data when I changed the python environment.

test code: 

import numpy as np
from fastdtw import fastdtw

x = [1, 6, 6, 33, 35]
y = [1, 6, 6, 33, 35]

d, path = fastdtw(x, y, radius=len(x))

print(f"d: {d}")
print(f"path: {path}")

environment a: WinPython 3.8.5 on Windows 10 Pro 64bit

$ python --version
Python 3.8.5

$ pip list | grep fastdtw
fastdtw                0.3.4

$ python test_fastdtw.py
d: 0.0
path: [(0, 0), (1, 1), (1, 2), (2, 2), (3, 3), (4, 4)]

environment b: Debian GNU/Linux(buster) (WSL)

$ python -V
Python 3.7.3

$ pip list | grep fastdtw
fastdtw                0.3.4

$ python test_fastdtw.py
d: 0.0
path: [(0, 0), (1, 1), (2, 2), (3, 3), (4, 4)]

The result of latter is what I need. Could anyone please advise me to avoid this inconsistency?

taroushirani commented 4 years ago

I found that pip of WinPython installed a pure python version of fastdtw. When I built fastdtw from source and changed to use the compiled version of fastdtw, the results of two environments matched. So this phenomenon results from the difference between two fastdtw implementation(pure python version or c++ version).

slaypni commented 4 years ago

Thank you for the information. Maybe better to change setup.py to install cython version always. If someone knows how to do that without dropping portability (architecture compatibility), I would like to know it.

apacha commented 3 years ago

I experience inconsistencies when using FastDTW too. I have two Mac Books with a very similar environment, but the results are different, causing some of my tests to fail:

My test code is simply

from fastdtw import fastdtw

x = [1, 4, 2, 6, 3, 3, 3, 5, 7, 1, 2, 1]
y = [1, 4, 6, 3, 3, 3, 3, 5, 8, 1, 2, 1, 9]
distance, path = fastdtw(x, y)
print(distance)
print(path)

Environment 1

(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % python --version
Python 3.8.2
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip --version
pip 21.0.1 from /Users/alex/Repositories/omr-reconstruction/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip freeze | grep fastdtw
fastdtw==0.3.4
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip freeze | grep numpy  
numpy==1.20.1

Result:

11.0
[(0, 0), (1, 1), (2, 1), (3, 2), (4, 3), (4, 4), (5, 5), (6, 6), (7, 7), (8, 8), (9, 9), (10, 10), (11, 11), (11, 12)]

Environment 2

(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % python --version
Python 3.8.7
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip --version
pip 21.0.1 from /Users/alex/Repositories/omr-reconstruction/venv/lib/python3.8/site-packages/pip (python 3.8)
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip freeze | grep fastdtw
fastdtw==0.3.4
(venv) (base) alex@Alexanders-MacBook-Pro omr-reconstruction % pip freeze | grep numpy
numpy==1.20.1

Result:

11.0
[(0, 0), (1, 1), (2, 1), (3, 2), (4, 3), (4, 4), (4, 5), (4, 6), (5, 6), (6, 6), (7, 7), (8, 8), (9, 9), (10, 10), (11, 11), (11, 12)]

Do you have any idea what might be causing these inconsistencies? If more system information is needed, I will gladly provide it.

tsh11na commented 1 year ago

As written in https://github.com/slaypni/fastdtw/issues/46#issuecomment-692341110, that inconsistency is due to wheter the installed version internally uses Cython or not.

In #62, I modified version description for users to identify the actual installed version. If your version uses Cython, the description will be "0.3.4+Cython", otherwise "0.3.4+raw".

As far as I've investigated, the differences between Cython and Python versions were:

mrodencal-tbiov commented 1 year ago

@tsh11na can you please share your environment set up (which libs you have installed & what version they are at) to get the cython version to work on Python 3.8? I can't get the cython version it to build at all in a codespace venv

tsh11na commented 1 year ago

@mrodencal-tbiov Hello! I've confirmed that the Cython version works with Python 3.8.10 (venv environment). You may need to install Cython (pip install cython) before installing fastdtw and other packages.

The packages I've installed are:

Package             Version     
------------------- ------------
commonmark          0.9.1       
contourpy           1.0.7       
cycler              0.11.0      
Cython              0.29.33     
fastdtw             0.3.4+cython
fonttools           4.38.0      
importlib-resources 5.12.0      
kiwisolver          1.4.4       
markdown-it-py      2.2.0       
matplotlib          3.7.0       
mdurl               0.1.2       
numpy               1.24.2      
packaging           23.0        
pandas              1.5.3       
Pillow              9.4.0       
pip                 20.0.2      
pkg-resources       0.0.0       
Pygments            2.14.0      
pyparsing           3.0.9       
python-dateutil     2.8.2       
pytz                2022.7.1    
rich                13.3.1      
seaborn             0.12.2      
setuptools          44.0.0      
six                 1.16.0      
typing-extensions   4.5.0       
wheel               0.38.4      
wslPath             0.3.1       
zipp                3.15.0
mrodencal-tbiov commented 1 year ago

@tsh11na I figured out my problem, and seeing how I stumbled onto it by accident I figured I will post here so hopefully future ppl using this library don't lose days to this like I have.

I was installing all of the requirements from a requirements.txt and using a make file to build the project. You were right that you need to have all of the libraries fastdtw uses installed BEFORE installing it, even if you use a requirements.txt and have fastdtw listed at the bottom of the list. This means that your Makefile needs to install numpy before trying to install fastdtw with pip.

steps I used to get repeatability:

  1. uninstall all of the pip packages to start with a clean slate "pip freeze | xargs pip uninstall -y"
  2. upgrade pip to the latest "pip install --upgrade pip"
  3. install your requirements.txt which must include numpy "pip install --no-cache-dir -r requirements.txt"
  4. install fastdtw "pip install fastdtw==0.3.4"

I also checked and you do not need Cython installed for it to build properly. and I was able to repeat this successfully in both python version 3.8.10 and python version 3.9.16

The Makefile commands to successfully install fastdtw for your ci-cd piplines are as follows: install:

install commands

pip install --upgrade pip &&\
    pip install --no-cache-dir -r requirements.txt
pip install fastdtw==0.3.4

Hope this helps someone in the future.