search

ramachandran-lab / SWIFr

Sweep Inference Framework (controlling for correlation)
GNU General Public License v3.0
28 stars 4 forks source link

sklearn associated error #1

Closed oeco28 closed 5 years ago

oeco28 commented 5 years ago

Hello, I have python 2.7.15+ running on ubuntu 18.04.2 LTS. I have scipy v '0.19.1', matplotlib v '2.1.1' and sklearn '0.20.3'

I tried running the example of the training set following the example provided:

python SWIFr_train.py --path example_2classes/

I get the following error: learning neutral marginal distributions for Fst... learning sweep marginal distributions for Fst... learning neutral marginal distributions for XP-EHH... learning sweep marginal distributions for XP-EHH... learning neutral marginal distributions for iHS... learning sweep marginal distributions for iHS... learning neutral marginal distributions for DDAF... learning sweep marginal distributions for DDAF... learning neutral joint distributions for Fst and XP-EHH... learning sweep joint distributions for Fst and XP-EHH... learning neutral joint distributions for Fst and iHS... learning sweep joint distributions for Fst and iHS... learning neutral joint distributions for Fst and DDAF... learning sweep joint distributions for Fst and DDAF... learning neutral joint distributions for XP-EHH and iHS... learning sweep joint distributions for XP-EHH and iHS... learning neutral joint distributions for XP-EHH and DDAF... learning sweep joint distributions for XP-EHH and DDAF... learning neutral joint distributions for iHS and DDAF... learning sweep joint distributions for iHS and DDAF... learning number of Gaussian mixture components for Fst Traceback (most recent call last): File "SWIFr_train.py", line 333, in A.run_bic() File "SWIFr_train.py", line 173, in run_bic self.plot_bic_1D(stat,scenario) File "SWIFr_train.py", line 143, in plot_bic_1D H = mixture.GMM(n_components=n) AttributeError: 'module' object has no attribute 'GMM'

Is it possible this is a problem of using a slightly newer version of sklearn? Your recommended version is v0.17, but I have 0.20.3

if you have seen this issue, please let me know. I will continue digging.

cheers, Omar

lasugden commented 5 years ago

Hi Omar,

The problem is indeed with the newer version of sklearn -- if you drop down to an earlier version, this should be cleared up. We will be addressing this issue in an upcoming new release!

oeco28 commented 5 years ago

I still get an issue, Now with matplotlib. When I tried to drop down to matplotlib 1.7 I can't. See error message:

Could not find a version that satisfies the requirement matplotlib==1.7 (from versions: 0.86, 0.86.1, 0.86.2, 0.91.0, 0.91.1, 1.0.1, 1.1.0, 1.1.1, 1.2.0, 1.2.1, 1.3.0, 1.3.1, 1.4.0, 1.4.1rc1, 1.4.1, 1.4.2, 1.4.3, 1.5.0, 1.5.1, 1.5.2, 1.5.3, 2.0.0b1, 2.0.0b2, 2.0.0b3, 2.0.0b4, 2.0.0rc1, 2.0.0rc2, 2.0.0, 2.0.1, 2.0.2, 2.1.0rc1, 2.1.0, 2.1.1, 2.1.2, 2.2.0rc1, 2.2.0, 2.2.2, 2.2.3, 2.2.4) No matching distribution found for matplotlib==1.7

are you sure the working version is 1.7?

Omar

On Fri, Aug 2, 2019 at 8:31 AM lasugden notifications@github.com wrote:

Closed #1 https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ramachandran-2Dlab_SWIFr_issues_1&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=ufztN6KMc8q6PzgjbanQ7t46_XgFmdenduwQg7tNack&s=a7udlRKxrrcc9tg0T6koaowY7mYMaUYZxROo5MXTztc&e= .

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ramachandran-2Dlab_SWIFr_issues_1-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DADNNQ266ZSZ4B4R55XP5PB3QCRHOFA5CNFSM4IIVKDI2YY3PNVWWK3TUL52HS4DFWZEXG43VMVCXMZLOORHG65DJMZUWGYLUNFXW5KTDN5WW2ZLOORPWSZGOS3GUOMY-23event-2D2530035507&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=ufztN6KMc8q6PzgjbanQ7t46_XgFmdenduwQg7tNack&s=wLLuzvRF35_07ajYQPu5jgUWgU_9pl4pibKZvQNHNiM&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADNNQ23LDWUEPFBFJ46XOXLQCRHOFANCNFSM4IIVKDIQ&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=ufztN6KMc8q6PzgjbanQ7t46_XgFmdenduwQg7tNack&s=f4r5wz0XGKjlQKWEnkv7ryzMuuvhFPjy4UaivQhWJw0&e= .

-- Omar E. Cornejo, PhD Assistant Professor, School of Biological Sciences Washington State University Heald 429. PO Box 644236 Pullman, WA. 99164-4236

ghost commented 5 years ago

I can't reproduce this bug exactly, but I believe matplotlib version shouldn't matter like sklearn version, it's just that sklearn changed some feature names in a more recent release. I would try letting matplotlib run with the latest version, and just control the sklearn version.

oeco28 commented 5 years ago

thanks you I am sorry to keep bothering you. I tried going to a previous version of matplotlib (v1.5.3) but then I got a different problem which seems to be related to tcl/tk version. When I tried solving it, it seemed that the problem was the numpy version . What numpy version did you have in the working pipe?

thanks!

On Fri, Aug 2, 2019 at 11:16 AM kahlquist-brown notifications@github.com wrote:

I can't reproduce this bug exactly, but I believe matplotlib version shouldn't matter like sklearn version, it's just that sklearn changed some feature names in a more recent release. I would try letting matplotlib run with the latest version, and just control the sklearn version.

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ramachandran-2Dlab_SWIFr_issues_1-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DADNNQ24SSPMIENICHTMXZFDQCR2Y7A5CNFSM4IIVKDI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OPRQA-23issuecomment-2D517798080&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=dgx6e_tpg0wetwMkCNYcRbB7PWxtwfoLws3nd9mhnz8&s=n3aG5IE5i2ZV2xYnPaekt49tI0-B6xqZeXbtXc_m3qc&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADNNQ2462R3INXVLTNFXHHTQCR2Y7ANCNFSM4IIVKDIQ&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=dgx6e_tpg0wetwMkCNYcRbB7PWxtwfoLws3nd9mhnz8&s=E3GFyAEnllkHz1A_VPSBdKTCLIX50tPF6Tk4ZATJCXE&e= .

-- Omar E. Cornejo, PhD Assistant Professor, School of Biological Sciences Washington State University Heald 429. PO Box 644236 Pullman, WA. 99164-4236

ghost commented 5 years ago

I have a version running locally with the following: numpy==1.13.3 scikit-learn==0.19.1 matplotlib==2.1.0

Below I'll dump the result of pip freeze if there are other versions you are wondering about. You should generate warnings when using SWIFr_train.py with these versions installed (they are warning about the coming changes to sklearn in 0.20) but they should be harmless.

alabaster==0.7.10 anaconda-client==1.6.5 anaconda-navigator==1.6.9 anaconda-project==0.8.0 appnope==0.1.0 appscript==1.0.1 args==0.1.0 asn1crypto==0.22.0 astroid==1.5.3 astropy==2.0.2 awesome-slugify==1.6.5 Babel==2.5.0 backports-abc==0.5 backports.functools-lru-cache==1.4 backports.shutil-get-terminal-size==1.0.0 backports.ssl-match-hostname==3.5.0.1 beautifulsoup4==4.6.0 biopython==1.70 bitarray==0.8.1 bkcharts==0.2 blaze==0.11.3 bleach==2.0.0 bokeh==0.12.10 boto==2.48.0 Bottleneck==1.2.1 cdecimal==2.3 certifi==2017.7.27.1 cffi==1.10.0 chardet==3.0.4 click==6.7 clint==0.5.1 cloudpickle==0.4.0 clyent==1.2.2 colorama==0.3.9 configparser==3.5.0 contextlib2==0.5.5 cryptography==2.0.3 cycler==0.10.0 Cython==0.26.1 cytoolz==0.8.2 cyvcf2==0.8.7 dask==0.15.3 datashape==0.5.4 decorator==4.1.2 distributed==1.19.1 docopt==0.6.2 docutils==0.14 entrypoints==0.2.3 enum34==1.1.6 et-xmlfile==1.0.1 fastcache==1.0.2 filelock==2.0.12 Flask==0.12.2 Flask-Cors==3.0.3 funcsigs==1.0.2 functools32==3.2.3.post2 futures==3.1.1 gevent==1.2.2 glob2==0.5 gmpy2==2.0.8 greenlet==0.4.12 grin==1.2.1 h5py==2.7.0 heapdict==1.0.0 html5lib==0.999999999 idna==2.6 imageio==2.2.0 imagesize==0.7.1 intervaltree==2.1.0 ipaddress==1.0.18 ipykernel==4.6.1 ipython==5.4.1 ipython-genutils==0.2.0 ipywidgets==7.0.0 isort==4.2.15 itsdangerous==0.24 jdcal==1.3 jedi==0.10.2 Jinja2==2.9.6 jsonschema==2.6.0 jupyter-client==5.1.0 jupyter-console==5.2.0 jupyter-core==4.3.0 jupyterlab==0.27.0 jupyterlab-launcher==0.4.0 lazy-object-proxy==1.3.1 llvmlite==0.20.0 locket==0.2.0 lxml==4.1.0 MarkupSafe==1.0 matplotlib==2.1.0 mccabe==0.6.1 mistune==0.7.4 mpmath==0.19 msgpack-python==0.4.8 multipledispatch==0.4.9 navigator-updater==0.1.0 nbconvert==5.3.1 nbformat==4.4.0 networkx==1.11 nltk==3.2.4 nose==1.3.7 notebook==5.0.0 numba==0.35.0+6.gaa35fb1.dirty numexpr==2.6.2 numpy==1.13.3 numpydoc==0.7.0 odo==0.5.1 olefile==0.44 openpyxl==2.4.8 packaging==16.8 pandas==0.20.3 pandocfilters==1.4.2 partd==0.3.8 path.py==10.3.1 pathlib2==2.3.0 patsy==0.4.1 pep8==1.7.0 pexpect==4.2.1 pickleshare==0.7.4 Pillow==4.2.1 pkginfo==1.4.1 ply==3.10 prompt-toolkit==1.0.15 psutil==5.4.0 ptyprocess==0.5.2 py==1.4.34 PyAudio==0.2.7 pycodestyle==2.3.1 pycosat==0.6.2 pycparser==2.18 pycrypto==2.6.1 pycurl==7.43.0 pyflakes==1.6.0 Pygments==2.2.0 pylint==1.7.4 pyodbc==4.0.17 pyOpenSSL==17.2.0 pyparsing==2.2.0 PySocks==1.6.7 pytest==3.2.1 python-dateutil==2.6.1 pytz==2017.2 PyWavelets==0.5.2 PyYAML==3.12 pyzmq==16.0.2 QtAwesome==0.4.4 qtconsole==4.3.1 QtPy==1.3.1 regex==2018.2.21 requests==2.18.4 rope==0.10.5 ruamel-yaml==0.11.14 scandir==1.6 scikit-image==0.13.0 scikit-learn==0.19.1 scipy==0.19.1 seaborn==0.8 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.11.0 snowballstemmer==1.2.1 sortedcollections==0.5.3 sortedcontainers==1.5.7 Sphinx==1.6.3 sphinxcontrib-websupport==1.0.1 spyder==3.2.4 SQLAlchemy==1.1.13 statsmodels==0.8.0 subprocess32==3.2.7 sympy==1.1.1 tables==3.4.2 tabulate==0.8.2 tblib==1.3.2 terminado==0.6 testpath==0.3.1 toolz==0.8.2 tornado==4.5.2 traitlets==4.3.2 typing==3.6.2 unicodecsv==0.14.1 Unidecode==0.4.21 urllib3==1.22 VCF-kit==0.1.6 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.12.2 widgetsnbextension==3.0.2 wrapt==1.10.11 xlrd==1.1.0 XlsxWriter==1.0.2 xlwings==0.11.4 xlwt==1.2.0 yahmm==1.1.2 zict==0.1.3

oeco28 commented 5 years ago

I re-created the environment and it seems to work well. Do you get warnings though? While training I get the following warnings:

warnings.warn(msg, category=DeprecationWarning) /usr/local/lib/python2.7/dist-packages/sklearn/utils/deprecation.py:77: DeprecationWarning: Function log_multivariate_normal_density is deprecated; The function log_multivariate_normal_density is deprecated in 0.18 and will be removed in 0.20.

Omar

On Fri, Aug 2, 2019 at 11:39 AM kahlquist-brown notifications@github.com wrote:

I have a version running locally with the following: numpy==1.13.3 scikit-learn==0.19.1 matplotlib==2.1.0

Below I'll dump the result of pip freeze if there are other versions you are wondering about. You should generate warnings when using SWIFr_train.py with these versions installed (they are warning about the coming changes to sklearn in 0.20) but they should be harmless.

alabaster==0.7.10 anaconda-client==1.6.5 anaconda-navigator==1.6.9 anaconda-project==0.8.0 appnope==0.1.0 appscript==1.0.1 args==0.1.0 asn1crypto==0.22.0 astroid==1.5.3 astropy==2.0.2 awesome-slugify==1.6.5 Babel==2.5.0 backports-abc==0.5 backports.functools-lru-cache==1.4 backports.shutil-get-terminal-size==1.0.0 backports.ssl-match-hostname==3.5.0.1 beautifulsoup4==4.6.0 biopython==1.70 bitarray==0.8.1 bkcharts==0.2 blaze==0.11.3 bleach==2.0.0 bokeh==0.12.10 boto==2.48.0 Bottleneck==1.2.1 cdecimal==2.3 certifi==2017.7.27.1 cffi==1.10.0 chardet==3.0.4 click==6.7 clint==0.5.1 cloudpickle==0.4.0 clyent==1.2.2 colorama==0.3.9 configparser==3.5.0 contextlib2==0.5.5 cryptography==2.0.3 cycler==0.10.0 Cython==0.26.1 cytoolz==0.8.2 cyvcf2==0.8.7 dask==0.15.3 datashape==0.5.4 decorator==4.1.2 distributed==1.19.1 docopt==0.6.2 docutils==0.14 entrypoints==0.2.3 enum34==1.1.6 et-xmlfile==1.0.1 fastcache==1.0.2 filelock==2.0.12 Flask==0.12.2 Flask-Cors==3.0.3 funcsigs==1.0.2 functools32==3.2.3.post2 futures==3.1.1 gevent==1.2.2 glob2==0.5 gmpy2==2.0.8 greenlet==0.4.12 grin==1.2.1 h5py==2.7.0 heapdict==1.0.0 html5lib==0.999999999 idna==2.6 imageio==2.2.0 imagesize==0.7.1 intervaltree==2.1.0 ipaddress==1.0.18 ipykernel==4.6.1 ipython==5.4.1 ipython-genutils==0.2.0 ipywidgets==7.0.0 isort==4.2.15 itsdangerous==0.24 jdcal==1.3 jedi==0.10.2 Jinja2==2.9.6 jsonschema==2.6.0 jupyter-client==5.1.0 jupyter-console==5.2.0 jupyter-core==4.3.0 jupyterlab==0.27.0 jupyterlab-launcher==0.4.0 lazy-object-proxy==1.3.1 llvmlite==0.20.0 locket==0.2.0 lxml==4.1.0 MarkupSafe==1.0 matplotlib==2.1.0 mccabe==0.6.1 mistune==0.7.4 mpmath==0.19 msgpack-python==0.4.8 multipledispatch==0.4.9 navigator-updater==0.1.0 nbconvert==5.3.1 nbformat==4.4.0 networkx==1.11 nltk==3.2.4 nose==1.3.7 notebook==5.0.0 numba==0.35.0+6.gaa35fb1.dirty numexpr==2.6.2 numpy==1.13.3 numpydoc==0.7.0 odo==0.5.1 olefile==0.44 openpyxl==2.4.8 packaging==16.8 pandas==0.20.3 pandocfilters==1.4.2 partd==0.3.8 path.py==10.3.1 pathlib2==2.3.0 patsy==0.4.1 pep8==1.7.0 pexpect==4.2.1 pickleshare==0.7.4 Pillow==4.2.1 pkginfo==1.4.1 ply==3.10 prompt-toolkit==1.0.15 psutil==5.4.0 ptyprocess==0.5.2 py==1.4.34 PyAudio==0.2.7 pycodestyle==2.3.1 pycosat==0.6.2 pycparser==2.18 pycrypto==2.6.1 pycurl==7.43.0 pyflakes==1.6.0 Pygments==2.2.0 pylint==1.7.4 pyodbc==4.0.17 pyOpenSSL==17.2.0 pyparsing==2.2.0 PySocks==1.6.7 pytest==3.2.1 python-dateutil==2.6.1 pytz==2017.2 PyWavelets==0.5.2 PyYAML==3.12 pyzmq==16.0.2 QtAwesome==0.4.4 qtconsole==4.3.1 QtPy==1.3.1 regex==2018.2.21 requests==2.18.4 rope==0.10.5 ruamel-yaml==0.11.14 scandir==1.6 scikit-image==0.13.0 scikit-learn==0.19.1 scipy==0.19.1 seaborn==0.8 simplegeneric==0.8.1 singledispatch==3.4.0.3 six==1.11.0 snowballstemmer==1.2.1 sortedcollections==0.5.3 sortedcontainers==1.5.7 Sphinx==1.6.3 sphinxcontrib-websupport==1.0.1 spyder==3.2.4 SQLAlchemy==1.1.13 statsmodels==0.8.0 subprocess32==3.2.7 sympy==1.1.1 tables==3.4.2 tabulate==0.8.2 tblib==1.3.2 terminado==0.6 testpath==0.3.1 toolz==0.8.2 tornado==4.5.2 traitlets==4.3.2 typing==3.6.2 unicodecsv==0.14.1 Unidecode==0.4.21 urllib3==1.22 VCF-kit==0.1.6 wcwidth==0.1.7 webencodings==0.5.1 Werkzeug==0.12.2 widgetsnbextension==3.0.2 wrapt==1.10.11 xlrd==1.1.0 XlsxWriter==1.0.2 xlwings==0.11.4 xlwt==1.2.0 yahmm==1.1.2 zict==0.1.3

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_ramachandran-2Dlab_SWIFr_issues_1-3Femail-5Fsource-3Dnotifications-26email-5Ftoken-3DADNNQ2Z6QLD4XJUOXYCRWU3QCR5PHA5CNFSM4IIVKDI2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD3OREDA-23issuecomment-2D517804556&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=7W_uRJ1Ex1akebq7kJuPzdHVv4l433c929ichxeHP_4&s=wIOKIkxIRWyKLxpgCArAXPMTMaqqsws2mioU8MJbRlQ&e=, or mute the thread https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_notifications_unsubscribe-2Dauth_ADNNQ2232MLAN5LBQHTYKIDQCR5PHANCNFSM4IIVKDIQ&d=DwMCaQ&c=C3yme8gMkxg_ihJNXS06ZyWk4EJm8LdrrvxQb-Je7sw&r=1YnLN6ptmFumCtKMs62AHQdWF0mxKnIbu0uh8D6U1rE&m=7W_uRJ1Ex1akebq7kJuPzdHVv4l433c929ichxeHP_4&s=lp7t3NbYgKrUdZfwmarBxcand7JMMHsnTgx9N7qM_gg&e= .

-- Omar E. Cornejo, PhD Assistant Professor, School of Biological Sciences Washington State University Heald 429. PO Box 644236 Pullman, WA. 99164-4236

ghost commented 5 years ago

Yes, with the versions I specified you will generate warnings from sklearn. This will be resolved in a new version of SWIF(r).