search

scramblingbalam / Alta_Real

The python back end of a data product that aims to get at the truthiness of Trump's tweets. It uses PyStruct.EdgeFeatureGraphCRF to leverage the tree structure of twitter replies to gage veracity by the amount of support of denial that his tweets elicit.
MIT License
1 stars 0 forks source link

Install PyStruct #14

Open scramblingbalam opened 7 years ago

scramblingbalam commented 7 years ago

Install pyStruct a graph based learning library that seems to be the only one that supports Tree Structures for Conditional Random Fields (t-CRFs)

scramblingbalam commented 7 years ago

The directions for installing pyStruct on Anaconda don't work

scramblingbalam commented 7 years ago

The first problem is that a conda cvxopt package doesn't seem to be built by Continuum because
conda install cvxopt fails I found a built package on the Anaconda cloud at https://anaconda.org/omnia/cvxopt and the command line conda install -c omnia cvxopt=1.1.8 seems to have worked

scramblingbalam commented 7 years ago

The actual install of pyStruct failed though at least form https://anaconda.org/pypi/pystruct I tried pip install -i https://pypi.anaconda.org/pypi/simple pystruct & pip install pystruct and I got a long print out mostly installing and uninstalling due to a failure but it seems that the primary error was error: command 'gcc' failed: No such file or directory

scramblingbalam commented 7 years ago

used conda install mingw and pip install -i https://pypi.anaconda.org/pypi/simple pystruct install didn't fail

scramblingbalam commented 7 years ago

conda install -c omnia cvxopt=1.1.8 doesn't work with Anaconda, it was a little stupid to think it would but I wasn't aware that Omnia was a python distribution or whatever it is the cvxopt module shows up in via conda list but import cvxopt doesn't work in the code or in the interpreter

scramblingbalam commented 7 years ago

I just did pip install cvxopt it seems to work cvxopt imports fine in the interpreter but I'm worried that it may have a C implementation and a pure python implementation that takes over if a C compiler isn't found (I had this issue with Gensim) Which means that the runtime of t-CRFs might be rediculous

scramblingbalam commented 7 years ago

so far conda install mingw good pip install cvxopt works and pip install -i https://pypi.anaconda.org/pypi/simple pystruct works but pystruct code throws

ImportError: cannot import name check_arrays

This error is because the original version of PyStruct uses the old version of scikit-learn check_arrays which has been removed. Pystruct 2.4 solves this but the code at https://pypi.anaconda.org/pypi/simple pystruct is for 2.0 I need to figure out a way to get a newer version solved this before but didn't document how

scramblingbalam commented 7 years ago

Tried pip install git+https://github.com/pystruct/pystruct.git to install directly from github and git clone https://github.com/pystruct/pystruct.git + python setup.py install both throw errors relating to the C compiler

python27.lib(python27.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86' error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe' failed with exit status 1112

full error

C:\Users\scram\Documents\github\pystruct [master]> python setup.py install running install running bdist_egg running egg_info creating pystruct.egg-info writing requirements to pystruct.egg-info\requires.txt writing pystruct.egg-info\PKG-INFO writing top-level names to pystruct.egg-info\top_level.txt writing dependency_links to pystruct.egg-info\dependency_links.txt writing manifest file 'pystruct.egg-info\SOURCES.txt' reading manifest file 'pystruct.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '' under directory 'tests' warning: no previously-included files matching '.pyc' found under directory 'doc' warning: no previously-included files matching '.pyo' found under directory 'doc' warning: no previously-included files matching '.pyc' found under directory 'tests' warning: no previously-included files matching '*.pyo' found under directory 'tests' no previously-included directories found matching 'docs_build' no previously-included directories found matching 'docs\auto_examples' no previously-included directories found matching 'docs\generated' writing manifest file 'pystruct.egg-info\SOURCES.txt' installing library code to build\bdist.win-amd64\egg running install_lib running build_py creating build creating build\lib.win-amd64-2.7 creating build\lib.win-amd64-2.7\pystruct copying pystruct\plot_learning.py -> build\lib.win-amd64-2.7\pystruct copying pystruct__init.py -> build\lib.win-amd64-2.7\pystruct creating build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\downhill_simplex_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\frankwolfe_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\latent_structured_svm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\n_slack_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\one_slack_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\structured_perceptron.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\subgradient_latent_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\subgradient_ssvm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners\svm.py -> build\lib.win-amd64-2.7\pystruct\learners copying pystruct\learners__init.py -> build\lib.win-amd64-2.7\pystruct\learners creating build\lib.win-amd64-2.7\pystruct\inference copying pystruct\inference\common.py -> build\lib.win-amd64-2.7\pystruct\inference copying pystruct\inference\inference_methods.py -> build\lib.win-amd64-2.7\pystruct\inference copying pystruct\inference\linear_programming.py -> build\lib.win-amd64-2.7\pystruct\inference copying pystruct\inference\maxprod.py -> build\lib.win-amd64-2.7\pystruct\inference copying pystruct\inference__init.py -> build\lib.win-amd64-2.7\pystruct\inference creating build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\base.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\chain_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\edge_feature_graph_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\graph_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\grid_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\latent_graph_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\latent_grid_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\latent_node_crf.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\multilabel_svm.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\setup.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models\unstructured_svm.py -> build\lib.win-amd64-2.7\pystruct\models copying pystruct\models__init.py -> build\lib.win-amd64-2.7\pystruct\models creating build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils\graph.py -> build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils\graph_functions.py -> build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils\inference.py -> build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils\logging.py -> build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils\plotting.py -> build\lib.win-amd64-2.7\pystruct\utils copying pystruct\utils__init.py -> build\lib.win-amd64-2.7\pystruct\utils creating build\lib.win-amd64-2.7\pystruct\datasets copying pystruct\datasets\dataset_loaders.py -> build\lib.win-amd64-2.7\pystruct\datasets copying pystruct\datasets\synthetic_grids.py -> build\lib.win-amd64-2.7\pystruct\datasets copying pystruct\datasets\init.py -> build\lib.win-amd64-2.7\pystruct\datasets creating build\lib.win-amd64-2.7\pystruct\tests copying pystruct\tests\test_datasets.py -> build\lib.win-amd64-2.7\pystruct\tests copying pystruct\tests\test_libraries.py -> build\lib.win-amd64-2.7\pystruct\tests copying pystruct\tests\init__.py -> build\lib.win-amd64-2.7\pystruct\tests creating build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_binary_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_crammer_singer_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_edge_feature_graph_learning.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_frankwolfe_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_graph_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_latent_node_crf_learning.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_latent_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_n_slack_ssvm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_one_slack_ssvm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_perceptron.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_primal_dual.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_structured_perceptron.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_subgradient_latent_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\test_subgradient_svm.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners copying pystruct\tests\test_learners\init.py -> build\lib.win-amd64-2.7\pystruct\tests\test_learners creating build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_chain_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_directional_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_edge_feature_graph_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_graph_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_grid_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_latent_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_latent_node_crf.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\test_multilabel_problem.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models copying pystruct\tests\test_models\init.py -> build\lib.win-amd64-2.7\pystruct\tests\test_models creating build\lib.win-amd64-2.7\pystruct\tests\test_inference copying pystruct\tests\test_inference\test_exact_inference.py -> build\lib.win-amd64-2.7\pystruct\tests\test_inference copying pystruct\tests\test_inference\test_maxprod.py -> build\lib.win-amd64-2.7\pystruct\tests\test_inference copying pystruct\tests\test_inference\init__.py -> build\lib.win-amd64-2.7\pystruct\tests\test_inference creating build\lib.win-amd64-2.7\pystruct\tests\test_utils copying pystruct\tests\test_utils\test_utils_inference.py -> build\lib.win-amd64-2.7\pystruct\tests\test_utils copying pystruct\tests\test_utils\test_utils_logging.py -> build\lib.win-amd64-2.7\pystruct\tests\test_utils copying pystruct\tests\test_utils\init__.py -> build\lib.win-amd64-2.7\pystruct\tests\test_utils copying pystruct\inference_viterbi.c -> build\lib.win-amd64-2.7\pystruct\inference copying pystruct\datasets\letters.pickle -> build\lib.win-amd64-2.7\pystruct\datasets copying pystruct\datasets\scene.pickle -> build\lib.win-amd64-2.7\pystruct\datasets copying pystruct\datasets\snakes.pickle -> build\lib.win-amd64-2.7\pystruct\datasets running build_ext building 'pystruct.models.utils' extension creating build\temp.win-amd64-2.7 creating build\temp.win-amd64-2.7\Release creating build\temp.win-amd64-2.7\Release\src C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\scram\Anaconda2\lib\site-packages\numpy\core\include -IC:\Users\scram\Anaconda2\include -IC:\Users\scram\Anaconda2\PC /Tcsrc/utils.c /Fobuild\temp.win-amd64-2.7\Release\src/utils.obj utils.c src/utils.c(2979) : warning C4244: '=' : conversion from 'int64' to 'int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\scram\Anaconda2\libs /LIBPATH:C:\Users\scram\Anaconda2\PCbuild\amd64 /LIBPATH:C:\Users\scram\Anaconda2\PC\VS9.0\amd64 /EXPORT:initutils build\temp.win-amd64-2.7\Release\src/utils.obj /OUT:build\lib.win-amd64-2.7\pystruct\models\utils.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\src\utils.lib /MANIFESTFILE:build\temp.win-amd64-2.7\Release\src\utils.pyd.manifest python27.lib(python27.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86' error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe' failed with exit status 1112 C:\Users\scram\Documents\github\pystruct [master]> python setup.py install running install running bdist_egg running egg_info writing requirements to pystruct.egg-info\requires.txt writing pystruct.egg-info\PKG-INFO writing top-level names to pystruct.egg-info\top_level.txt writing dependency_links to pystruct.egg-info\dependency_links.txt reading manifest file 'pystruct.egg-info\SOURCES.txt' reading manifest template 'MANIFEST.in' warning: no files found matching '' under directory 'tests' warning: no previously-included files matching '.pyc' found under directory 'doc' warning: no previously-included files matching '.pyo' found under directory 'doc' warning: no previously-included files matching '.pyc' found under directory 'tests' warning: no previously-included files matching '*.pyo' found under directory 'tests' no previously-included directories found matching 'docs_build' no previously-included directories found matching 'docs\auto_examples' no previously-included directories found matching 'docs\generated' writing manifest file 'pystruct.egg-info\SOURCES.txt' installing library code to build\bdist.win-amd64\egg running install_lib running build_py running build_ext building 'pystruct.models.utils' extension C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\cl.exe /c /nologo /Ox /MD /W3 /GS- /DNDEBUG -IC:\Users\scram\Anaconda2\lib\site-packages\numpy\core\include -IC:\Users\scram\Anaconda2\include -IC:\Users\scram\Anaconda2\PC /Tcsrc/utils.c /Fobuild\temp.win-amd64-2.7\Release\src/utils.obj utils.c src/utils.c(2979) : warning C4244: '=' : conversion from 'int64' to 'int', possible loss of data C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe /DLL /nologo /INCREMENTAL:NO /LIBPATH:C:\Users\scram\Anaconda2\libs /LIBPATH:C:\Users\scram\Anaconda2\PCbuild\amd64 /LIBPATH:C:\Users\scram\Anaconda2\PC\VS9.0\amd64 /EXPORT:initutils build\temp.win-amd64-2.7\Release\src/utils.obj /OUT:build\lib.win-amd64-2.7\pystruct\models\utils.pyd /IMPLIB:build\temp.win-amd64-2.7\Release\src\utils.lib /MANIFESTFILE:build\temp.win-amd64-2.7\Release\src\utils.pyd.manifest python27.lib(python27.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86' error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe' failed with exit status 1112

scramblingbalam commented 7 years ago

Found MS_Visual studio installer in downloads so gonna try that then install again

scramblingbalam commented 7 years ago

Installer says installed so uninstalling and reinstalling

scramblingbalam commented 7 years ago

pip install pystruct==0.2.4 works IDK why pip doesn't point to the newest version but that's it

scramblingbalam commented 7 years ago

so these are the steps I took that seem to make PyStruct Note: I say seem to because I haven't gotten a model to work yet but I'm getting errors from the trian model about incorrect data structure

scramblingbalam commented 7 years ago

Steps to install PyStruct on Windows 10 using Anaconda conda install mingw may need to install Visual Studio 2008 as well pip install cvxopt pip install pystruct==0.2.4

scramblingbalam commented 7 years ago

AD3 installs without error using pip install -i https://pypi.anaconda.org/pypi/simple ad3 I then install PyStruct using pip install pystruct==0.2.4 strangely running from console works but running form spyder3 throws this error

File "C:\Users\scram\Anaconda2\lib\site-packages\pystruct\inference\inference_methods.py", line 354, in inference_ad3 import ad3

File "C:\Users\scram\Anaconda2\lib\site-packages\ad3init.py", line 1, in from factor_graph import PBinaryVariable, PFactorGraph, PMultiVariable

ImportError: DLL load failed: The specified procedure could not be found.

scramblingbalam commented 7 years ago

More problems with ad3 this time in the root py2.7 environment this could be that I changed something but I don't think so the error comes from code which doesn't seem to be in the current github master so I'm going to try to install directly for there: I'm modifying instructions from SO https://stackoverflow.com/questions/19042389/conda-installing-upgrading-directly-from-github pip install git+git://github.com/scrappy/scrappy@master+https://github.com/andre-martins/AD3 => pip install git+git://github.com/andre-martins/AD3@master

scramblingbalam commented 7 years ago

installation errored with this message:

python27.lib(python27.dll) : fatal error LNK1112: module machine type 'x64' conflicts with target machine type 'X86' error: command 'C:\Program Files (x86)\Microsoft Visual Studio 9.0\VC\Bin\link.exe' failed with exit status 1112

scramblingbalam commented 7 years ago

uninstalled Visual studio 9.0 and ran install and get this message: error: Microsoft Visual C++ 9.0 is required. Get it from http://aka.ms/vcpython27 went to http://aka.ms/vcpython27 and am going to install that and then try a gain

scramblingbalam commented 7 years ago

This installation seems to work fine for python 2.7 so the install would be: conda install mingw 'pip install cvxopt' 'pip install pystruct ' install C++ form http://aka.ms/vcpython27 pip install git+git://github.com/andre-martins/AD3@master

scramblingbalam commented 7 years ago

trained the model seems to work

scramblingbalam commented 7 years ago

I tested it with the Trump Data and it give predictions so pretty close to being done with #44

scramblingbalam commented 7 years ago

In order to asses if the C++ compiler from http://aka.ms/vcpython27 was working and to ensure I do go through the same process with python 3.5 I uninstalled all MS C++ related programs except the above and a 2008 distribution I got a warning that was a dependency of other programs now I'm going to try and get it working in python 3.5

scramblingbalam commented 7 years ago

When trying to install ad3 via spyder 3 python 3.5 command line ie. !pip install git+git://github.com/andre-martins/AD3@master I was directed to http://landinghub.visualstudio.com/visual-cpp-build-tools

scramblingbalam commented 7 years ago

This doesn't work out of the box, so to speak so I'll have to work at it