search

GeoscienceAustralia / uncover-ml

Machine Learning system for Geoscience Australia uncover project
Apache License 2.0
30 stars 20 forks source link

Requirements #105

Closed bluetyson closed 4 years ago

bluetyson commented 4 years ago

I am following the Windows 10 install instructions.

There are other requirements files, presumably need to do all of them?

Docs just have pip install -r requirements-pykrige.txt pip install .

currently running that order gets :-

\uncover-ml>pip install -r requirements-pykrige.txt Collecting Cython==0.29.13 Downloading Cython-0.29.13-cp37-cp37m-win_amd64.whl (1.7 MB) |████████████████████████████████| 1.7 MB 939 kB/s Collecting numpy==1.17.2 Downloading numpy-1.17.2-cp37-cp37m-win_amd64.whl (12.8 MB) |████████████████████████████████| 12.8 MB 3.3 MB/s Collecting scipy==1.3.1 Downloading scipy-1.3.1-cp37-cp37m-win_amd64.whl (30.3 MB) |████████████████████████████████| 30.3 MB 3.2 MB/s Collecting matplotlib==3.1.1 Downloading matplotlib-3.1.1-cp37-cp37m-win_amd64.whl (9.1 MB) |████████████████████████████████| 9.1 MB 233 kB/s Requirement already satisfied: pyparsing!=2.0.4,!=2.1.2,!=2.1.6,>=2.0.1 in c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncovermlold\lib\site-packages (from matplotlib==3.1.1->-r requirements-pykrige.txt (line 4)) (2.4.7) Collecting cycler>=0.10 Using cached cycler-0.10.0-py2.py3-none-any.whl (6.5 kB) Collecting kiwisolver>=1.0.1 Downloading kiwisolver-1.2.0-cp37-none-win_amd64.whl (57 kB) |████████████████████████████████| 57 kB 1.3 MB/s Collecting python-dateutil>=2.1 Using cached python_dateutil-2.8.1-py2.py3-none-any.whl (227 kB) Requirement already satisfied: six in c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncovermlold\lib\site-packages (from cycler>=0.10->matplotlib==3.1.1->-r requirements-pykrige.txt (line 4)) (1.15.0) ERROR: unmixing 0.2.4.dev0 requires Pillow>=2.3.0, which is not installed. ERROR: unmixing 0.2.4.dev0 requires pysptools>=0.13.5, which is not installed. ERROR: unmixing 0.2.4.dev0 requires scikit-learn>=0.21.3, which is not installed. Installing collected packages: Cython, numpy, scipy, cycler, kiwisolver, python-dateutil, matplotlib Attempting uninstall: numpy Found existing installation: numpy 1.18.5 Uninstalling numpy-1.18.5: Successfully uninstalled numpy-1.18.5 Successfully installed Cython-0.29.13 cycler-0.10.0 kiwisolver-1.2.0 matplotlib-3.1.1 numpy-1.17.2 python-dateutil-2.8.1 scipy-1.3.1

\uncover-ml ERROR: Command errored out with exit status 1: command: 'C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\uncovermlold\python.exe' -c 'import sys, setuptools, tokenize; sys.argv[0] = '"'"'C:\Users\rscott\AppData\Local\Temp\pip-req-build-bi0vcvf6\setup.py'"'"'; file='"'"'C:\Users\rscott\AppData\Local\Temp\pip-req-build-bi0vcvf6\setup.py'"'"';f=getattr(tokenize, '"'"'open'"'"', open)(file);code=f.read().replace('"'"'\r\n'"'"', '"'"'\n'"'"');f.close();exec(compile(code, file, '"'"'exec'"'"'))' egg_info --egg-base 'C:\Users\rscott\AppData\Local\Temp\pip-pip-egg-info-83razedx' cwd: C:\Users\rscott\AppData\Local\Temp\pip-req-build-bi0vcvf6\ Complete output (13 lines): fatal: No tags can describe '478620809f8e82f459f3962f2478631a62f9bee1'. Try --always, or create some tags. Traceback (most recent call last): File "", line 1, in File "C:\Users\rscott\AppData\Local\Temp\pip-req-build-bi0vcvf6\setup.py", line 57, in version=git_desc(), File "C:\Users\rscott\AppData\Local\Temp\pip-req-build-bi0vcvf6\setup.py", line 49, in git_desc return subprocess.check_output(desc).decode().strip() File "C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\uncovermlold\lib\subprocess.py", line 376, in check_output **kwargs).stdout File "C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\uncovermlold\lib\subprocess.py", line 468, in run output=stdout, stderr=stderr) subprocess.CalledProcessError: Command '['git', 'describe', '--tags', '--abbrev=0']' returned non-zero exit status 128.

ERROR: Command errored out with exit status 1: python setup.py egg_info Check the logs for full command output.

brenmous commented 4 years ago

Looks like an issue caused by using git describe to manage the uncover-ml version number. Are you able to run git --version and tell me your git version? Apparently git describe is only in versions of git >2.4

brenmous commented 4 years ago

And to answer your question requirements-pykrige.txt and requirements.txt are enough to run uncover-ml. requirements-dev.txt is intended if you're going to develop and run tests.

bluetyson commented 4 years ago

Installed pysptools manually.

Also, rasterio installed in conda instructions and it is in the requirements.txt file. I just ran that the -dev requirements file too to see how it would go.

Pip rasterio installs can give GDAL-CONFIG environment errors too in this sort of istuation.

bluetyson commented 4 years ago

git version 2.25.1.windows.1 I am using git bash

brenmous commented 4 years ago

Thanks for pointing out that duplication with rasterio. We've been running exclusively on Linux/NCI and this is the first pass at Windows compatibility so it's appreciated. For the time being I'm going to disable the git describe command and hardcode the version number (commit).

Try pulling the change and see how it goes.

bluetyson commented 4 years ago

The install . worked (when I took rasterio out of requirements) - not sure what else that will affect currently, going to have rasterio issues now probably with pip/conda varieties.

brenmous commented 4 years ago

Were you getting errors about installing rasterio when pip install failed?

Also did you activate the conda venv in git bash before running the pip install? If the pip install is run within the conda venv it's going to ignore the rasterio install directive because it already exists.

bluetyson commented 4 years ago

Yeah, but it could be because of install it again (in requirements) or something I did. I am trying a new environment again. However, from requirements-pykrige ERROR: uncover-ml 0.3.1 has requirement click==7.0, but you'll have click 7.1.2 which is incompatible.

bluetyson commented 4 years ago

(uncoverml2) C:\Users\rscott\OneDrive - OZ Minerals\Exploration\uncover-ml>uncoverml learn configs\windows_random_forest.yaml Traceback (most recent call last): File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\uncoverml2\Scripts\uncoverml.exe__main.py", line 4, in File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\site-packages\uncoverml\scripts\init.py", line 20, in import uncoverml.features File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\site-packages\uncoverml\features.py", line 9, in from uncoverml import mpiops, patch, transforms, diagnostics File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\site-packages\uncoverml\diagnostics.py", line 11, in import rasterio File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\uncoverml2\lib\site-packages\rasterio\init__.py", line 22, in from rasterio._base import gdal_version ImportError: DLL load failed: The specified procedure could not be found.

bluetyson commented 4 years ago

Same thing.

bluetyson commented 4 years ago

Which I have seen before and I think I solved it with an earlier version of rasterio. Not sure that is doable here?

bluetyson commented 4 years ago

Although this is ok: (uncoverml2) C:\Users\rscott\OneDrive - OZ Minerals\Exploration\uncover-ml>python Python 3.7.1 | packaged by conda-forge | (default, Mar 13 2019, 13:32:59) [MSC v.1900 64 bit (AMD64)] :: Anaconda, Inc. on win32 Type "help", "copyright", "credits" or "license" for more information.

import rasterio from rasterio import gdal_version quit Use quit() or Ctrl-Z plus Return to exit quit()

brenmous commented 4 years ago

Not sure what's happening with Click, unless one of the packages in the pykrige deps is installing Click 7.1.2. You can run pip install click==7.0 --force-reinstall in the venv before running those pip installs.

Given it works in the REPL my first thought is interpreter issues and possibly that the pip being used to install uncoverml isn't linked with the default python interpreter in that venv. Could you try installing Uncoverml with python -m pip install . and see how that goes?

I'll dig into this again tomorrow. For now here's also the output of conda list in the Win10 environment where I've got it working, to confirm the gdal/libgdal/rasterio versions are correct:

affine                    2.3.0                      py_0
attrs                     19.3.0                     py_0
blas                      1.0                         mkl
bzip2                     1.0.8                he774522_0
ca-certificates           2020.1.1                      0
certifi                   2020.6.20                py37_0
cfitsio                   3.470                he774522_5
click                     7.0                      pypi_0    pypi
click-plugins             1.1.1                      py_0
cligj                     0.5.0                    py37_0
curl                      7.67.0               h2a8f88b_0
cycler                    0.10.0                   pypi_0    pypi
cython                    0.29.13                  pypi_0    pypi
decorator                 4.4.2                    pypi_0    pypi
eli5                      0.10.1                   pypi_0    pypi
expat                     2.2.5                he025d50_0
fiona                     1.8.13.post1     py37hd760492_0
freexl                    1.0.5                hfa6e2cd_0
gdal                      3.0.2            py37hdf43c64_0
geopandas                 0.6.3                    pypi_0    pypi
geos                      3.8.0                h33f27b4_0
geotiff                   1.5.1                h5770a2b_1
git                       2.23.0               h6bb4b03_0
hdf4                      4.2.13               h712560f_2
hdf5                      1.10.4               h7ebc959_0
icc_rt                    2019.0.0             h0cc432a_1
icu                       58.2                 ha925a31_3
imageio                   2.8.0                    pypi_0    pypi
impi_rt                   2019.7                intel_216    intel
intel-openmp              2020.1                      216
intelpython               2020.1                        0    intel
jinja2                    2.11.2                   pypi_0    pypi
joblib                    0.15.1                   pypi_0    pypi
jpeg                      9b                   hb83a4c4_2
kealib                    1.4.7                h07cbb95_6
kiwisolver                1.2.0                    pypi_0    pypi
krb5                      1.16.4               hc04afaa_0
libboost                  1.67.0               hd9e427e_4
libcurl                   7.67.0               h2a8f88b_0
libgdal                   3.0.2                h1155b67_0
libiconv                  1.15                 h1df5818_7
libkml                    1.3.0                he5f2a48_4
libnetcdf                 4.6.1                h411e497_2
libpng                    1.6.37               h2a8f88b_0
libpq                     11.2                 h3235a2c_0
libspatialite             4.3.0a               h7ffb84d_0
libssh2                   1.9.0                h7a1dbc1_1
libtiff                   4.1.0                h56a325e_0
libxml2                   2.9.10               h464c3ec_1
lz4-c                     1.8.1.2              h2fa13f4_0
m2w64-expat               2.1.1                         2
m2w64-gcc-libgfortran     5.3.0                         6
m2w64-gcc-libs            5.3.0                         7
m2w64-gcc-libs-core       5.3.0                         7
m2w64-gettext             0.19.7                        2
m2w64-gmp                 6.1.0                         2
m2w64-libiconv            1.14                          6
m2w64-libwinpthread-git   5.0.0.4634.697f757               2
m2w64-xz                  5.2.2                         2
markupsafe                1.1.1                    pypi_0    pypi
matplotlib                3.1.1                    pypi_0    pypi
mkl                       2020.1                      216
mkl-service               2.3.0            py37hb782905_0
mkl_fft                   1.1.0            py37h45dec08_0
mkl_random                1.1.1            py37h47e9c7a_0
mpi4py                    3.0.3                    py37_6    intel
msys2-conda-epoch         20160418                      1
munch                     2.5.0                      py_0
networkx                  2.4                      pypi_0    pypi
numpy                     1.17.2                   pypi_0    pypi
openjpeg                  2.3.0                h5ec785f_1
openssl                   1.1.1g               he774522_0
pandas                    0.25.1                   pypi_0    pypi
pcre                      8.44                 ha925a31_0
pillow                    7.0.0                    pypi_0    pypi
pip                       20.1.1                   py37_1
postgresql                11.2                 h3235a2c_0
ppretty                   1.3                      pypi_0    pypi
proj                      6.2.1                h9f7ef89_0
pykrige                   1.3.0                    pypi_0    pypi
pyparsing                 2.4.7                      py_0
pyproj                    2.4.1                    pypi_0    pypi
pyshp                     1.2.3                    pypi_0    pypi
python                    3.7.7                h81c818b_4
python-dateutil           2.8.1                    pypi_0    pypi
python-graphviz           0.14                     pypi_0    pypi
pytz                      2020.1                   pypi_0    pypi
pywavelets                1.1.1                    pypi_0    pypi
pyyaml                    5.1.2                    pypi_0    pypi
rasterio                  1.1.0            py37h039b02d_0
revrand                   1.0.0                    pypi_0    pypi
scikit-image              0.15.0                   pypi_0    pypi
scikit-learn              0.22.2                   pypi_0    pypi
scipy                     1.3.1                    pypi_0    pypi
seaborn                   0.9.0                    pypi_0    pypi
setuptools                47.3.1                   py37_0
shapely                   1.7.0            py37h210f175_0
simplekml                 1.3.3                    pypi_0    pypi
six                       1.15.0                     py_0
snuggs                    1.4.7                      py_0
sqlite                    3.32.3               h2a8f88b_0
tabulate                  0.8.7                    pypi_0    pypi
tbb                       2018.0.5             he980bc4_0
tiledb                    1.6.3                h7b000aa_0
tk                        8.6.10               he774522_0
uncover-ml                0.3.1                     dev_0    <develop>
vc                        14.1                 h0510ff6_4
vs2015_runtime            14.16.27012          hf0eaf9b_2
wheel                     0.34.2                   py37_0
wincertstore              0.2                      py37_0
xerces-c                  3.2.2                ha925a31_0
xgboost                   0.90                     pypi_0    pypi
xz                        5.2.5                h62dcd97_0
zlib                      1.2.11               h62dcd97_4
zstd                      1.3.7                h508b16e_0
bluetyson commented 4 years ago

Ok, I will have a go.

bluetyson commented 4 years ago 
    # Name                    Version                   Build  Channel
    affine                    2.3.0                      py_0    conda-forge
    attrs                     19.3.0                     py_0    conda-forge
    boost-cpp                 1.72.0               h2ba7cf6_1    conda-forge
    bzip2                     1.0.8                hfa6e2cd_2    conda-forge
    ca-certificates           2020.6.20            hecda079_0    conda-forge
    cairo                     1.16.0            h8a5e50b_1003    conda-forge
    certifi                   2020.6.20        py37hc8dfbb8_0    conda-forge
    cfitsio                   3.470                he774522_5    conda-forge
    click                     7.0                      pypi_0    pypi
    click-plugins             1.1.1                      py_0    conda-forge
    cligj                     0.5.0                      py_0    conda-forge
    curl                      7.71.0               h4b64cdc_0    conda-forge
    cycler                    0.10.0                   pypi_0    pypi
    cython                    0.29.13                  pypi_0    pypi
    decorator                 4.4.2                    pypi_0    pypi
    eli5                      0.10.1                   pypi_0    pypi
    expat                     2.2.9                he025d50_2    conda-forge
    fiona                     1.8.13           py37hef9e828_1    conda-forge
    freetype                  2.10.2               hd328e21_0    conda-forge
    freexl                    1.0.5             hd288d7e_1002    conda-forge
    gdal                      3.0.4            py37hd44be9e_9    conda-forge
    geopandas                 0.6.3                    pypi_0    pypi
    geos                      3.8.1                he025d50_0    conda-forge
    geotiff                   1.5.1               h3d29ae3_10    conda-forge
    gettext                   0.19.8.1          hb01d8f6_1002    conda-forge
    glib                      2.65.0               he4de6d7_0    conda-forge
    hdf4                      4.2.13            hf8e6fe8_1003    conda-forge
    hdf5                      1.10.6          nompi_ha405e13_100    conda-forge
    icu                       64.2                 he025d50_1    conda-forge
    imageio                   2.8.0                    pypi_0    pypi
    impi_rt                   2019.7                intel_216    intel
    intel-openmp              2020.1                      216
    intelpython               2020.1                        0    intel
    jinja2                    2.11.2                   pypi_0    pypi
    joblib                    0.16.0                   pypi_0    pypi
    jpeg                      9d                   he774522_0    conda-forge
    kealib                    1.4.13               h3b59ab9_1    conda-forge
    kiwisolver                1.2.0                    pypi_0    pypi
    krb5                      1.17.1               hc04afaa_1    conda-forge
    libblas                   3.8.0                    16_mkl    conda-forge
    libcblas                  3.8.0                    16_mkl    conda-forge
    libcurl                   7.71.0               h4b64cdc_0    conda-forge
    libffi                    3.2.1             h6538335_1007    conda-forge
    libgdal                   3.0.4                h6f60a84_9    conda-forge
    libiconv                  1.15              hfa6e2cd_1006    conda-forge
    libkml                    1.3.0             h7e985d0_1011    conda-forge
    liblapack                 3.8.0                    16_mkl    conda-forge
    libnetcdf                 4.7.4           nompi_h256d12c_104    conda-forge
    libpng                    1.6.37               hfe6a214_1    conda-forge
    libpq                     12.2                 hd9aa61d_1    conda-forge
    libspatialite             4.3.0a            h51df0ed_1038    conda-forge
    libssh2                   1.9.0                h3235a2c_2    conda-forge
    libtiff                   4.1.0                h885aae3_6    conda-forge
    libwebp-base              1.1.0                hfa6e2cd_3    conda-forge
    libxml2                   2.9.10               h5d81f1c_1    conda-forge
    lz4-c                     1.9.2                h62dcd97_1    conda-forge
    m2w64-expat               2.1.1                         2
    m2w64-gcc-libgfortran     5.3.0                         6
    m2w64-gcc-libs            5.3.0                         7
    m2w64-gcc-libs-core       5.3.0                         7
    m2w64-gettext             0.19.7                        2
    m2w64-gmp                 6.1.0                         2
    m2w64-libiconv            1.14                          6
    m2w64-libwinpthread-git   5.0.0.4634.697f757               2
    m2w64-xz                  5.2.2                         2
    markupsafe                1.1.1                    pypi_0    pypi
    matplotlib                3.1.1                    pypi_0    pypi
    mkl                       2020.1                      216
    mpi4py                    3.0.3                    py37_6    intel
    msys2-conda-epoch         20160418                      1
    munch                     2.5.0                      py_0    conda-forge
    networkx                  2.4                      pypi_0    pypi
    numpy                     1.17.2                   pypi_0    pypi
    openjpeg                  2.3.1                h57dd2e7_3    conda-forge
    openssl                   1.1.1g               he774522_0    conda-forge
    pandas                    0.25.1                   pypi_0    pypi
    pcre                      8.44                 h6538335_0    conda-forge
    pillow                    7.0.0                    pypi_0    pypi
    pip                       20.1.1                     py_1    conda-forge
    pixman                    0.38.0            hfa6e2cd_1003    conda-forge
    poppler                   0.87.0               h0cd1227_1    conda-forge
    poppler-data              0.4.9                         1    conda-forge
    postgresql                12.2                 he14cc48_1    conda-forge
    ppretty                   1.3                      pypi_0    pypi
    proj                      7.0.0                haa36216_3    conda-forge
    pykrige                   1.3.0                    pypi_0    pypi
    pyparsing                 2.4.7              pyh9f0ad1d_0    conda-forge
    pyproj                    2.4.1                    pypi_0    pypi
    pyshp                     1.2.3                    pypi_0    pypi
    pysptools                 0.15.0                   pypi_0    pypi
    python                    3.7.1             h9460c21_1003    conda-forge
    python-dateutil           2.8.1                    pypi_0    pypi
    python-graphviz           0.14                     pypi_0    pypi
    python_abi                3.7                     1_cp37m    conda-forge
    pytz                      2020.1                   pypi_0    pypi
    pywavelets                1.1.1                    pypi_0    pypi
    pyyaml                    5.1.2                    pypi_0    pypi
    rasterio                  1.1.5            py37h02db82b_0    conda-forge
    revrand                   1.0.0                    pypi_0    pypi
    scikit-image              0.15.0                   pypi_0    pypi
    scikit-learn              0.22.2                   pypi_0    pypi
    scipy                     1.3.1                    pypi_0    pypi
    seaborn                   0.9.0                    pypi_0    pypi
    setuptools                47.3.1           py37hc8dfbb8_0    conda-forge
    shapely                   1.7.0            py37he1cf020_3    conda-forge
    simplekml                 1.3.3                    pypi_0    pypi
    six                       1.15.0             pyh9f0ad1d_0    conda-forge
    snuggs                    1.4.7                      py_0    conda-forge
    sqlite                    3.32.3               he774522_0    conda-forge
    tabulate                  0.8.7                    pypi_0    pypi
    tbb                       2020.1               he980bc4_0    conda-forge
    tiledb                    1.7.7                h0b90766_3    conda-forge
    tk                        8.6.10               hfa6e2cd_0    conda-forge
    uncover-ml                0.3.1                    pypi_0    pypi
    vc                        14.1                 h869be7e_1    conda-forge
    vs2015_runtime            14.16.27012          h30e32a0_2    conda-forge
    wheel                     0.34.2                     py_1    conda-forge
    wincertstore              0.2                   py37_1003    conda-forge
    xerces-c                  3.2.2             h6538335_1004    conda-forge
    xgboost                   0.90                     pypi_0    pypi
    xz                        5.2.5                h2fa13f4_0    conda-forge
    zlib                      1.2.11            h2fa13f4_1006    conda-forge
    zstd                      1.4.4                h9f78265_3    conda-forge
bluetyson commented 4 years ago

so rasterio and gdal from conda, not conda-forge is a difference, anyway

bluetyson commented 4 years ago

Same error with the -m install.

bluetyson commented 4 years ago

tried another environment with your python, gdal, rasterio, fiona, same thing, so odd.

bluetyson commented 4 years ago

Weird gdal thing to work out as other working environments with gdal/rasterio/fiona on this machine, latest one was a python 3.8, but have 3.7 as well.

bluetyson commented 4 years ago

Ok, did yet another environment and installed gdal rasterio fiona and python 3.7.7 on environment creation. Then did the rest. That seems ok now. So one for tomororw :- (gdaltest) C:\Users\rscott\OneDrive - OZ Minerals\Exploration\uncover-ml>uncoverml learn configs\windows_random_forest.yaml c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\sklearn\utils\deprecation.py:144: FutureWarning: The sklearn.metrics.scorer module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.metrics. Anything that cannot be imported from sklearn.metrics is now part of the private API. warnings.warn(message, FutureWarning) c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\sklearn\utils\deprecation.py:144: FutureWarning: The sklearn.feature_selection.base module is deprecated in version 0.22 and will be removed in version 0.24. The corresponding classes / functions should instead be imported from sklearn.feature_selection. Anything that cannot be imported from sklearn.feature_selection is now part of the private API. warnings.warn(message, FutureWarning) +4s WARNING uncoverml.config [P0]: 'features' are required even when loading from pickled data - this is a work around for getting image specifications. Needs to be fixed. Uncaught exception on processor 0 Traceback (most recent call last): File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\runpy.py", line 193, in _run_module_as_main "main", mod_spec) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\runpy.py", line 85, in _run_code exec(code, run_globals) File "C:\Users\rscott\AppData\Local\Continuum\anaconda3\envs\gdaltest\Scripts\uncoverml.exe__main.py", line 7, in sys.exit(cli()) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 764, in call return self.main(args, kwargs) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 717, in main rv = self.invoke(ctx) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 1137, in invoke return _process_result(sub_ctx.command.invoke(sub_ctx)) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 956, in invoke return ctx.invoke(self.callback, ctx.params) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\click\core.py", line 555, in invoke return callback(args, **kwargs) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\scripts\init__.py", line 142, in learn learn_cli.main(config_file, partitions) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\scripts\learn_cli.py", line 44, in main training_data, oos_data = _load_data(config, partitions) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\scripts\learn_cli.py", line 96, in _load_data bounds = ls.geoio.get_image_bounds(config) File "c:\users\rscott\appdata\local\continuum\anaconda3\envs\gdaltest\lib\site-packages\uncoverml\geoio.py", line 419, in get_image_bounds image_file = config.feature_sets[0].files[0] IndexError: list index out of range

(gdaltest) C:\Users\rscott\OneDrive - OZ Minerals\Exploration\uncover-ml>

bluetyson commented 4 years ago

Had a look print("CONFIG FILE: ", config_file) config = ls.config.Config(config_file, learning=True) print(config)

CONFIG FILE: configs\windows_random_forest.yaml +2s WARNING uncoverml.config [P0]: 'features' are required even when loading from pickled data - this is a work around for getting image specifications. Needs to be fixed. <uncoverml.config.Config object at 0x0000026EE6B435C8> <uncoverml.config.Config object at 0x0000026EE6B435C8>

So exception there by the looks.

brenmous commented 4 years ago

Good to see the install is working.

It's not finding the feature files. Check that you've modified the paths in windows_random_forest.yaml to point to your uncover-ml directory so the test data can be found. I'm working on setting up env vars for this and will add some better error messaging.

bluetyson commented 4 years ago

Thanks - good old user forgot a hyphen problem.

r2_score        = -3.778909672648356
expvar  = -2.5983654641788725
smse    = 4.778909672648356
lins_ccc        = 0.30319986454845826
mll     = 9.025337146538163
r2_score_transformed    = -1.0490291022967688
expvar_transformed      = -0.45637290601685143
smse_transformed        = 2.0490291022967684
lins_ccc_transformed    = 0.18116475680790609
mll_transformed = 2.7658562876565145
adjusted_r2_score       = 15.137460144207907
adjusted_r2_score_transformed   = 6.773031101507955

+19s INFO uncoverml.scripts.learn_cli [P0]: Learning full multirandomforest model
+19s INFO uncoverml.models [P0]: training forest 0 using process 0
+19s INFO uncoverml.models [P0]: training forest 1 using process 0
+19s INFO uncoverml.models [P0]: training forest 2 using process 0
+19s INFO uncoverml.models [P0]: training forest 3 using process 0
+19s INFO uncoverml.models [P0]: training forest 4 using process 0
+19s INFO uncoverml.models [P0]: training forest 5 using process 0
+19s INFO uncoverml.models [P0]: training forest 6 using process 0
+19s INFO uncoverml.models [P0]: training forest 7 using process 0
+19s INFO uncoverml.models [P0]: training forest 8 using process 0
+19s INFO uncoverml.models [P0]: training forest 9 using process 0
+19s INFO uncoverml.models [P0]: training forest 10 using process 0
+19s INFO uncoverml.models [P0]: training forest 11 using process 0
+19s INFO uncoverml.models [P0]: training forest 12 using process 0
+19s INFO uncoverml.models [P0]: training forest 13 using process 0
+19s INFO uncoverml.models [P0]: training forest 14 using process 0
+19s INFO uncoverml.models [P0]: training forest 15 using process 0
+19s INFO uncoverml.models [P0]: training forest 16 using process 0
+19s INFO uncoverml.models [P0]: training forest 17 using process 0
+19s INFO uncoverml.models [P0]: training forest 18 using process 0
+19s INFO uncoverml.models [P0]: training forest 19 using process 0
+19s INFO uncoverml.scripts [P0]: Resource usage not yet implemented on Windows. Setting memory usage as 0.
+19s INFO uncoverml.scripts.learn_cli [P0]: Finished! Total mem = 0.0 GB
bluetyson commented 4 years ago

This is the basic setup I used. conda create --name gdaltest gdal rasterio fiona python==3.7.7

Then just followed through, ignoring pysptools for the time being. There's a minor possibility a VPN did something weird to python package downloads too (perhaps). E.d. DLL oddness.

bluetyson commented 4 years ago

The multiprocessing example gives some exception looking reports like before, is that a possible path error ? :-

+14s INFO uncoverml.scripts.learn_cli [P0]: Learning full multirandomforest model +14s INFO uncoverml.models [P0]: training forest 0 using process 0 +14s INFO uncoverml.models [P1]: training forest 5 using process 1 +14s INFO uncoverml.models [P2]: training forest 10 using process 2 +14s INFO uncoverml.models [P3]: training forest 15 using process 3 +14s INFO uncoverml.models [P2]: training forest 11 using process 2 +14s INFO uncoverml.models [P3]: training forest 16 using process 3 +14s INFO uncoverml.models [P0]: training forest 1 using process 0 +14s INFO uncoverml.models [P1]: training forest 6 using process 1 +14s INFO uncoverml.models [P2]: training forest 12 using process 2 +14s INFO uncoverml.models [P3]: training forest 17 using process 3 +14s INFO uncoverml.models [P1]: training forest 7 using process 1 +14s INFO uncoverml.models [P0]: training forest 2 using process 0 +14s INFO uncoverml.models [P3]: training forest 18 using process 3 +14s INFO uncoverml.models [P2]: training forest 13 using process 2 +14s INFO uncoverml.models [P1]: training forest 8 using process 1 +14s INFO uncoverml.models [P0]: training forest 3 using process 0 +14s INFO uncoverml.models [P2]: training forest 14 using process 2 +14s INFO uncoverml.models [P1]: training forest 9 using process 1 +14s INFO uncoverml.models [P0]: training forest 4 using process 0 +14s INFO uncoverml.models [P3]: training forest 19 using process 3 +14s INFO uncoverml.scripts [P0]: Resource usage not yet implemented on Windows. Setting memory usage as 0. +14s INFO uncoverml.scripts.learn_cli [P0]: Finished! Total mem = 0.0 GB <uncoverml.config.Config object at 0x0000020455F85448> <uncoverml.config.Config object at 0x000001A658CF5248> <uncoverml.config.Config object at 0x0000017BCD775308> <uncoverml.config.Config object at 0x0000017D4D02B288>

brenmous commented 4 years ago

Cheers for the setup, I'll add that as an option for installing via the command line. Isn't that the print you were doing before of the config?

bluetyson commented 4 years ago

No, that is at the end of doing :- mpiexec -n 4 uncoverml learn configs\windows_random_forest.yaml

So could it be mpi path problem? (given 4 of them?)

brenmous commented 4 years ago

I'm not sure, the output you posted is normal operation - what part is the exception looking report?

brenmous commented 4 years ago

Closing as the install has worked but feel free to open another issue regarding the output you mentioned.

bluetyson commented 4 years ago

Ok, this just looked like the exception I was getting earlier <uncoverml.config.Config object at 0x0000020455F85448> <uncoverml.config.Config object at 0x000001A658CF5248> <uncoverml.config.Config object at 0x0000017BCD775308> <uncoverml.config.Config object at 0x0000017D4D02B288>

but if this is standard 'this is the config object' output, then all is good!

brenmous commented 4 years ago

Ah right. That's not an exception, that's due to:

print("CONFIG FILE: ", config_file)
config = ls.config.Config(config_file, learning=True)
print(config)

or any other print(config) that may have been added. Printing an object in python outputs its name and memory address.

bluetyson commented 4 years ago

Yes, true, thanks!