test installation - Githubissues

smnorris commented 7 years ago

click recommends using setup.py, this probably removes need for requirements.txt http://click.pocoo.org/5/setuptools/

ateucher commented 7 years ago

Let me know if you want me to test this out as well.

smnorris commented 7 years ago

Definitely

ateucher commented 7 years ago

When I first ran pip install -e git+git://github.com/smnorris/conservationlands.git, I got this error:

-editable=git+git://github.com/smnorris/conservationlands.git is not the right format; it must have #egg=Package

So I ran pip install -e git+git://github.com/smnorris/conservationlands.git#egg=Package

And got this:

Obtaining Package from git+git://github.com/smnorris/conservationlands.git#egg=Package
  Cloning git://github.com/smnorris/conservationlands.git to c:\_dev\src\package

fatal: remote error:
  Repository not found.

Command "git clone -q git://github.com/smnorris/conservationlands.git C:\_dev\src\package" failed with error code 128 in None

I have my PAT set up for https authentication, so I tried: pip install -e git+https://github.com/smnorris/conservationlands.git#egg=Package and got:

Obtaining Package from git+https://github.com/smnorris/conservationlands.git#egg=Package
  Cloning https://github.com/smnorris/conservationlands.git to c:\_dev\conservationlands\src\package
    Complete output from command python setup.py egg_info:
    error in conservationlands setup command: 'install_requires' must be a string or list of strings containing valid project/version requirement specifiers

    ----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in C:\_dev\conservationlands\src\package\

smnorris commented 7 years ago

I've updated the README and the method of installation, it works for me at the moment but please do test on your end. Simply cloning the repo and installing all dependencies is now the method (rather than installing the script itself).

Just for my benefit (if I'm trying to package a similar click script in the future) these are the issues I worked through when packaging:

getting pip to handle Python dependencies from github via setup.py is documented but I couldn't get this to work, perhaps I wasn't forming the dependency link github urls quite right.

install_requires=[
        'Click',
        'Requests',
        'Fiona',
        'pgdb',
        'bcdata],
dependency_links =['git+https://github.com/smnorris/bcdata#egg=bcdata',
    'git+https://github.com/smnorris/pgdb#egg=pgdb']

installing just the script (rather than as a package) as noted here http://click.pocoo.org/5/setuptools/ works, but only the single .py file is installed. I moved the contents of utils.py and config.py into the main script to take care of this, but of course the sql folder is ignored by pip. Handling the non python files with pip would take more fiddling... so on further reflection, while installing the script might be handy, it isn't necessary and perhaps not worth the time struggling with the packaging tools
however, if I was to continue with tidying the packaging it might be cleanest to remove the pgdb dependency, and create postgres functions from the sql files as a pre installation step rather than trying to include them in the python package.

ateucher commented 7 years ago

Install seemed to work flawlessly, all dependencies were installed. But when I run conservationlands --help, I get command not found

smnorris commented 7 years ago

Yes- I just realized that the documentation still presumes the script is installed.

try: python conservationlands.py --help

Also, I've just pushed a fix to the download function, you should get the latest before using it.

ateucher commented 7 years ago

Ok, install worked. I just have to fix a fiona problem - I don't think it's finding my gdal libraries

ateucher commented 7 years ago

Does it matter if I've got gdal 1.x or 2.x?

smnorris commented 7 years ago

Probably not but I'm on 1.x at the moment, I'd test with that first

smnorris commented 7 years ago

The only reason Fiona is used is because I don't trust all the BCGW downloads to have the same layer name pattern within the downloaded gdb. Maybe they do, I haven't checked.

ateucher commented 7 years ago

Seems like a smart thing to check. So fiona and gdal are happy now...

This works: python conservationlands.py download --help

But this happens:

$ python conservationlands.py download --email andy.teucher@gov.bc.ca           
Traceback (most recent call last):
  File "conservationlands.py", line 499, in <module>
    cli()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "conservationlands.py", line 319, in download
    db = pgdb.connect()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\pgdb\__init__.py", line 13, in connect
    return Database(url, schema)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\pgdb\database.py", line 21, in __init__
    u = urlparse.urlparse(url)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\urlparse.py", line 143, in urlparse
    tuple = urlsplit(url, scheme, allow_fragments)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\urlparse.py", line 182, in urlsplit
    i = url.find(':')
AttributeError: 'NoneType' object has no attribute 'find'

Something with pgdb it seems - not finding my postgis installation?

smnorris commented 7 years ago

Excellent, almost there. That is another oversight, I'll add a default db url to the config

ateucher commented 7 years ago

Awesome. I think I can grok the error there now.

ateucher commented 7 years ago

Ok, after some fiddling with setting up postgresql (I had some authentication issues), I got this:

$ python conservationlands.py download -a park_provincial --email andy.teucher@gov.bc.ca
Traceback (most recent call last):
  File "conservationlands.py", line 507, in <module>
    cli()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 716, in __call__
    return self.main(*args, **kwargs)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 696, in main
    rv = self.invoke(ctx)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 1060, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 889, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\click\core.py", line 534, in invoke
    return callback(*args, **kwargs)
  File "conservationlands.py", line 328, in download
    db.create_schema(CONFIG["schema"])
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\pgdb\database.py", line 190, in create_schema
    self.engine.execute(CreateSchema(schema))
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 2057, in execute
    connection = self.contextual_connect(close_with_result=True)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 2106, in contextual_connect
    self._wrap_pool_connect(self.pool.connect, None),
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\base.py", line 2141, in _wrap_pool_connect
    return fn()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 387, in connect
    return _ConnectionFairy._checkout(self)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 766, in _checkout
    fairy = _ConnectionRecord.checkout(pool)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 516, in checkout
    rec = pool._do_get()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 1138, in _do_get
    self._dec_overflow()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\util\langhelpers.py", line 60, in __exit__
    compat.reraise(exc_type, exc_value, exc_tb)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 1135, in _do_get
    return self._create_connection()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 333, in _create_connection
    return _ConnectionRecord(self)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 461, in __init__
    self.__connect(first_connect_check=True)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\pool.py", line 661, in __connect
    exec_once(self.connection, self)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\event\attr.py", line 246, in exec_once
    self(*args, **kw)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\event\attr.py", line 256, in __call__
    fn(*args, **kw)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\util\langhelpers.py", line 1321, in go
    return once_fn(*arg, **kw)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\strategies.py", line 181, in first_connect
    dialect.initialize(c)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\dialects\postgresql\psycopg2.py", line 564, in initialize
    super(PGDialect_psycopg2, self).initialize(connection)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\dialects\postgresql\base.py", line 1984, in initialize
    super(PGDialect, self).initialize(connection)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\default.py", line 238, in initialize
    self._get_server_version_info(connection)
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\dialects\postgresql\base.py", line 2192, in _get_server_version_info
    v = connection.execute("select version()").scalar()
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\result.py", line 1224, in scalar
    return row[0]
  File "C:\Users\ateucher\AppData\Local\Continuum\Anaconda\lib\site-packages\sqlalchemy\engine\result.py", line 563, in _key_fallback
    expression._string_or_unprintable(key))
sqlalchemy.exc.NoSuchColumnError: "Could not locate column in row for column '0'"

smnorris commented 7 years ago

Yuck.

Can you connect to the db via pgdb at all? Try this in python:

>>> import pgdb
>>> db= pgdb.connect(<your db_url>)
>>> db.tables
[tables.........]

smnorris commented 7 years ago

One possibility is I did fix some small but breaking issues in pgdb and bcdata yesterday, so try re-installing those if you installed them prior to today. Doesn't look like that is the issue though.

ateucher commented 7 years ago

I think I haven't got the spatial database set up properly in Postgres. I'll have another look tomorrow morning

smnorris commented 7 years ago

This guide is slightly dated but should still mostly apply:

Install PostgreSQL/PostGIS

download desired version (9.5, 64bit) by following link at http://www.postgresql.org/download/windows/
follow wizard prompts and on completion of Setup, check the Launch Stackbuilder at exit box
select psqlODBC driver and PostGIS 2.2 bundle in the Stackbuilder
continue following the prompts, and unless you will be processing rasters in postgres, say NO to all the dialogs. Also, don't bother to create a sample spatial database.
see this guide for more info http://www.bostongis.com/PrinterFriendly.aspx?content_name=postgis_tut01

With postgres installed, open command line window and:

psql -U postgres 
create database pics;
\connect pics;
create extension postgis;
\q

ateucher commented 7 years ago

So the connection works. This is the output of db.tables:

['information_schema.key_column_usage', 'information_schema.information_schema_c
atalog_name', 'information_schema.columns', 'information_schema.applicable_roles
', 'information_schema.administrable_role_authorizations', 'information_schema.a
ttributes', 'information_schema.constraint_column_usage', 'information_schema.ch
aracter_sets', 'information_schema.check_constraint_routine_usage', 'information
_schema.check_constraints', 'information_schema.constraint_table_usage', 'inform
ation_schema.collations', 'information_schema.collation_character_set_applicabil
ity', 'information_schema.parameters', 'information_schema.column_domain_usage',
 'information_schema.domain_constraints', 'information_schema.column_privileges'
, 'information_schema.column_udt_usage', 'information_schema.domain_udt_usage',
'information_schema.domains', 'information_schema.referential_constraints', 'inf
ormation_schema.enabled_roles', 'information_schema.role_routine_grants', 'infor
mation_schema.role_column_grants', 'information_schema.sequences', 'information_
schema.routine_privileges', 'information_schema.routines', 'information_schema.s
chemata', 'information_schema.sql_features', 'information_schema.sql_implementat
ion_info', 'information_schema.sql_languages', 'information_schema.views', 'info
rmation_schema.sql_packages', 'information_schema.sql_parts', 'information_schem
a.triggers', 'information_schema.sql_sizing', 'information_schema.sql_sizing_pro
files', 'information_schema.view_column_usage', 'information_schema.table_constr
aints', 'information_schema.udt_privileges', 'information_schema.table_privilege
s', 'information_schema.role_table_grants', 'information_schema.tables', 'inform
ation_schema.transforms', 'information_schema.role_udt_grants', 'information_sch
ema.triggered_update_columns', 'information_schema.usage_privileges', 'informati
on_schema.view_routine_usage', 'information_schema.role_usage_grants', 'informat
ion_schema.user_defined_types', 'information_schema.element_types', 'information
_schema.view_table_usage', 'information_schema.data_type_privileges', 'informati
on_schema._pg_foreign_data_wrappers', 'information_schema._pg_foreign_table_colu
mns', 'information_schema.column_options', 'information_schema._pg_foreign_serve
rs', 'information_schema.foreign_data_wrapper_options', 'information_schema.fore
ign_data_wrappers', 'information_schema.foreign_server_options', 'information_sc
hema.foreign_servers', 'information_schema._pg_foreign_tables', 'information_sch
ema.foreign_table_options', 'information_schema.foreign_tables', 'information_sc
hema._pg_user_mappings', 'information_schema.user_mapping_options', 'information
_schema.user_mappings', 'public.geography_columns', 'public.geometry_columns', '
public.spatial_ref_sys', 'public.raster_columns', 'public.raster_overviews']

But I get the same error as above when I run python conservationlands.py download -a park_provincial --email andy.teucher@gov.bc.ca

and in python, running

>>> import pgdb
>>> db= pgdb.connect("postgresql://postgres:postgres@localhost:5432/postgis")
>>> db.create_schema("conservation_lands")

throws the same error.

ateucher commented 7 years ago

The issue is definitely on my end. I tried the sqlalchemy tutorial here:

>>> from sqlalchemy import create_engine
>>> engine = create_engine('postgresql://postgres:postgres@localhost:5432/postgis', echo=True)

>>> from sqlalchemy.ext.declarative import declarative_base
>>> from sqlalchemy import Column, Integer, String
>>> from geoalchemy2 import Geometry

>>> Base = declarative_base()

>>> class Lake(Base):
...     __tablename__ = 'lake'
...     id = Column(Integer, primary_key=True)
...     name = Column(String)
...     geom = Column(Geometry('POLYGON'))
>>> Lake.__table__
Table('lake', MetaData(bind=None), Column('id', Integer(), table=<lake>,
primary_key=True, nullable=False), Column('name', String(), table=<lake>),
Column('geom', Polygon(srid=4326), table=<lake>), schema=None)
>>> Lake.__table__.create(engine)

And at that point I get the same error. So it doesn't look like the issue is with pgdb. I'll keep working on it

ateucher commented 7 years ago

Ok, getting really close:

$ python conservationlands.py download -a park_provincial --email andy.teucher@gov.bc.ca
INFO:root:Downloading park_provincial
INFO:root:Loading src_park_provincial to postgresql://postgres:postgres@localhost:5432/postgis
FAILURE:
Unable to open datasource `user=postgres' with the following drivers.
  -> PCIDSK
  -> JP2ECW
  -> JP2OpenJPEG
  -> JPEG2000
  -> ESRI Shapefile
  -> MapInfo File
  -> UK .NTF
  -> OGR_SDTS
  -> S57
  -> DGN
  -> OGR_VRT
  -> REC
  -> Memory
  -> BNA
  -> CSV
  -> GML
  -> GPX
  -> LIBKML
  -> KML
  -> GeoJSON
  -> OGR_GMT
  -> GPKG
  -> SQLite
  -> ODBC
  -> WAsP
  -> PGeo
  -> MSSQLSpatial
  -> PostgreSQL
  -> OpenFileGDB
  -> XPlane
  -> DXF
  -> Geoconcept
  -> GeoRSS
  -> GPSTrackMaker
  -> VFK
  -> PGDUMP
  -> OSM
  -> GPSBabel
  -> SUA
  -> OpenAir
  -> OGR_PDS
  -> WFS
  -> HTF
  -> AeronavFAA
  -> Geomedia
  -> EDIGEO
  -> GFT
  -> SVG
  -> CouchDB
  -> Cloudant
  -> Idrisi
  -> ARCGEN
  -> SEGUKOOA
  -> SEGY
  -> XLS
  -> ODS
  -> XLSX
  -> ElasticSearch
  -> Walk
  -> CartoDB
  -> SXF
  -> Selafin
  -> JML
  -> PLSCENES
  -> CSW
  -> TIGER
  -> AVCBin
  -> AVCE00
  -> HTTP

It seems the subprocess.call on line 276 might not be getting created properly?

Incidentally, I had to create an environment variable BCDATA_EMAIL with my email address. I would have thought the --email option would work in place of having that set?

ateucher commented 7 years ago

As a note to myself in the future, and other Windows Users:

I installed Fiona and GDAL with python bindings from here and here (according to instructions here.
I set the GDAL_DATA environment variable to C:\Users\[username]\AppData\Local\Continuum\Anaconda\Lib\site-packages\osgeo\data\gdal and added C:\Users\[username]\AppData\Local\Continuum\Anaconda\Lib\site-packages\osgeo to my PATH
I removed my previous GDAL installation, which hopefully won't affect other things like QGIS and R

smnorris commented 7 years ago

Great, that looks like it is just how the quoted postgres string is handled by windows.

I don't have everything set up on windows, can you test tweaking this string?

               """PG:'host={h} user={u} dbname={db} password={pwd}'""".format(
                          h=db.host,
                          u=db.user,
                          db=db.database,
                          pwd=db.password),

smnorris commented 7 years ago

Note that when installing Fiona, you can probably avoid uninstalling a system GDAL if you remove the path to its executable from your PATH variable - I've managed to successfully install Fiona and GDAL to a windows virtualenv without touching the system's GDAL and ArcGIS.

smnorris commented 7 years ago

It looks like the PG connection string had to be surrounded by double quotes. I've fixed that.

ateucher commented 7 years ago

Haha. I just did that too!

smnorris commented 7 years ago

I also see what is going on with the email. I was getting fancy with the prompt but then it didn't actually get passed to the bcgw download function. I've simplified it. If the environment variable is present and --email isn't provided, the environment variable will be used.

ateucher commented 7 years ago

Ok, great!

bcgov / designatedlands

test installation #3

Install PostgreSQL/PostGIS