Closed borrob closed 4 years ago
The Travis failure is only a flake8
error.
:) yes, so the normal testing passes, so I guess it is something with my setup that the one test fails on my machine.
I haven't yet had much time to work on the docker issue, but otherwise I would say we are almost there.
I can helpout with any Docker issues (what was the problem?).
I have been able to track the error, but haven't yet able to solve it. I have updated the Dockerfile to use python3.7.4-alpine3.10
as a starting image. Somewhere in the building process install.sh
installs the python dependencies from the requirements.txt
and in there: pyproj
causes an error:
Proj executable not found. Please set PROJ_DIR variable.
I tried adding proj
to the apk installation in the Dockerfile but to no avail. An alternative solution that I found on the web is to include the python-dev library, but that gets the development files for python 3.7.3, which make me think the virtual env with python 3.7.4
still cannot use it (and I did some other testing suggesting that solely adding python-dev is not enough).
Any ideas or suggestions are welcome. Unfortunately I won't be able to work on this over the weekend.
Docker: Think Alpine images are problematic with binaries like GDAL and Proj. May end up doing compiles, or have things improved recently? Debian Slim is a better option. As with Stetl py3 migration we end up using python:3.6-slim-stretch
. And py 3.6 i.s.o. 3.7, can't recall why.
(also busy w other stuff over weekend further...)
Sorry, forget my comment on Alpine: was mistaken with other projects (Stetl, pygeoapi, workshop) where GDAL was used. Alpine was already in use with GHC for Python 2.7. I can look into the Docker build further.
Globally testing @borrob py3
branch on 24.09.2019:
pyenv
on Mac OSX).paver
installation: OK.(geohealthcheck3.7.1) nusa:borrob.git just$ flake8
./tests/test_plugins.py:41:1: E402 module level import not at top of file
./tests/test_plugins.py:42:1: E402 module level import not at top of file
./tests/test_plugins.py:43:1: E402 module level import not at top of file
./tests/test_plugins.py:44:1: E402 module level import not at top of file
./tests/test_plugins.py:45:1: E402 module level import not at top of file
./tests/test_resources.py:41:1: E402 module level import not at top of file
./tests/test_resources.py:42:1: E402 module level import not at top of file
./tests/test_resources.py:43:1: E402 module level import not at top of file
./tests/test_resources.py:44:1: E402 module level import not at top of file
./tests/test_resources.py:45:1: E402 module level import not at top of file
Easily fixed by moving imports to top-level.
2019-09-24 12:39:33,105 - healthcheck - ERROR - Getting metadata failed: 'str' object has no attribute 'decode'
Traceback (most recent call last):
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/healthcheck.py", line 247, in sniff_test_resource
title = title.decode('utf-8')
AttributeError: 'str' object has no attribute 'decode'
Removing this line title = title.decode('utf-8') fixes that. (all str
is unicode in py3 IMHO).
decode
: in models.py : token generation for user-initiated password reset.Looked into the pyproj
Docker build failure:
python:3.7.4-alpine3.10
Proj executable not found. Please set PROJ_DIR variable.
pyproj
build problem in GHC and also similar Docker build failures.Thanks. I think we are actually almost there. From the top of my head:
Docker seems to be working now (by downgrading pyproj), no flake8 issues anymore, all tests are passing, and I merged to latest master branch.
I'll start with reviewing the documentation to see if that is in line with py3.
Started testing locally with py3 branch, will add findings in comments:
map
not list
in Py3File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/plugins/probe/wfs.py", line 146, in expand_params
if len(ft_namespaces) > 0:
TypeError: object of type 'filter' has no len()
ft_namespaces = list(filter(None, list(ft_namespaces)))
filter()
occurrences assuming list
...(ok, no other occurs)
Probe
for any GetTile
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/plugins/probe/tms.py", line 110, in get_metadata
return TileMapService(resource.url, version=version)
File "/Users/just/.pyenv/versions/3.7.1/envs/geohealthcheck3.7.1/lib/python3.7/site-packages/owslib/tms.py", line 61, in __init__
self.version, url=self.url, un=self.username, pw=self.password
AttributeError: 'TileMapService' object has no attribute 'username'
OWSLib
upgrade in GHC..Yes OWSLib
issue, opened: see https://github.com/geopython/OWSLib/issues/614."message": "Perform_request Err: TypeError expected bytes-like object, not str",
Tested after pulling above commits.
Was testing Basic Auth. Curious why even without credentials HTTPGet Probe
was successful. We trapped into the div issue from Python2 to Python3! Problem is this line in checks.py:
def perform(self):
"""Default check: Resource should at least give no error"""
status = self.probe.response.status_code
==> overall_status = status / 100
if overall_status in [4, 5]:
self.set_result(False, 'HTTP Error status=%d' % status)
In Python2 gives floor
value, but Python3 a float
.
Fix needs to be: overall_status = status // 100
AFAICS.
(Now Basic Auth always fails, even with right credentials, but is other (encoding?) issue...)
For a correctly encoded header string like 'Basic aWF...XS1\n'
this line gives something like 'Basic b\'aWF...XS1\\n\''
(the \n
is later always stripped).
Ok, tested again: 3 (OWSLib TMS) and 5 (HTTP status check) above found solved. 4 (auth encoding) only with SQLite as DB.
Only occurs when using Postgres, not with SQLite. Background: GHC encodes/encrypts a generic auth dict info structure via JSON string to be stored as textfield in DB. It decodes/decrypts when reading. This way we can support multiple auth types with a single auth
column in resource
table. So this is another encoding than in 4 for HTTP auth headers, but think similar problem.
We also need to deal with existing PG DBs that have auth
columns already present in resource
table.
psycopg2
: pip install psycopg2
config_site.py
: SQLALCHEMY_DATABASE_URI = 'postgresql://name:passw@localhost:5432/ghc'
, but may create newFile "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/models.py", line 584, in auth_type
return self.auth['type']
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/models.py", line 578, in auth
return ResourceAuth.decode(self._auth)
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/resourceauth.py", line 87, in decode
raise err
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/resourceauth.py", line 83, in decode
s = decode(APP.config['SECRET_KEY'], encoded)
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/util.py", line 247, in decode
string = base64.urlsafe_b64decode(string + b'===')
TypeError: can only concatenate str (not "bytes") to str
Analysis: problem appears in decode()
(and probably encode()
as well) in util.py :
def decode(key, string):
string = base64.urlsafe_b64decode(string + b'===')
string = string.decode('latin') if six.PY3 else string
encoded_chars = []
for i in range(len(string)):
key_c = key[i % len(key)]
encoded_c = chr((ord(string[i]) - ord(key_c) + 256) % 256)
encoded_chars.append(encoded_c)
encoded_string = ''.join(encoded_chars)
return encoded_string
With SQLite string
is of type bytes
but with PG of type string
.
Tried something like:
def decode(key, string):
if type(string) is not bytes:
string = string.encode()
string = base64.urlsafe_b64decode(string + b'===')
But then get other error on Edit and Test:
File "/Users/just/project/geohealthcheck/borrob.git/GeoHealthCheck/util.py", line 247, in decode
string = base64.urlsafe_b64decode(string + b'===')
File "/Users/just/.pyenv/versions/3.7.1/lib/python3.7/base64.py", line 133, in urlsafe_b64decode
return b64decode(s)
File "/Users/just/.pyenv/versions/3.7.1/lib/python3.7/base64.py", line 87, in b64decode
return binascii.a2b_base64(s)
binascii.Error: Invalid base64-encoded string: number of data characters (213) cannot be 1 more than a multiple of 4
six
is required if we do Py3-only?Weird... I would expect SQLAlchemy
to deal with the abstraction and get the string/byte conversion right regardless of which database is used. I will look into it.
Agree: we should be able to drop six
.
We're not there yet: I noticed some paver
commands still need an update because of issues with the importing of modules. Also: the docker image is building, but I haven't checked yet if it is actually working. I had to change the gunicorn
configuration on a non-docker deploy.
@borrob good to see you synced with master
! Will try not to do too many big changes.
The Postgres-char-issue: my bad! For security reasons the SECRET_KEY
is used for encoding/decoding stored auth creds, and the Py3 instance had a different key! Using the same key: no problem. Pff, was thinking that we had a very serious encoding issue with difficult DB migrations. So we're getting closer!
I fixed one paver issue and did some testing (also with docker). I think we're good to go and I'm curious what the results of the demo environment will be.
I removed the draft tag from this pull request (that was there for the automated testing with travis). Please review and let's hope we can move to py3 soon!
Yes, good, I only want to release 0.7.0 from current master
first, and create a 07.0 maintenance branch. Then further test in particular with Docker and then merge this PR and have a quite some testing time on demo site. OK, @tomkralidis ? Let's aim for all of this before nov 1 ok?
NB solved a nasty concurrency bug (two lines) with #301 #302 today.
A great thank-you @borrob! This was no easy PR to do.
Thanks for the awesome contribution @borrob!
Please do NOT approve this pull request. This is Work In Progress for the python3 conversion. I have an issue with one specific test and we want to see if Travis gets the same error or if it passes without problems.