Leaner Travis-CI configuration.

[ocefpaf]

Closes #64

[ocefpaf]

In theory all tests are passing now with the same number of skipped tests and warnings . See the full log in: https://travis-ci.org/ODM2/WOFpy/builds/194939748

I also added a linter, non-mandatory, test to help improve code readability. See https://travis-ci.org/ODM2/WOFpy/jobs/194939753

Current:

============================ pytest-warning summary ============================
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSite' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSpatialReference' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestVariable' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestUnits' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSeries' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestMethod' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestMetadata' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSource' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestQualifier' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestOffsetType' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestDataValue' because it has a __init__ constructor
========== 42 passed, 14 skipped, 11 pytest-warnings in 12.52 seconds ==========

Before:

=================================== FAILURES ===================================
________________________ TestOdmDao.test_get_all_sites _________________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_all_sites>
    def test_get_all_sites(self):
>       siteResultList = self.dao.get_all_sites()
test/test_odm2_dao_sqlite.py:266: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:41: in get_all_sites
    join(odm2_models.FeatureActions).\
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:1964: in join
    from_joinpoint=from_joinpoint)
<string>:2: in _join
    ???
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/base.py:201: in generate
    fn(self, *args[1:], **kw)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2108: in _join
    outerjoin, full, create_aliases, prop)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2181: in _join_left_to_right
    self._join_to_left(l_info, left, right, onclause, outerjoin, full)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <sqlalchemy.orm.query.Query object at 0x7fed634c2050>
l_info = <Mapper at 0x7fed6e8654d0; Sites>
left = <Mapper at 0x7fed6e8654d0; Sites>
right = <class 'odm2api.ODM2.models.FeatureActions'>, onclause = None
outerjoin = False, full = False
    def _join_to_left(self, l_info, left, right, onclause, outerjoin, full):
        info = l_info
        left_mapper = getattr(info, 'mapper', None)
        left_selectable = info.selectable

        if self._from_obj:
            replace_clause_index, clause = sql_util.find_join_source(
                self._from_obj,
                left_selectable)
            if clause is not None:
                try:
                    clause = orm_join(clause,
                                      right,
                                      onclause, isouter=outerjoin, full=full)
                except sa_exc.ArgumentError as ae:
                    raise sa_exc.InvalidRequestError(
                        "Could not find a FROM clause to join from.  "
                        "Tried joining to %s, but got: %s" % (right, ae))

                self._from_obj = \
                    self._from_obj[:replace_clause_index] + \
                    (clause, ) + \
                    self._from_obj[replace_clause_index + 1:]
                return

        if left_mapper:
            for ent in self._entities:
                if ent.corresponds_to(left):
                    clause = ent.selectable
                    break
            else:
                clause = left
        else:
            clause = left_selectable

        assert clause is not None
        try:
            clause = orm_join(
                clause, right, onclause, isouter=outerjoin, full=full)
        except sa_exc.ArgumentError as ae:
            raise sa_exc.InvalidRequestError(
                "Could not find a FROM clause to join from.  "
>               "Tried joining to %s, but got: %s" % (right, ae))
E           InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'odm2api.ODM2.models.FeatureActions'>, but got: Can't find any foreign key relationships between 'sites' and 'featureactions'.
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2310: InvalidRequestError
______________________ TestOdmDao.test_get_all_variables _______________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_all_variables>
    def test_get_all_variables(self):
>       varResultList = self.dao.get_all_variables()
test/test_odm2_dao_sqlite.py:283: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:140: in get_all_variables
    v_arr = self.get_variables_from_results()
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <examples.flask.odm2.timeseries.odm2_timeseries_dao.Odm2Dao object at 0x7fed62ddef10>
var_codes = None
    def get_variables_from_results(self,var_codes=None):
        l_var_codes = None
        if var_codes is not None:
            if not isinstance(var_codes, list):
                l_var_codes = []
                l_var_codes.append(var_codes)
            else:
                l_var_codes = var_codes

        r_t_Arr = []
        if l_var_codes is None:
            r_t = self.db_session.query(odm2_models.TimeSeriesResultValues).\
                                        join(odm2_models.TimeSeriesResults).\
>                                       group_by(odm2_models.TimeSeriesResults.VariableID).all()
E           AttributeError: type object 'TimeSeriesResults' has no attribute 'VariableID'
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:114: AttributeError
_______________________ TestOdmDao.test_get_site_by_code _______________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_site_by_code>
    def test_get_site_by_code(self):
        for known_code in self.known_site_codes:
            siteResult = self.dao.get_site_by_code(known_code)
>           self.assertEqual(known_code, siteResult.SiteCode)
E           AttributeError: 'NoneType' object has no attribute 'SiteCode'
test/test_odm2_dao_sqlite.py:274: AttributeError
______________________ TestOdmDao.test_get_sites_by_codes ______________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_sites_by_codes>
    def test_get_sites_by_codes(self):
        siteResultList = self.dao.get_sites_by_codes(self.known_site_codes)
        resultSiteCodes = [s.SiteCode for s in siteResultList]
        for known_code in self.known_site_codes:
>           self.assertTrue(known_code in resultSiteCodes)
E           AssertionError: False is not true
test/test_odm2_dao_sqlite.py:280: AssertionError
_______________________ TestOdmDao.test_get_var_by_code ________________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_var_by_code>
    def test_get_var_by_code(self):
        for known_code in self.known_var_codes:
>           varResult = self.dao.get_variable_by_code(known_code)
test/test_odm2_dao_sqlite.py:290: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:145: in get_variable_by_code
    v_arr = self.get_variables_from_results(var_code)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:120: in get_variables_from_results
    join(odm2_models.Variables).\
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:1964: in join
    from_joinpoint=from_joinpoint)
<string>:2: in _join
    ???
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/base.py:201: in generate
    fn(self, *args[1:], **kw)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2108: in _join
    outerjoin, full, create_aliases, prop)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2181: in _join_left_to_right
    self._join_to_left(l_info, left, right, onclause, outerjoin, full)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <sqlalchemy.orm.query.Query object at 0x7fed62ba12d0>
l_info = <sqlalchemy.orm.util._ORMJoin at 0x7fed62ee7f50; Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)>
left = <sqlalchemy.orm.util._ORMJoin at 0x7fed62ee7f50; Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)>
right = <class 'odm2api.ODM2.models.Variables'>, onclause = None
outerjoin = False, full = False
    def _join_to_left(self, l_info, left, right, onclause, outerjoin, full):
        info = l_info
        left_mapper = getattr(info, 'mapper', None)
        left_selectable = info.selectable

        if self._from_obj:
            replace_clause_index, clause = sql_util.find_join_source(
                self._from_obj,
                left_selectable)
            if clause is not None:
                try:
                    clause = orm_join(clause,
                                      right,
                                      onclause, isouter=outerjoin, full=full)
                except sa_exc.ArgumentError as ae:
                    raise sa_exc.InvalidRequestError(
                        "Could not find a FROM clause to join from.  "
>                       "Tried joining to %s, but got: %s" % (right, ae))
E                   InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'odm2api.ODM2.models.Variables'>, but got: Can't find any foreign key relationships between 'Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)' and 'variables'.
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2285: InvalidRequestError
______________________ TestOdmDao.test_get_vars_by_codes _______________________
self = <test_odm2_dao_sqlite.TestOdmDao testMethod=test_get_vars_by_codes>
    def test_get_vars_by_codes(self):
>       varResultList = self.dao.get_variables_by_codes(self.known_var_codes)
test/test_odm2_dao_sqlite.py:294: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:151: in get_variables_by_codes
    v_arr = self.get_variables_from_results(var_codes_arr)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/examples/flask/odm2/timeseries/odm2_timeseries_dao.py:120: in get_variables_from_results
    join(odm2_models.Variables).\
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:1964: in join
    from_joinpoint=from_joinpoint)
<string>:2: in _join
    ???
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/base.py:201: in generate
    fn(self, *args[1:], **kw)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2108: in _join
    outerjoin, full, create_aliases, prop)
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2181: in _join_left_to_right
    self._join_to_left(l_info, left, right, onclause, outerjoin, full)
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
self = <sqlalchemy.orm.query.Query object at 0x7fed6e62bb90>
l_info = <sqlalchemy.orm.util._ORMJoin at 0x7fed6494abd0; Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)>
left = <sqlalchemy.orm.util._ORMJoin at 0x7fed6494abd0; Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)>
right = <class 'odm2api.ODM2.models.Variables'>, onclause = None
outerjoin = False, full = False
    def _join_to_left(self, l_info, left, right, onclause, outerjoin, full):
        info = l_info
        left_mapper = getattr(info, 'mapper', None)
        left_selectable = info.selectable

        if self._from_obj:
            replace_clause_index, clause = sql_util.find_join_source(
                self._from_obj,
                left_selectable)
            if clause is not None:
                try:
                    clause = orm_join(clause,
                                      right,
                                      onclause, isouter=outerjoin, full=full)
                except sa_exc.ArgumentError as ae:
                    raise sa_exc.InvalidRequestError(
                        "Could not find a FROM clause to join from.  "
>                       "Tried joining to %s, but got: %s" % (right, ae))
E                   InvalidRequestError: Could not find a FROM clause to join from.  Tried joining to <class 'odm2api.ODM2.models.Variables'>, but got: Can't find any foreign key relationships between 'Join object on timeseriesresultvalues(140657734838672) and timeseriesresults(140657735600272)' and 'variables'.
../../../virtualenv/python2.7.12/lib/python2.7/site-packages/sqlalchemy/orm/query.py:2285: InvalidRequestError
============================ pytest-warning summary ============================
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSite' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSpatialReference' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestVariable' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestUnits' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSeries' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestMethod' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestMetadata' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestSource' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestQualifier' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestOffsetType' because it has a __init__ constructor
WC1 /home/travis/build/ODM2/WOFpy/test/test_dao.py cannot collect test class 'TestDataValue' because it has a __init__ constructor
===== 6 failed, 36 passed, 14 skipped, 11 pytest-warnings in 10.62 seconds =====

[emiliom]

@valentinedwv, can you take a look at these changes from @ocefpaf (Filipe) and merge if they look ok?

Next week we can also discuss giving @ocefpaf admin access to the WOFpy repo.

[emiliom]

@valentinedwv, I should've added these comments/caveats from @ocefpaf, for your consideration before merging:

There is a lot there to discuss before we merge it, from the use of conda to get the dependencies, to the removal of some unused dependencies. At least Travis-CI is green again! But I am not sure it means anything because the test coverage is questionable because the tests are passing when Python 3 is used when it should .not!

[ocefpaf]

This is ready for review but if the PR has too many commit/changes let me know and I will reduce the scope to easy the review.

[emiliom]

Thanks, @ocefpaf! I'm planning to get to this by tomorrow (Friday). I'll tell you then if the PR has too many changes (for my taste, anyway).

[emiliom]

@ocefpaf, I think I'm done reviewing your PR. Mostly ok's (or I'm too ignorant to really judge), with a couple of questions thrown in. Will wait for your response before merging the PR. Thanks!

[emiliom]

@ocefpaf, any comments or follow-ups on my PR comments? Should I go ahead and merge the PR?

[ocefpaf]

@ocefpaf, any comments or follow-ups on my PR comments? Should I go ahead and merge the PR?

I made a few minor comments but mostly we need to disambiguate what I call dependency and what this projects calls dependency.

There are test/examples/extra code that uses pyodbc, odm2api, suds, and geoalchemy (actually I could not find a single reference to geoalchemy).

When they are test dependency I add them to requirements-dev.txt as the end user does not need them. When they are dependency need by the examples we can add them to requirements-dev.txt, if the example is run like a test, but that does not mean we should require them.

I am trying to get a single requirements.txt and requirements-dev.txt that are both pip and conda installable. I achieved that for the former but not the latter due to geoalchemy-odm2 (unavailable at PyPI) and odm2api (available but not usable at the moment).

I submitted a few extra commits and this is good to go IMO. The remaining points we raised, like the requirements-dev.txt, should be addressed in another PR to avoid cluttering this one.

[ocefpaf]

OK. I've reache the same status of the latest master merges :smile:

===== 6 failed, 36 passed, 14 skipped, 11 pytest-warnings in 15.96 seconds =====

Fixing those failures are beyond the purpose of this PR and will need someone with intimate knowledge of those examples and WOFpy.

[emiliom]

Thanks, @ocefpaf. There are some issues here that will take me more time to go through. Tomorrow.

[emiliom]

Ok, I think I'm good to go on this PR. I assume the Travis CI failure is not something to be worried about here.

And regarding this:

===== 6 failed, 36 passed, 14 skipped, 11 pytest-warnings in 15.96 seconds =====

Fixing those failures are beyond the purpose of this PR and will need someone with intimate knowledge of those examples and WOFpy.

They only involve some examples (or tests?), so we should ignore in this PR and move on?

[ocefpaf]

They only involve some examples (or tests?), so we should ignore in this PR and move on?

I vote to merge and address those failures in another PR. This one is already to big and fixing those failures should be in another context IMO.

[emiliom]

Thanks for clarifying. Will do.