search

mediawiki-utilities / python-mwdb

A set of utilities for connecting to and querying a MediaWiki database.
MIT License
1 stars 0 forks source link

sqlalchemy.Metadata.reflect failed because `ONLY_TABLES` contains weird view names on labs #2

Open chelsyx opened 6 years ago

chelsyx commented 6 years ago

When using mwdb.Schema to connect to production database, tables cannot be load because of those weird view names on labs:

>>> import mwdb
>>> enwiki = mwdb.Schema("mysql+pymysql://analytics-store.eqiad.wmnet/enwiki" + "?read_default_file=/path/to/my.cnf")
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/path/to/my/virtualenv/lib/python3.5/site-packages/mwdb/schema.py", line 114, in __init__
    self.meta.reflect(views=True, only=only_tables or self.ONLY_TABLES)
  File "/path/to/my/virtualenv/lib/python3.5/site-packages/sqlalchemy/sql/schema.py", line 3957, in reflect
    (bind.engine, s, ', '.join(missing)))
sqlalchemy.exc.InvalidRequestError: Could not reflect: requested table(s) not available in Engine(mysql+pymysql://analytics-store.eqiad.wmnet/enwiki?read_default_file=/path/to/my.cnf): (archive_userindex, filearchive_userindex, ipblocks_ipindex, localisation, localisation_file_hash, logging_logindex, logging_userindex, mark_as_helpful, msg_resource_links, oldimage_userindex, recentchanges_userindex, revision_userindex, updates, user_properties_anon, watchlist_count, wikilove_image_log)

It works after I remove those names from ONLY_TABLES (https://github.com/mediawiki-utilities/python-mwdb/blob/master/mwdb/schema.py#L37). But I'm sure it's not the right way to fix it because it won't work on both production database and public replicas.

halfak commented 6 years ago

Yes. This is a serious issue. I've reported it to the SQLAlchemy folks and it might be fixed in the current version. I can't find the bug in a quick web search so maybe it was addressed? It would be worth updating mwdb's requirements to pull in a more recent version of SQLAlchemy to see if it is still an issue. I'm pretty backlogged so please feel free to try to beat me to it :D

chelsyx commented 6 years ago

Hmmm... Updating mwdb's requirements doesn't help: I changed the requirement to sqlalchemy>1.2 (the most recent version of SQLAlchemy is 1.2.11 released on 2018-08-20) but the error is still there.