Antar1011 / Onix

The Pokemon Data-Mining Package
http://onix.readthedocs.io/en/latest/
GNU General Public License v3.0
7 stars 0 forks source link

Reduce the number of variables in the SQL query #62

Closed Antar1011 closed 8 years ago

Antar1011 commented 8 years ago

Currently the integration test throws the following error:

E       sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) too many SQL variables

(it goes on for quite some time, but it's referring to the giant CASE clause that is the species lookup)

This is a known limitation with SQLite where one is limited to 999 "?" variables. Ref: https://www.sqlite.org/limits.html

Upon further investigation, I see that the insert expressions are going to have issues as well for any large-scale operations (though this can be mitigated by setting batch_size really low).

So the solution has two elements:

Antar1011 commented 8 years ago

In progress on the integration-test branch

Antar1011 commented 8 years ago

The species lookup part is done as of e1a95dda60985bba4a7c0dd3c72bb1d5cd48f0a4

Antar1011 commented 8 years ago

Note that if I didn't want to have to rewrite all my unit tests, I could always keep the tuples and then zip with the table columns before insert, but it's not like there's any reason to do that aside from laziness, and this way will result in much clearer tests

Antar1011 commented 8 years ago

Second part is done with ba37f5ccd4e4d081c293c98bbbd94e16ae75a7d7