openzim / gutenberg

Scraper for downloading the entire ebooks repository of project Gutenberg
https://download.kiwix.org/zim/gutenberg
GNU General Public License v3.0
130 stars 37 forks source link

Too many SQL Variables on Export #11

Closed teamcoltra closed 10 years ago

teamcoltra commented 10 years ago

root@gutenberg:/var/www/gutenberg# ./dump-gutenberg.py --export
EXPORTING ebooks to static folder (and JSON) Filtered book collection size: 45463 Filtered book collection, PDF: 962 Filtered book collection, ePUB: 45326 Filtered book collection, HTML: 45294 Dumping full_by_popularity.js Dumping full_by_title.js Dumping lang_en_by_popularity.js Dumping lang_en_by_title.js Dumping authors_lang_en.js Traceback (most recent call last): File "./dump-gutenberg.py", line 150, in main(docopt(help, version=0.1)) File "./dump-gutenberg.py", line 137, in main only_books=BOOKS) File "/var/www/gutenberg/gutenberg/export.py", line 96, in export_all_books formats=formats) File "/var/www/gutenberg/gutenberg/export.py", line 569, in export_to_json_helpers Author.first_names.asc())], File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2139, in iter return iter(self.execute()) File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2132, in execute self._qr = ResultWrapper(model_class, self._execute(), query_meta) File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 1838, in _execute return self.database.execute_sql(sql, params, self.require_commit) File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2414, in execute_sql self.commit() File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2283, in exit reraise(new_type, new_type(*exc_value.args), traceback) File "/usr/local/lib/python2.7/dist-packages/peewee.py", line 2406, in execute_sql cursor.execute(sql, params or ()) peewee.OperationalError: too many SQL variables

teamcoltra commented 10 years ago

I also want to point out that the static js files will come out to 12MB together.

Seb35 commented 10 years ago

I also had this problem on my computer for a big number of books, but it disappeared with a lower number of books. On some computers this error never appears, so it’s a performance issue. I don’t know how to improve the situation/performance.

kelson42 commented 10 years ago

Duplicate of https://github.com/kiwix/gutenberg/issues/8

kelson42 commented 10 years ago

Slow, but works fine!!! THX