search

gnosygnu / xowa

xowa offline wiki application
Other
375 stars 41 forks source link

SQLITE_ERROR in enwiki build #527

Open desb42 opened 5 years ago

desb42 commented 5 years ago

I have just been rebuilding enwiki html (2019-06-01) and the build phase has finished - still waiting for the whole process to terminate (see #526)

In the logfile there are many SQLITE_ERROR entries There seem to be two styles of error one is 'db already in use'

mass_parse.fail:ns=4 ttl=WikiProject_Wine/right_panel
 err=[err 0] <org.sqlite.SQLiteException> [SQLITE_ERROR] SQL error or missing database (database link_db_1 is already in use)
  [err 1] <db> db.engine:exec failed: url=data source=g:\xowa\wiki\en.wikipedia.org\en.wikipedia.org-xtn.category.core.xowa;version=3; sql=ATTACH 'g:\xowa\wiki\en.wikipedia.org\en.wikipedia.org-xtn.category.link-db.001.xowa' AS link_db_1;
  [trace]:
    org.sqlite.core.DB.newSQLException(DB.java:909)
    org.sqlite.core.DB.newSQLException(DB.java:921)
    org.sqlite.core.DB.throwex(DB.java:886)
    org.sqlite.core.NativeDB._exec_utf8(Native Method)
    org.sqlite.core.NativeDB._exec(NativeDB.java:87)
    org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:116)
    gplx.dbs.engines.Db_engine_sql_base.Exec_as_int(Db_engine_sql_base.java:48)
    gplx.dbs.engines.sqlite.Sqlite_engine.Env_db_attach(Sqlite_engine.java:45)
    gplx.dbs.Db_conn.Env_db_attach(Db_conn.java:35)
    gplx.dbs.Db_attach_mgr.Attach(Db_attach_mgr.java:45)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catlink_loader.Load_catlinks(Xoctg_catlink_loader.java:152)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catlink_loader.Run(Xoctg_catlink_loader.java:29)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catpage_loader.Load_ctg_or_null(Xoctg_catpage_loader.java:64)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.Xoctg_catpage_mgr.Get_or_load_or_null(Xoctg_catpage_mgr.java:61)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.Xoctg_catpage_mgr.Renderchild(Xoctg_catpage_mgr.java:280)
    gplx.xowa.xtns.categorytrees.Categorytree_nde.Xtn_write(Categorytree_nde.java:83)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Xnde(Xoh_html_wtr.java:683)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:93)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:97)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:96)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:95)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:97)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:96)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:95)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:84)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_doc(Xoh_html_wtr.java:64)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_body_wikitext(Xoh_page_wtr_wkr.java:237)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_body(Xoh_page_wtr_wkr.java:192)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_hdump(Xoh_page_wtr_wkr.java:156)
gplx.xowa.htmls.core.bldrs.Xob_hdump_bldr.Insert(Xob_hdump_bldr.java:55)
gplx.xowa.addons.bldrs.mass_parses.parses.wkrs.Xomp_parse_wkr.Exec(Xomp_parse_wkr.java:143)
gplx.xowa.addons.bldrs.mass_parses.parses.wkrs.Xomp_parse_wkr.Invk(Xomp_parse_wkr.java:213)
gplx.Gfo_invk_.Invk_by_msg(Gfo_invk_.java:34)
gplx.core.threads.Thread_adp.run(Thread_adp.java:33)
java.lang.Thread.run(Thread.java:748)

and the other 'db locked'

mass_parse.fail:ns=100 ttl=Colonialism/Categories
 err=[err 0] <org.sqlite.SQLiteException> [SQLITE_ERROR] SQL error or missing database (database link_db_1 is locked)
  [err 1] <db> db.engine:exec failed: url=data source=g:\xowa\wiki\en.wikipedia.org\en.wikipedia.org-xtn.category.core.xowa;version=3; sql=DETACH link_db_1;
  [trace]:
    org.sqlite.core.DB.newSQLException(DB.java:909)
    org.sqlite.core.DB.newSQLException(DB.java:921)
    org.sqlite.core.DB.throwex(DB.java:886)
    org.sqlite.core.NativeDB._exec_utf8(Native Method)
    org.sqlite.core.NativeDB._exec(NativeDB.java:87)
    org.sqlite.jdbc3.JDBC3Statement.executeUpdate(JDBC3Statement.java:116)
    gplx.dbs.engines.Db_engine_sql_base.Exec_as_int(Db_engine_sql_base.java:48)
    gplx.dbs.engines.sqlite.Sqlite_engine.Env_db_detach(Sqlite_engine.java:46)
    gplx.dbs.Db_conn.Env_db_detach(Db_conn.java:36)
    gplx.dbs.Db_attach_mgr.Detach(Db_attach_mgr.java:52)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catlink_loader.Load_catlinks(Xoctg_catlink_loader.java:169)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catlink_loader.Run(Xoctg_catlink_loader.java:29)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.dbs.Xoctg_catpage_loader.Load_ctg_or_null(Xoctg_catpage_loader.java:64)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.Xoctg_catpage_mgr.Get_or_load_or_null(Xoctg_catpage_mgr.java:61)
    gplx.xowa.addons.wikis.ctgs.htmls.catpages.Xoctg_catpage_mgr.Renderchild(Xoctg_catpage_mgr.java:280)
    gplx.xowa.xtns.categorytrees.Categorytree_nde.Xtn_write(Categorytree_nde.java:83)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Xnde(Xoh_html_wtr.java:683)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:93)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:97)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:96)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Tblw(Xoh_html_wtr.java:855)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:95)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_tkn(Xoh_html_wtr.java:84)
    gplx.xowa.htmls.core.htmls.Xoh_html_wtr.Write_doc(Xoh_html_wtr.java:64)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_body_wikitext(Xoh_page_wtr_wkr.java:237)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_body(Xoh_page_wtr_wkr.java:192)
    gplx.xowa.htmls.Xoh_page_wtr_wkr.Write_hdump(Xoh_page_wtr_wkr.java:156)
    gplx.xowa.htmls.core.bldrs.Xob_hdump_bldr.Insert(Xob_hdump_bldr.java:55)
    gplx.xowa.addons.bldrs.mass_parses.parses.wkrs.Xomp_parse_wkr.Exec(Xomp_parse_wkr.java:143)
    gplx.xowa.addons.bldrs.mass_parses.parses.wkrs.Xomp_parse_wkr.Invk(Xomp_parse_wkr.java:213)
    gplx.Gfo_invk_.Invk_by_msg(Gfo_invk_.java:34)
    gplx.core.threads.Thread_adp.run(Thread_adp.java:33)
    java.lang.Thread.run(Thread.java:748)

These errors cannot be reproduced if I use xowa-gui to look at them individually

I suspect a multi thread conflict

I attach the full log session.zip

This may help with #526

One more thing, I am still using the attach category database mechanism

gnosygnu commented 5 years ago

Yeah, this looks like a threading mechanism. The strange thing is that I don't think I've ever seen this error in my builds, and I've built enwiki at least a few dozen times.

Just like #526, let me try to reproduce this on my side in the next few weeks. Will report here. Thanks!

desb42 commented 5 years ago

for my mass_parse.init I use this

  add     ('en.wikipedia.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '0|4|8|12|14|100';}}

Perhaps you choose a different set of namespaces

desb42 commented 4 years ago

I thought I would take a closer look at this error As this only occurs on an html build, it took a bit to reproduce.

For experimentation I used enwikibooks because its quite small The only namespace I used was Category: (14)

add     ('en.wikibooks.org' , 'wiki.mass_parse.init') {cfg {ns_ids = '14';}}

The first change I made was to 400_xowa\src\gplx\xowa\addons\wikis\ctgs\htmls\catpages\Xoctg_catpage_mgr.java in Get_by_cache_or_null, I changed the synchronized to a static ReentrantReadWriteLock I believe that even with many instances of Xoctg_catpage_mgr this lock ensures that the access to the database calls is stricktly one at a time

This did not in fact make any difference

I then displayed all the database calls, by all the treads and this lead to the fact that, if there is a page in one thread accessing a dynamicpagelist and in another page a #PAGESINCATEGORY (and only when the wind is right)

In 400_xowa\src\gplx\xowa\wikis\data\tbls\Xowd_cat_core_tbl.java Select I replaced the synchronized with a call to the same static ReentrantReadWriteLock as in Xoctg_catpage_mgr.java (Probably not the right place) This seems to have cured the problem for enwikibooks - I have yet to try it on anything larger

The dynamicpagelist uses the category-core db with two other attached dbs and #PAGESINCATEGORY just uses the category-core db

SQLITE locks the attached table when category-core tables are being access

If the thread running #PAGESINCATEGORY is interrupted during the row reading and the dynamicpagelist thread continues on to a DETACH a lock error occurs

Hence the need to synchronize the threads (but not by using synchronized)

(The actual code I use is in myxowa project)

desb42 commented 4 years ago

I have just had another thought about this.

Sqlite attach database is on a per connection basis (according to my read of the Sqlite documentation). If, as I believe, xowa only uses one database connection, this could lead to problems in the multi thread mode as attaching and detaching is performed a lot and perhaps could interfere causing this issue