search

WeblateOrg / weblate

Web based localization tool with tight version control integration.
https://weblate.org/
GNU General Public License v3.0
4.54k stars 1.01k forks source link

Weblate didn't detect new translations files #789

Closed Insoleet closed 9 years ago

Insoleet commented 9 years ago

Hello,

We are using weblate for ucoin project, and it's a great tool. Nice job !

Today, I wanted to add the portuguese-brazil language. Weblate did detect the update of the other .ts files, but didn't detect the new pt_BR.ts file. I checked and the file is present in the filesystem. The ts files are the following files :

Weblate did receive the github hook, but maybe I forced a pull too early ?
How can I force weblate to detect the new language again ?

Here are the logs : 

INFO GitHub notification will update cutecoin/Cutecoin-lang
INFO:weblate:GitHub notification will update cutecoin/Cutecoin-lang
[23/Jul/2015 06:21:47]"POST /hooks/github/ HTTP/1.1" 200 16
INFO cutecoin/cutecoin-lang: updating repository
INFO:weblate:cutecoin/cutecoin-lang: updating repository
INFO cutecoin/cutecoin-lang: update took 1.89 seconds:
INFO:weblate:cutecoin/cutecoin-lang: update took 1.89 seconds:
DEBUG cutecoin/cutecoin-lang: update: Fetching origin
DEBUG:weblate:cutecoin/cutecoin-lang: update: Fetching origin
INFO cutecoin/cutecoin-lang: merge remote into repo
INFO:weblate:cutecoin/cutecoin-lang: merge remote into repo
INFO cutecoin/cutecoin-lang: checking res/i18n/ts/de_DE.ts [1/7]
INFO:weblate:cutecoin/cutecoin-lang: checking res/i18n/ts/de_DE.ts [1/7]
INFO cutecoin/cutecoin-lang/de_DE: processing res/i18n/ts/de_DE.ts, revision has changed
INFO:weblate:cutecoin/cutecoin-lang/de_DE: processing res/i18n/ts/de_DE.ts, revision has changed
[23/Jul/2015 06:21:56]"GET /projects/cutecoin/cutecoin-lang/ HTTP/1.1" 200 41695
[23/Jul/2015 06:21:56]"GET /js/i18n/ HTTP/1.1" 200 2418
INFO cutecoin/cutecoin-lang: checking res/i18n/ts/es_ES.ts [2/7]
INFO:weblate:cutecoin/cutecoin-lang: checking res/i18n/ts/es_ES.ts [2/7]
INFO cutecoin/cutecoin-lang/es_ES: processing res/i18n/ts/es_ES.ts, revision has changed
INFO:weblate:cutecoin/cutecoin-lang/es_ES: processing res/i18n/ts/es_ES.ts, revision has changed
[23/Jul/2015 06:22:09]"GET /projects/rtm/zooms-sur-l-histoire-de-la-monnaie/en_US/translate/?sid=7e922b2c-2c89-11e5-94d0-040156fcab01&offset=117 HTTP/1.1" 302 0
[23/Jul/2015 06:22:16]"GET /commit/cutecoin/cutecoin-lang/ HTTP/1.1" 302 0
INFO cutecoin/cutecoin-lang: checking res/i18n/ts/fr_FR.ts [3/7]
INFO:weblate:cutecoin/cutecoin-lang: checking res/i18n/ts/fr_FR.ts [3/7]
INFO cutecoin/cutecoin-lang/fr_FR: processing res/i18n/ts/fr_FR.ts, revision has changed
INFO:weblate:cutecoin/cutecoin-lang/fr_FR: processing res/i18n/ts/fr_FR.ts, revision has changed
INFO cutecoin/cutecoin-lang: updating repository
INFO:weblate:cutecoin/cutecoin-lang: updating repository
INFO cutecoin/cutecoin-lang: update took 0.83 seconds:
INFO:weblate:cutecoin/cutecoin-lang: update took 0.83 seconds:
DEBUG cutecoin/cutecoin-lang: update: Fetching origin
DEBUG:weblate:cutecoin/cutecoin-lang: update: Fetching origin
[23/Jul/2015 06:22:26]"GET /update/cutecoin/cutecoin-lang/ HTTP/1.1" 302 0
[23/Jul/2015 06:22:28]"GET /projects/cutecoin/cutecoin-lang/ HTTP/1.1" 200 41888
[23/Jul/2015 06:22:29]"GET /js/i18n/ HTTP/1.1" 200 2418
Exception in thread Thread-5253:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/weblate/weblate/trans/models/subproject.py", line 665, in do_update
    self.create_translations(request=request)
  File "/opt/weblate/weblate/trans/models/subproject.py", line 924, in create_translations
    self, lang, code, path, force, request=request
  File "/opt/weblate/weblate/trans/models/translation.py", line 68, in check_sync
    translation.check_sync(force, request=request)
  File "/opt/weblate/weblate/trans/models/translation.py", line 496, in check_sync
    self, unit, pos
  File "/opt/weblate/weblate/trans/models/unit.py", line 99, in update_from_unit
    dbunit.update_from_unit(unit, pos, created)
  File "/opt/weblate/weblate/trans/models/unit.py", line 459, in update_from_unit
    same_state=same_state
  File "/opt/weblate/weblate/trans/models/unit.py", line 715, in save
    update_index_unit(self, force_insert)
  File "/opt/weblate/weblate/trans/search.py", line 205, in update_index_unit
    index = get_source_index()
  File "/opt/weblate/weblate/trans/search.py", line 119, in get_source_index
    index = STORAGE.open_index('source')
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/filedb/filestore.py", line 176, in open_index
    return indexclass(self, schema=schema, indexname=indexname)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/index.py", line 623, in read
    stream = storage.open_file(tocfilename)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/filedb/filestore.py", line 497, in open_file
    f = StructFile(open(self._fpath(name), "rb"), name=name, **kwargs)
INFO cutecoin/cutecoin-lang: checking res/i18n/ts/it_IT.ts [4/7]
INFO:weblate:cutecoin/cutecoin-lang: checking res/i18n/ts/it_IT.ts [4/7]
INFO cutecoin/cutecoin-lang/it_IT: processing res/i18n/ts/it_IT.ts, revision has changed
INFO:weblate:cutecoin/cutecoin-lang/it_IT: processing res/i18n/ts/it_IT.ts, revision has changed
[23/Jul/2015 06:22:36]"GET /projects/cutecoin/cutecoin-lang/ HTTP/1.1" 200 41763
[23/Jul/2015 06:22:36]"GET /js/i18n/ HTTP/1.1" 200 2418
INFO cutecoin/cutecoin-lang: checking res/i18n/ts/pl_PL.ts [5/7]
INFO:weblate:cutecoin/cutecoin-lang: checking res/i18n/ts/pl_PL.ts [5/7]
INFO cutecoin/cutecoin-lang/pl_PL: processing res/i18n/ts/pl_PL.ts, revision has changed
INFO:weblate:cutecoin/cutecoin-lang/pl_PL: processing res/i18n/ts/pl_PL.ts, revision has changed
Exception in thread Thread-7270:
Traceback (most recent call last):
  File "/usr/lib/python2.7/threading.py", line 810, in __bootstrap_inner
    self.run()
  File "/usr/lib/python2.7/threading.py", line 763, in run
    self.__target(*self.__args, **self.__kwargs)
  File "/opt/weblate/weblate/trans/models/subproject.py", line 665, in do_update
    self.create_translations(request=request)
  File "/opt/weblate/weblate/trans/models/subproject.py", line 924, in create_translations
    self, lang, code, path, force, request=request
  File "/opt/weblate/weblate/trans/models/translation.py", line 68, in check_sync
    translation.check_sync(force, request=request)
  File "/opt/weblate/weblate/trans/models/translation.py", line 496, in check_sync
    self, unit, pos
  File "/opt/weblate/weblate/trans/models/unit.py", line 99, in update_from_unit
    dbunit.update_from_unit(unit, pos, created)
  File "/opt/weblate/weblate/trans/models/unit.py", line 459, in update_from_unit
    same_state=same_state
  File "/opt/weblate/weblate/trans/models/unit.py", line 715, in save
    update_index_unit(self, force_insert)
  File "/opt/weblate/weblate/trans/search.py", line 205, in update_index_unit
    index = get_source_index()
  File "/opt/weblate/weblate/trans/search.py", line 119, in get_source_index
    index = STORAGE.open_index('source')
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/filedb/filestore.py", line 176, in open_index
    return indexclass(self, schema=schema, indexname=indexname)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/index.py", line 421, in __init__
    TOC.read(self.storage, self.indexname, schema=self._schema)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/index.py", line 623, in read
    stream = storage.open_file(tocfilename)
  File "/usr/local/lib/python2.7/dist-packages/Whoosh-2.7.0-py2.7.egg/whoosh/filedb/filestore.py", line 497, in open_file
    f = StructFile(open(self._fpath(name), "rb"), name=name, **kwargs)
IOError: [Errno 2] No such file or directory: '/opt/weblate/data/whoosh/_source_8460.toc'

And here are the versions I use : 

/opt/weblate# ./manage.py list_versions
 * Weblate weblate-2.3-226-g9f16961
 * Python 2.7.6
 * Django 1.8.2
 * python-social-auth 0.2.10
 * Translate Toolkit 1.13.0
 * Whoosh 2.7.0
 * Git 1.9.1
 * Pillow (PIL) 1.1.7
 * dateutil 2.4.2
 * lxml 3.3.3
 * django-crispy-forms 1.4.0
 * pyuca N/A
nijel commented 9 years ago

Looks like some problem with updating Whoosh index. Maybe it's corrupted or it's having some problems in multi threaded environment.

Can you please try:

Insoleet commented 9 years ago

Should I commit the translations status before rebuilding index so that it wont be lost ?

nijel commented 9 years ago

Not needed - it's just fulltext index, it does not anyhow affect content.

Insoleet commented 9 years ago

Thanks.

So here are the results :

./manage.py rebuild_index --clean --all

Didn't fix the problem, I still can't see the portuguese language. I set "OFFLOAD_INDEXING = True", restarted the webserver, the language is still not detected.

EDIT : using loadpo, I was able to get the new language in weblate. I'll see in the next updates if that happens again or not. Thanks !

nijel commented 9 years ago

Great, please monitor the logs if you will get the error again...

jportoles commented 9 years ago

We are seeing the same issue with Weblate 2.3 after adding a new language to two components within the same project; the new language showed up in only one of the two components and pulling the repos wouldn't help with the other. ./manage.py rebuild_index didn't solve the issue, but manage.py loadpo --all --lang xx-XX did.

nijel commented 9 years ago

Did you get same error? If so, the rebuild_index won't make the languages appear, it will just fix the error. Still I wonder why the index got corrupt, but maybe Whoosh isn't really good for this size of data (see #800).

jportoles commented 9 years ago

We get ocassional Permission denied lock errors during normal usage and 'NoneType' object has no attribute '__getitem__' errors when memcached is restarted, but other than that I didn't see any particular error in this case, only the language didn't show up when it should have.

jportoles commented 9 years ago

We reproduced this again today, we got a fatal: There is no merge to abort (MERGE_HEAD missing) error after pulling the updates repos via weblate, and after manually pulling them directly from the local copy of the repos, we had to use manage.py to add the new languages.

nijel commented 9 years ago

Can you please open separate issues for these? I haven't seen either of them so far, so it would be worth of further investigation.