Use Weblate for translations

[agaida]

hi - we have our weblate instance up and running - so i would like to suggest using it. With weblate we should move back our translations back from lxqt-l10 to the repositories they belong to - there are several reasons for it:

• small components are easy handled with weblate
• the chance of conflicting commits raise with the amount of components in one repository - a pita to maintain
• we can move step by step, starting with the leaf packages that don't have a dependency to liblxqt

We tested the weblate instance with featherpad and it was fun to work with - the only thing missed right now in weblate are automatic translations and suggestions - will implement it with the upcoming new version of weblate.

@all - opinions?

[PCMan]

@agaida How does this compare to Pootle? We tried Pootle in the past but it does not seem to work well with Qt *.ts files. (Also we tried transifex, which is no longer free, long time ago.)

[agaida]

@PCMan basicly it is the same transition engine as in pootle with a new interface. As far as i can tell it works nice with *.ts. A very big plus it the git orientated workflow. We learned about weblate a few years ago, but it seems nobody was able/brave/crazy enough to set it up production ready. So i finally bite the bullet and took the old installation, took it to our new machine into a LXD container, update it and reconfigured it from the scratch.

Time is the key - meanwhle weblate is mature and fun to setup and to work with. In the end it took some days to make it work like i want it to. That means: Auto-import of new changes from the repos, semi-automatic sending pull requests to github - right now it works robust and it is easy to maintain.

I also maintain the pootle installation for LXDE - and to be honest, the implementation quality of weblate is on a much higher level - it might be our fault because the pootle guys has changed the preferred database to my/mariasql - we run postgres and basic database things don't worked with - it should, but doesn't - so i had to fix some constraints, keys etc manually. No of the problems with weblate. So i would give it a go.

[agaida]

i forgot - one could have a look into https://github.com/tsujan/FeatherPad - https://github.com/tsujan/FeatherPad/commit/bbc9c5af49267f3219d02f22a14e9c737cf0e1ab as an example. What i like most is the PR thing (achived with hub - a go-based git wrapper esp. for github. So no hard commits from weblate into the project - i love it.

[agaida]

Disclaimer - i'm not much into translations, nearly no clue, not the python guy, nearly no clue, not the postgres guy, nearly no clue, i don't know linguist well - same for containers, ip-routing and python hosting - if a project based on these requirements runs stable and reliable some people have done a great job. :sunglasses:

[PCMan]

@agaida My vote is +1 for this one. Sounds awesome! The current lxqt-l10n repo is not easy to use for ordinary translators who don't know programming & git. @luis-pereira What do you think? Since you are the master of *.ts and Qt linguist and does most of the LXQtTranslate stuff, your opinion is quite important here.

[yan12125]

Hi @agaida, I tried to login to https://weblate.lxqt.org/ with GitHub OAuth2 but it says:

已停用新註冊！

(New registrations are disabled!)

Is it still under construction?

[agaida]

Right now there are three projects - featherpad, screengrab and qps - ir acked i can enable the registration thing - and yes, i don't wanted new registrations in this service right now. Doesn't make sense. An open registration would make think people that the instance is up and ready for production - and that was not the case right now.

If we decide to go with weblate i will import some leaf packages first and enable registrations. If one want an account right now - drop me a mail via the contact form.

[agaida]

@luis-pereira - dunno if it is possible - but i played a bit with not so common git commands in preparation of putting back the translations with history to the repos they belong to. Would it be much effort if we place the translations in /translations/l10n and leave the possible desktop files in /translations - or is it to much work and /translation/$component would be better?

[yarons]

You can create a different branch for that so it won't clog your logs if that bothers you.

[agaida]

@yarons: but one has to maintain the translation branch - right now it is fine as it is, we should only place our translations again in a sane place :) - second thing about the translation process: we use hub for pushing the translations as pull requests - we could even squash things, but that would need a bit of change at the weblate side - forcefully pull the original sources if needed.

EDIT: But such things will come later - first of all we should have a working and sane base configuration.

[yarons]

@agaida So you're suggesting keeping it in the same repo? It's also a possibility, maybe @nijel can advise you as well :smile:.

[agaida]

i really want it in the same repo - hup helps a lot - so we (will) get pull requests from weblate - right now i see no reason to split the translation process out. My main concern right now is to make the process clean on our site - weblate is up and running (latest 2.xx, pip installation in a container only for weblate). So no problems right now.

One remark: We really want the 3.0 asap - but i'm not desperate enough for checking out git :sunglasses:

[yarons]

@agaida including the latest docker changes? 😀 https://blog.cihar.com/archives/2018/05/29/improved-docker-container-weblate/?utm_source=rss2

[agaida]

@yarons - we don't use docker :smile:

[nijel]

Weblate 3.0 will be out on Friday, probably with no code changes compared to current git...

[agaida]

Cool, really great news - so i will upgrade when it is available via pip

[agaida]

@nijel and @yarons - thanks for the heads up - as i wrote earlier, weblate is pure fun (as far as translations can be fun) to work with. The only thing is our mentioned translation repo. Kind of works, but a horrible base for any webbased translation process and maintenance - so i guess we should change it first. I've learned the needed git magic two days ago - so that should be no problem at all.

[nijel]

Indeed it's usually better to keep translations with the code. In case you want more complex git workflow from Weblate side, you can use wlc to eg. reset the Weblate repository to upstream one...

[tsujan]

Indeed it's usually better to keep translations with the code.

I completely agree. Translators need to know about the context.

[agaida]

ludi incipiant

% pip search weblate
weblate-tools (0.1.0)  - weblate文本处理工具
Weblate (3.0)          - A web-based translation tool with tight version control integration
wlc (0.8)              - A command line utility for Weblate, translation tool with tight version control integration

@nijel - that was fast

[agaida]

@luis-pereira @tsujan @palinek - i have played a bit with merging back translations into the repos they belong to - the script works fine: https://github.com/4sid/translation-move/blob/master/move-translations

but the result looks a bit brain damaged - so i would like to change the structure to https://github.com/4sid/translation-move/commit/3d9a20221fcb5bf03d13700b487b6de2085ac6ac?diff=split

[agaida]

Weblate upgraded to 3.0 - that was easy and pure fun. So i guess we should work on the things to translate and migrate asap.

[agaida]

@luis-pereira @tsujan @palinek @yan12125 - finished the translation move script - the nice thing about is: it just works™ - the not so nice thing is that we need to modify lxqt-build-tools and the CMakeLists.txt's a bit. No big deal, just removing the git pulls and targeting the new translation dirs.

One thing i really like to have - it is implemented in the script now: uniform translation locations. So instead of /$foo/translations/$component or /$foo/translations/$component/$subcomponent it should just be $foo/translations/l10n - will prepare the needed PRs

Can also be filed under "filigree work with a chain saw"

[ghost]

Hello ! What a nice move ! Thank you ! <3 Waiting for the regisrations to open up :) Closing https://github.com/lxqt/lxqt-l10n/issues/345

Best, regards !

[tsujan]

Very good!

[agaida]

Follow up bug: https://github.com/lxqt/lxqt/issues/1497

[palinek]

Available in weblate (working things are linked)

New registrations are still disabled. Is it still in pilot?

[agaida]

will add the first translations today and open the service - would like to add the first users manually in the last two days, but ~ENOTIME

[stefonarch]

Good work! (but I can still not login with GH...)

[agaida]

would like to add the first users manually in the last two days, but ~ENOTIME

[yarons]

Any way I can be part of these hand picked users? :grin:

[agaida]

yarons: Would be happy to set you up - but:

IntegrityError at /accounts/complete/email/

insert or update on table "authtoken_token" violates foreign key constraint "authtoken_token_user_id_35299eff_fk_auth_user_id"
DETAIL:  Key (user_id)=(12) is not present in table "auth_user".

Request Method:     GET
Request URL:    https://weblate.lxqt.org/accounts/complete/email/?verification_code=a4d305a93cf64234838691d6ad08e458&partial_token=0462acd28b914649b8481798ff14b57c
Django Version:     1.11.13
Exception Type:     IntegrityError
Exception Value:    

insert or update on table "authtoken_token" violates foreign key constraint "authtoken_token_user_id_35299eff_fk_auth_user_id"
DETAIL:  Key (user_id)=(12) is not present in table "auth_user".

Exception Location:     /home/weblate/weblate.env/local/lib/python2.7/site-packages/django/db/backends/base/base.py in _commit, line 236
Python Executable:  /usr/bin/uwsgi-core
Python Version:     2.7.13
Python Path:    

['/home/weblate/',
 '.',
 '',
 '/home/weblate/weblate.env/lib/python2.7',
 '/home/weblate/weblate.env/lib/python2.7/plat-x86_64-linux-gnu',
 '/home/weblate/weblate.env/lib/python2.7/lib-tk',
 '/home/weblate/weblate.env/lib/python2.7/lib-old',
 '/home/weblate/weblate.env/lib/python2.7/lib-dynload',
 '/usr/lib/python2.7',
 '/usr/lib/python2.7/plat-x86_64-linux-gnu',
 '/usr/lib/python2.7/lib-tk',
 '/home/weblate/weblate.env/local/lib/python2.7/site-packages',
 '/home/weblate/weblate.env/lib/python2.7/site-packages']

Server time:    Do, 7 Jun 2018 15:59:32 +0000

fix it and you have your account soon maybe @nijel can help, seems related to https://github.com/WeblateOrg/weblate/issues/2074 and I'm not really in the mood to spend hours to fix it.

[agaida]

Available in weblate (working things are linked)

• compton-conf
• libfm-qt
• liblxqt
• lximage-qt
• lxqt-about
• lxqt-admin
  • lxqt-admin-time
  • lxqt-admin-user
• lxqt-config
  • liblxqt-config-cursor
  • lxqt-config
  • lxqt-config-appearance
  • lxqt-config-brightness
  • lxqt-config-file-associations
  • lxqt-config-locale
  • lxqt-config-monitor
• lxqt-globalkeys
• lxqt-notificationd
  • lxqt-config-notificationd
  • lxqt-notificationd
• lxqt-openssh-askpass
• lxqt-panel
  • panel
  • plugin-colorpicker
  • plugin-cpuload
  • plugin-desktopswitch
  • plugin-directorymenu
  • plugin-dom
  • plugin-kbindicator
  • plugin-mainmenu
  • plugin-mount
  • plugin-networkmonitor
  • plugin-quicklaunch
  • plugin-sensors
  • plugin-showdesktop
  • plugin-spacer
  • plugin-statusnotifier
  • plugin-taskbar
  • plugin-tray
  • plugin-volume
  • plugin-worldclock
• lxqt-policykit
• lxqt-powermanagement
  • lxqt-config-powermanagement
  • lxqt-powermanagement
• lxqt-runner
• lxqt-session
  • lxqt-config-session
  • lxqt-leave
  • lxqt-session
• lxqt-sudo
• obconf-qt
• pavucontrol-qt
• pcmanfm-qt
• qps
• qterminal
• qtermwidget
• screengrab

[yarons]

@agaida Yeah, that sucks, let's hope we can have some insights from @nijel.

It does look like there's some restriction on the DB, did you change something in the DB schema?

[agaida]

@yarons - i'm not the postgres pro - so it would cost me hours to a) locate and analyse the wrong constraint b) we are not alone, such things should be done upstream. Didn't think that it is a big thing - but for me it would be big. You might have noticed that i spended my time with other things like importing our projects. So no bad things happend right now :D

[yarons]

Well, there's a fix that can be applied manually or you can simply wait for 3.0.1, the bug is documented here: https://github.com/WeblateOrg/weblate/issues/2074

[yan12125]

The process looks kind of complicated -

you need to start again from 2.20 to test this

(From https://github.com/WeblateOrg/weblate/issues/2074#issuecomment-395969033)

BTW, looks like 3.0.1 is coming soon - https://github.com/WeblateOrg/weblate/commit/e468f154c9a7a60ae20bf45e8399f5b83d621a38

[HelixBot]

@yan12125 for us it is dead easy - the fix is confirmed, we have to wait for the additional migration - i can't and i don't want to rolback - that would mean that the database must be rolled back too - and that would mean: all newly migrated things would be lost - so a clear "No Way".

EDIT: or as @yarons suggests - apply the fix manually - will copy the container and give it a shot.

[HelixBot]

@yarons - it seems to me the damage is done :stuck_out_tongue: - so we had to kill fix the database things manually or just wait for the additional migration

[nijel]

The fixup migration is now in place as well: https://github.com/WeblateOrg/weblate/commit/939f5d3ac8b446d1b184d1715452749bfa508d66 , I might be able to release 3.0.1 tomorrow, but I can't promise that.

[HelixBot]

@nijel - cool, good news - thank you very much

[agaida]

sorry if i might sound a bit negative - it just don't work. period. maybe it will be the best idea to forget about it and just delete the container-

[agaida]

i heared rumors that QtLinguist just works fine - any other ideas?

Edit: Sorry if this don't sound nice and funny, but i'm more than a little bit pissed - that damn thing was at all approx. 10-15 man days. And right now i see no other choice than start from the very beginning. @nijel any ideas?

[agaida]

a few hours later - i'm about to loose the faith that weblate is a good idea - rationale: one migration let me start from the very beginning? Really? Not serious. Ok, at least it was a nice try and cost only a few man days. But if we go into production with this we should be sure that a installation will not be destroyed by a simple migration with no way back - maybe i'm wrong. Just prove me wrong. Right now i'm pissed, tired and not exactly happy and will have a long sleep.

[agaida]

@nijel - ok - calmed down a bit - is it possible just to dump the trans_* tables and load them into a new database? or doing this with only the tables that contain real data? like project definitions and components? If this is possible i would prefer this with a clean and new container - that would reduce the loss of time to a few hour - and wouldn't hurt much. The benefit of a brand new container would be worth it.

[nijel]

The 3.0.1 will work just fine, no need for any manual actions. Really this should not have happened, but the migration to 3.0 is really tricky due to changing user model, what is not supported directly by Django

10. června 2018 4:06:10 SELČ, Alf Gaida notifications@github.com napsal:

    @nijel - ok - calmed down a bit - is it possible just to dump the trans_* tables and load them into a new database? or doing this with only the tables that contain real data? like project definitions and components? If this is possible i would prefer this with a clean and new container - that would reduce the loss of time to a few hour - and wouldn't hurt much. The benefit of a brand new container would be worth it.

[tsujan]

@agaida Whatever decision you might make, please don't move ts files to anywhere else. They are where they should have been from the start.

[agaida]

@tsujan - i will not move anything. All these preps are fine and we would need it for any other application. The only thing i'm not happy right now is: Some implementation details in the weblate migrations. I know that migrating to a different data model have some pitfalls and can be made mostly only one time. And my problem with the old installation is: The new version might and will work. But my old installation is some kind of destroyed. And the additional migrations will never reached. There is a bug before. And the only thing i get is a not readable backtrace with no hint whats happend - and far more: Where (which migration) and why . So far me these three big W concern me a bit: What, Where, Why. And if that can happend again in future - maybe with a few hundred users on the system.

Bitten once, twice shy: I was there with pootle - upgrading the LXDE pootle instance with over 800 accounts at the time. After the upgrade the database model was kind of fucked and i was forced to go through every constraint and key and fix offending things manually - the upstream answer was: Oh, should not happend, our migtations do exactly what you have done manually :sunglasses: - erm, no. At this time starting with a new instance was no option - and we kind of managed it to get the thing to work again with only a remaining fault that occour from time to time.

So my suggestion is: Please, please, please - make future migrations more fault tolerant and verbose in a meaningful way - the admins of the systems will hail and praise you.

Sorry for the wall of text :smile:

@nijel - any way to have only the projects and component ex- and imported? Second question: The "old" installation was python2 - the next fear i would have. Do you think that setting up the new container with python3 would be a good idea? I really don't want to do the python2 -> python3 migration with a system in production so if possible the time would be now.

[agaida]

@nijel - the projects and components thing would be nice - tbh i'm not eager to type all the things again :smile: - just a lazy guy

[nijel]

The docker container is already on Python 3. The migration to Python 3 shouldn't be much effort, just fulltext index is better to be rebuilt (it was needed some time ago, not sure if recent Whoosh versions changed something here).

The export/import is not yet fully there. You can export everything using REST API and import components using import_json, projects will have to be created manually. Still after applying additional migration, your database will be perfectly okay, so there is really no need to do that.

10. června 2018 13:23:14 SELČ, Alf Gaida notifications@github.com napsal:

    @tsujan - i will not move anything. All these preps are fine and we would need it for any other application. The only thing i'm not happy right now is: Some implementation details in the weblate migrations. I know that migrating to a different data model have some pitfalls and can be made mostly only one time. And my problem with the old installation is: The new version might and will work. But my old installation is some kind of destroyed. And the additional migrations will never reached. There is a bug before. And the only thing i get is a not readable backtrace with no hint whats happend - and far more: Where (which migration) and why . So far me these three big W concern me a bit: What, Where, Why. And if that can happend again in future - maybe with a few hundred users on the system.

    Bitten once, twice shy: I was there with pootle - upgrading the LXDE pootle instance with over 800 accounts at the time. After the upgrade the database model was kind of fucked and i was forced to go through every constraint and key and fix offending things manually - the upstream answer was: Oh, should not happend, our migtations do exactly what you have done manually :sunglasses: - erm, no. At this time starting with a new instance was no option - and we kind of managed it to get the thing to work again with only a remaining fault that occour from time to time.

    So my suggestion is: Please, please, please - make future migrations more fault tolerant and verbose in a meaningful way - the admins of the systems will hail and praise you.

    Sorry for the wall of text :smile:

    @nijel - any way to have only the projects and component ex- and imported? Second question: The "old" installation was python2 - the next fear i would have. Do you think that setting up the new container with python3 would be a good idea? I really don't want to do the python2 -> python3 migration with a system in production so if possible the time would be now.