Report of research with patched pyside2-lupdate

[Kuzma30]

Want share some info and results of research. (https://github.com/Kuzma30/FreeCAD/commits/master)

1. I build https://github.com/chennes/pyside2-debian with python3.10 (after applying patches from debian/patches dir)
2. Then I run updatets.py with this version. In my fork patch for AddonManager was previously added (https://github.com/Kuzma30/FreeCAD/commit/030b10a4dc4a117fcd6ca9b6004e7c0245cabbf4) It add strings from Path Draft Arch and delete from Part.
3. Then I run Black for all FreeCAD source. log deleted as it very big :)
4. Then I make test changes in one .py file where was defined _translate() function and replace it to translate = FreeCAD.Qt.translate. Then I make rerun updatets.py and this messages start shown in crowdin. Conclusion _translate ("Some texts") not recognized by patched pyside2-lupdate, but recognized with translate().
5. I decide replace _translate with translate(). After rerun updatets.py it add messages in crowdin to Part (36 strings same as deleted on 1st step) Image.

Conclusion. patched pyside2-lupdate works fine with comma at the end of string (from Black) and works with multiline string if they use translate() (not _translate())

ping @chennes

[Kuzma30]

Report from crowdin

[Kuzma30]

Still have unicode problem..

[chennes]

Yes, I think @yorikvanhavre was going to test to see if changing the default fro 8859 to UTF-8 resolved that.

[yorikvanhavre]

this is super useful research! i don't think _translate is used a lot, but might be a good time to kill them all. yes, i'll try to do the unicode fix tomorrow

[Kuzma30]

After changing codec to UTF-8 in pyside2-debian it still not recognize Unicode... Looks like it will be better add support for ending comma to the pyqt5-debian @chennes @yorikvanhavre

[chennes]

Can you test the Unicode support in the official Qt 6.x dev branch? If it needs fixing there, then it's probably better to resolve the unicode issue instead (since the fix will be similar).

[Kuzma30]

I have not Qt6 yet. I will spend much time with it to build dependencies (I think)

[chennes]

You don't have to build all of Qt6, just Linguist. I was able to build it without much pain (that I remember!).

[Kuzma30]

What repository is it?

[chennes]

https://code.qt.io/cgit/qt/qttools.git

[Kuzma30]

Ok I build QT6 source and qttools too. Now I have such binary:

~/QT/bin$ ls -1
androiddeployqt
androidtestrunner
assistant
balsam
balsamui
canbusutil
designer
instancer
lconvert
linguist
lrelease
lupdate
materialeditor
meshdebug
pixeltool
qdbus
qdbuscpp2xml
qdbusviewer
qdbusxml2cpp
qdistancefieldgenerator
qdoc
qmake
qmake6
qml
qmldom
qmleasing
qmlformat
qmllint
qmlls
qmlplugindump
qmlpreview
qmlprofiler
qmlscene
qmltc
qmltestrunner
qmltime
qsb
qt-cmake
qt-cmake-private
qt-cmake-private-install.cmake
qt-cmake-standalone-test
qt-configure-module
qtdiag
qtdiag6
qtpaths
qtpaths6
qtplugininfo
shadergen
shapegen

There is lupdate but no pylupdate.

./lupdate
Usage:
    lupdate [options] [project-file]...
    lupdate [options] [source-file|path|@lst-file]... -ts ts-files|@lst-file

lupdate is part of Qt's Linguist tool chain. It extracts translatable
messages from Qt UI files, C++, Java and JavaScript/QtScript source code.
Extracted messages are stored in textual translation source files (typically
Qt TS XML). New and modified messages can be merged into existing TS files.

Passing .pro files to lupdate is deprecated.
Please use the lupdate-pro tool instead.

Options:
    -help  Display this information and exit.
    -no-obsolete
           Drop all obsolete and vanished strings.
    -extensions <ext>[,<ext>]...
           Process files with the given extensions only.
           The extension list must be separated with commas, not with whitespace.
           Default: 'java,jui,ui,c,c++,cc,cpp,cxx,ch,h,h++,hh,hpp,hxx,js,qs,qml,qrc'.
    -pluralonly
           Only include plural form messages.
    -silent
           Do not explain what is being done.
    -no-sort
           Do not sort contexts in TS files.
    -no-recursive
           Do not recursively scan directories.
    -recursive
           Recursively scan directories (default).
    -I <includepath> or -I<includepath>
           Additional location to look for include files.
           May be specified multiple times.
    -locations {absolute|relative|none}
           Specify/override how source code references are saved in TS files.
           Guessed from existing TS files if not specified.
           Default is absolute for new files.
    -no-ui-lines
           Do not record line numbers in references to UI files.
    -disable-heuristic {sametext|similartext|number}
           Disable the named merge heuristic. Can be specified multiple times.
    -project <filename>
           Name of a file containing the project's description in JSON format.
           Such a file may be generated from a .pro file using the lprodump tool.
    -pro <filename>
           Name of a .pro file. Useful for files with .pro file syntax but
           different file suffix. Projects are recursed into and merged.
           This option is deprecated. Use the lupdate-pro tool instead.
    -pro-out <directory>
           Virtual output directory for processing subsequent .pro files.
    -pro-debug
           Trace processing .pro files. Specify twice for more verbosity.
    -source-language <language>[_<region>]
           Specify the language of the source strings for new files.
           Defaults to POSIX if not specified.
    -target-language <language>[_<region>]
           Specify the language of the translations for new files.
           Guessed from the file name if not specified.
    -tr-function-alias <function>{+=,=}<alias>[,<function>{+=,=}<alias>]...
           With +=, recognize <alias> as an alternative spelling of <function>.
           With  =, recognize <alias> as the only spelling of <function>.
           Available <function>s (with their currently defined aliases) are:
             Q_DECLARE_TR_FUNCTIONS (=Q_DECLARE_TR_FUNCTIONS)
             QT_TR_N_NOOP (=QT_TR_N_NOOP)
             QT_TRID_N_NOOP (=QT_TRID_N_NOOP)
             QT_TRANSLATE_N_NOOP (=QT_TRANSLATE_N_NOOP)
             QT_TRANSLATE_N_NOOP3 (=QT_TRANSLATE_N_NOOP3)
             QT_TR_NOOP (=QT_TR_NOOP)
             QT_TRID_NOOP (=QT_TRID_NOOP)
             QT_TRANSLATE_NOOP (=QT_TRANSLATE_NOOP)
             QT_TRANSLATE_NOOP3 (=QT_TRANSLATE_NOOP3)
             QT_TR_NOOP_UTF8 (=QT_TR_NOOP_UTF8)
             QT_TRANSLATE_NOOP_UTF8 (=QT_TRANSLATE_NOOP_UTF8)
             QT_TRANSLATE_NOOP3_UTF8 (=QT_TRANSLATE_NOOP3_UTF8)
             findMessage (=findMessage)
             qtTrId (=qtTrId)
             tr (=tr)
             trUtf8 (=trUtf8)
             translate (=translate)
             qsTr (=qsTr)
             qsTrId (=qsTrId)
             qsTranslate (=qsTranslate)
    -ts <ts-file>...
           Specify the output file(s). This will override the TRANSLATIONS.
    -version
           Display the version of lupdate and exit.
    -clang-parser [compilation-database-dir]
           Use clang to parse cpp files. Otherwise a custom parser is used.
           This option needs a clang compilation database (compile_commands.json)
           for the files that needs to be parsed.
           The path to the directory containing this file can be specified on the 
           command line, directly after the -clang-parser option, or in the .pro file
           by setting the variable LUPDATE_COMPILE_COMMANDS_PATH.
           A directory specified on the command line takes precedence.
           If no path is given, the compilation database will be searched
           in all parent paths of the first input file.
    -project-roots <directory>...
           Specify one or more project root directories.
           Only files below a project root are considered for translation when using
           the -clang-parser option.
    @lst-file
           Read additional file names (one per line) or includepaths (one per
           line, and prefixed with -I) from lst-file.

There is no .py files in list

./lupdate -version
lupdate version 6.4.0

[chennes]

The lupdate command now processes Python, there is no separate pylupdate anymore. It just goes off the extension of the input file, if I recall correctly.

[Kuzma30]

Test on Draft.ts. Look like it recognize unicode fine. Lines with comma it adds too. I have only one problem with Part. Script stop on it and after long waiting I press Ctrl-C. Script continue work.

I change PYLUPDATE to LUPDATE in script.

Edit. Script stops here

sed 's/<translation.*>.*<\/translation>/<translation type="unfinished"><\/translation>/g' Gui/Resources/translations/Part.ts > Gui/Resources/translations/Part.ts.temp
mv Gui/Resources/translations/Part.ts.temp Gui/Resources/translations/Part.ts
/home/alexandr/Qt/6.4.0/gcc_64/bin/lupdate -no-obsolete `find ./ -name "*.py"` -ts Gui/Resources/translations/Partpy.ts

ping @chennes

[Kuzma30]

I found what file make problem. It is FreeCAD/src/Mod/Part/JoinFeatures.py. After deleting commented part of code (except header) it start work fine.

Edit. Conclusion. _translate() don't recognize. We need cleanup code from it. Unicode (test on "Bézier curve") and comma ending strings looks like work with lupdate 6.4.0. We don't need build it. We can install it from QT online installer qt-unified-linux-x64-4.3.0-1-online.run ping @yorikvanhavre

[Kuzma30]

Some code cleanup (not tested in all situations) https://github.com/Kuzma30/FreeCAD/commit/3a3d1cea99b37238ecdbad2a5366a926b34ebae9

I found also many place in code where translate() function is defined. For example here https://github.com/Kuzma30/FreeCAD/blob/a7462e60f8eb440b19e78bf8638aa0e9a24cbd79/src/Mod/Arch/ArchAxis.py#L26-L41 Is my code correct (translate = FreeCAD.Qt.translate) or I must made it like in ArchAxis.py?

[chennes]

We don't need build it. We can install it from QT online installer qt-unified-linux-x64-4.3.0-1-online.run

Does that version include my trailing comma fix? I think it's scheduled for release in 6.4.

[Kuzma30]

I use dev branch for build (and dev snapshot on online installer) I think we need change updatets.py script to make it work with only 6.4 version (another version get incorrect results)

Edit. I also think that we need one update of .ts before release (to get time for translators) and second update on release. But we need decide what to do with _translate().

[chennes]

I think after the 0.20 release I will have time to refactor the updatets.py script to better support the Qt 6.4 version of lupdate: we can simplify the script quite a bit, removing the generation of the .pro file and just directly doing a search for *.cpp, *.h, *.ui, and *.py files, all at once (and .java and .js if needed as well).

What was the _translate() supposed to do?

[Kuzma30]

https://github.com/Kuzma30/FreeCAD/commit/3a3d1cea99b37238ecdbad2a5366a926b34ebae9 @chennes look here. Messages with _translate() in *.py not recognized by lupdate.

[chennes]

Right, I wouldn't expect them to be: what was the point of the underscore in the first place?

[Kuzma30]

This is not my code(with _translate()) :) I propose refactor this code to translate()

[Kuzma30]

@yorikvanhavre I want save you some time I prepare changes (disable _translate patch) and run updatets.py with latest version of lupdate (6.4.0). Can you look this and merge https://github.com/Kuzma30/FreeCAD/commits/test_up ?

[yorikvanhavre]

There are many commits in that branch... Not sure what I must do? In any case, indeed replacing all _translate with translate seem absolutely necessary and urgent. Doing that today.

[Kuzma30]

@yorikvanhavre One commit is replacing _translate with translate https://github.com/Kuzma30/FreeCAD/commit/8a69188cd97d37552c0002fdb3a6a5d65f0fba35 https://github.com/Kuzma30/FreeCAD/commit/7f716a5d6b30be7fb44ecf9220d35298f7240a60 this is master commit for .ts. You can merge it without need of updating your lupdate. As variant you can make it himself, but you need install lupdate 6.4.0 (dev) version and edit updatets.py script. https://github.com/Kuzma30/FreeCAD/commit/418c9102a7a33f28813f76b81fa455bb54e2b900 this commit fixes founded typo

[yorikvanhavre]

I'd like to merge the first one, but I don't think we should remove all the commented out commands in JoinFeatures, usually when someone comments out blocks of code like this is because they think it can/should still be used. It's always best to follow a "one commit, one feature" policy

[Kuzma30]

@yorikvanhavre with commented part of code lupdate (6.4.0) stops. I describe this in prev comments.

[yorikvanhavre]

Wow, that's weird... But we should find out which part I think, probably better than deleting the whole block. I'll look at it too when possible

[Kuzma30]

@yorikvanhavre I can made commit that return commented code. But I think it is not principal now. Can you merge .ts update to make some translators happy, and give some time to translate new strings before final release crowdin update?

[yorikvanhavre]

Done: FreeCAD/FreeCAD@b903e949ae Updating crowdin now. I'll try lupdate 6.4...

[chennes]

If we can figure out what the problem is with lupdate I can make a patch upstream, the process was fairly painless. I will be back at a computer Thursday or Friday.

[Kuzma30]

I am trying delete commented blocks by small parts. Lupdate stops all time. When I delete all commented blocks it start working

[Kuzma30]

@yorikvanhavre I prepare commit https://github.com/FreeCAD/FreeCAD/pull/6932 Here I change _translate to translate. File with problem for lupdate I stay as is. Are you try lupdate 6.4.0?

[yorikvanhavre]

Getting pylupdate 6.4.0 now...

[Kuzma30]

There is no pylupdate-6.4.0 only lupdate for all format.

[Kuzma30]

@yorikvanhavre There are many new commits related to translation. Most of them require update .ts files. Do you install lupdate 6.4.0? Can you make commit with updated .ts?

[yorikvanhavre]

I cannot make lupdate 6.4 compile... Can you make a PR in the meantime?

[Kuzma30]

@yorikvanhavre No need compile itself. You can use online installer from QT https://www.qt.io/download-qt-installer?hsCtaTracking=99d9dd4f-5681-48d2-b096-470725510d34%7C074ddad0-fdef-4e53-8aa8-5e8a876d6ab4 and select Qt Design Studio 3.4.0-snapshot and Qt 6.4.0 Snapshot from "dev"

[yorikvanhavre]

Ok trying again now

[Kuzma30]

https://github.com/FreeCAD/FreeCAD/pull/6984 I made PR, @yorikvanhavre

[Kuzma30]

@yorikvanhavre Any progress with installing lupdate 6.4.0?

[yorikvanhavre]

Yes, i succeeded, and by running it i get the exact same output as your last PR, so I guess we're all set!

[Kuzma30]

Excellent.

[Kuzma30]

As we have working solution, I close issues.