mike-fabian
commented
2 years ago What system do you run the build on and how do you do the build?
Closed kloczek closed 2 years ago
mike-fabian
commented
2 years ago What system do you run the build on and how do you do the build?
kloczek
commented
2 years ago What system do you run the build on and how do you do the build?
I'm ruuning that on Linux inside LXC zone created to only build that package. Part of my spec filre used to build ibus-typing-booster:
%build
autoreconf -fiv
%configure \
PYTHON=%__python3 \
--disable-additional \
%{nil}
%make_build -C po update-po
%make_buildSo after unpacking source tar ball I'm just regenerate all GNO tools files and run configure without any specyfic params.
BTW: it woud be good to update configure.ac for latest autoconf 2.71 (look on the autoreconf warnings).
mike-fabian
commented
2 years ago Apparently this already fails on your system:
$ python3
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
>>> Is that right? Do you get an error already when you try this?
kloczek
commented
2 years ago Ha .. so libvoikko python module needs to be added to build dependencies 🤔 .. OK .. added.
[tkloczko@devel-g2v SPECS]$ python3
Python 3.8.13 (default, Apr 5 2022, 21:00:20)
[GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.8/site-packages/libvoikko.py", line 447, in __init__
raise VoikkoException("Initialization of Voikko failed: " + unicode_str(error.value, "UTF-8"))
libvoikko.VoikkoException: Initialization of Voikko failed: No valid dictionaries were found
>>>Hmm .. more tahn only modle needs to be added 🤔
I'll try to add build env necessary dictionaries however I think that that diff in output has a bit different bacgroud. Am I right or not? 🤔 If you see in above output something else missing in my build env please let me know.
mike-fabian
commented
2 years ago I think I will add code to skip that voikko test case if the initialization libvoikko.Voikko('fi') fails for whatever reason.
mike-fabian
commented
2 years ago By the way, did the same failures already occur with the previous ibus-typing-booster release https://github.com/mike-fabian/ibus-typing-booster/releases/tag/2.15.27 ?
I still thinking why the emoji test cases fail on your system.
mike-fabian
commented
2 years ago Ha .. so
libvoikkopython module needs to be added to build dependencies thinking .. OK .. added. ... I'll try to add build env necessary dictionaries however I think that that diff in output has a bit different bacgroud. Am I right or not? thinking If you see in above output something else missing in my build env please let me know.
Still searching ...
Does
$ python3
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
>>>work for you now?
kloczek
commented
2 years ago work for you now?
I still don't know what is missing for that python module 🤔
mike-fabian
commented
2 years ago On Fedora 35, the 3 packages python3-libvoikko, libvoikko, and voikko-fi are required for the python module to work.
python3-libvoikko requires the other two though:
[mfabian@fedora ~]$ cat /etc/fedora-release
Fedora release 36 (Thirty Six)
[mfabian@fedora ~]$ rpm -qa | grep voikko
voikko-fi-2.5-1.fc36.noarch
libvoikko-4.3.1-3.fc36.x86_64
python3-libvoikko-4.3.1-3.fc36.noarch
[mfabian@fedora ~]$ python3
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
>>>
[mfabian@fedora ~]$ When I try to remove voikko-fi, I see that it is needed by libvoikko:
[mfabian@fedora ~]$ sudo rpm -e voikko-fi
error: Failed dependencies:
voikko-fi is needed by (installed) libvoikko-4.3.1-3.fc36.x86_64
[mfabian@fedora ~]$And if I force removal ignoring the dependency
[mfabian@fedora ~]$ sudo rpm -e voikko-fi --nodeps Then I get the same failure you see in the test suite:
[mfabian@fedora ~]$ python3
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
File "/usr/lib/python3.10/site-packages/libvoikko.py", line 447, in __init__
raise VoikkoException("Initialization of Voikko failed: " + unicode_str(error.value, "UTF-8"))
libvoikko.VoikkoException: Initialization of Voikko failed: No valid dictionaries were found
>>>
[mfabian@fedora ~]$And installing it again fixes the problem again:
[mfabian@fedora ~]$ sudo dnf install voikko-fi
Last metadata expiration check: 0:02:13 ago on Thu 09 Jun 2022 17:09:26 CEST.
Dependencies resolved.
=============================================================================================
Package Architecture Version Repository Size
=============================================================================================
Installing:
voikko-fi noarch 2.5-1.fc36 fedora 1.4 M
Transaction Summary
=============================================================================================
Install 1 Package
Total download size: 1.4 M
Installed size: 3.8 M
Is this ok [y/N]: y
Downloading Packages:
voikko-fi-2.5-1.fc36.noarch.rpm 734 kB/s | 1.4 MB 00:01
---------------------------------------------------------------------------------------------
Total 636 kB/s | 1.4 MB 00:02
Running transaction check
Transaction check succeeded.
Running transaction test
Transaction test succeeded.
Running transaction
Preparing : 1/1
Installing : voikko-fi-2.5-1.fc36.noarch 1/1
Running scriptlet: voikko-fi-2.5-1.fc36.noarch 1/1
Verifying : voikko-fi-2.5-1.fc36.noarch 1/1
Installed:
voikko-fi-2.5-1.fc36.noarch
Complete!
[mfabian@fedora ~]$ python3
Python 3.10.4 (main, Mar 25 2022, 00:00:00) [GCC 12.0.1 20220308 (Red Hat 12.0.1-0)] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import libvoikko
>>> voikko = libvoikko.Voikko('fi')
>>>
[mfabian@fedora ~]$ ====================================================================== FAIL: test_similar_camel_es_ES (test_emoji_similar.EmojiSimilarTestCase)
Traceback (most recent call last): File "/home/tkloczko/rpmbuild/BUILD/ibus-typing-booster-2.16.0/tests/test_emoji_similar.py", line 116, in test_similar_camel_es_ES self.assertEqual( AssertionError: Lists differ: [('🐫', 'camello [🐫, nature]', 2), ('🐪', 'dromedario [nature]'[85 chars], 1)] != [('🐫', 'camello [🐫, bactriano, camello, desierto, dromedario,[203 chars], 3)]
First differing element 0: ('🐫', 'camello [🐫, nature]', 2) ('🐫', 'camello [🐫, bactriano, camello, desierto, dromedario, jorobas]', 6)
- [('🐫', 'camello [🐫, nature]', 2),
- ('🐪', 'dromedario [nature]', 1),
- ('🦙', 'llama [nature]', 1),
- ('🐐', 'cabra [nature]', 1),
- ('🦒', 'jirafa [nature]', 1)]
- [('🐫', 'camello [🐫, bactriano, camello, desierto, dromedario, jorobas]', 6),
- ('🐪', 'dromedario [camello, desierto, dromedario]', 3),
- ('🏜️', 'desierto [desierto]', 1),
- ('🐫', 'cammello [🐫, animale, cammello, gobba]', 4),
- ('🐪', 'dromedario [animale, cammello, gobba]', 3)]
If you run that test case manually in the tests directory, what is the output?
It should look like this:
mfabian@taka:/local/mfabian/src/ibus-typing-booster/tests (release-candidate-2.16.1 $)
$ ./test_emoji_similar.py EmojiSimilarTestCase.test_similar_camel_es_ES
itb_emoji.find_cldr_annotation_path('en')->/local/mfabian/src/ibus-typing-booster/tests/../engine/../data/annotations/en.xml
.
----------------------------------------------------------------------
Ran 1 test in 1.089s
OK
mfabian@taka:/local/mfabian/src/ibus-typing-booster/tests (release-candidate-2.16.1 $)
$ i.e. it should print the path where the cldr emoji annotations are loaded from, in my case this is '/local/mfabian/src/ibus-typing-booster/tests/../engine/../data/annotations/en.xml'.
So this should print a path pointing to the cldr emoji annotations included in the ibus-typing-booster sources.
If emoji annotations are loaded from any other place, the emoji test cases are skipped because in that case the emoji annotation data is most likely older than those included in ibus-typing-booster. The emoji features in ibus-typing-booster will still work when loading the emoji annotations from an external location, but when the files are not up-to-date, some test cases fail. Therefore, I try to skip the emoji test cases when I can detect that the emoji annotation data is loaded from an external directory, see this code in the test case:
https://github.com/mike-fabian/ibus-typing-booster/blob/main/tests/test_emoji_similar.py#L45
@unittest.skipIf(
'..' not in itb_emoji.find_cldr_annotation_path('en'),
'Using external emoji annotations: %s '
% itb_emoji.find_cldr_annotation_path('en')
+ 'Testing with older emoji annotations instead '
'of those included in the ibus-typing-booster source is likely '
'to create meaningless test failures.')
class EmojiSimilarTestCase(unittest.TestCase):If the emoji annotations are loaded from a different place in your build, then I wonder why the test case was not skipped.
mike-fabian
commented
2 years ago Can you check from where the emoji annotations are loaded during your build?
mike-fabian
commented
2 years ago I have fixed the problem with the test case for voikko by skipping the test case if voikko initialization fails
I did not yet fix the problem with the emoji test cases for you. They work for me and for some reason the emoji annotation files seem to be loaded from a different directory in your system.
If you want me to help debugging this, you can open a new issue for the fails in the emoji test cases.
kloczek
commented
2 years ago Just trested 1.16.1 and I see that still it fails in that unit. I have installed in buukd env cldr-emoji-annotation. Is it what excly is needed or it is something else?
IMO it would be better to not skip voikko test if initialisation fail because it is exectly what testsuite suppose to do which is exposing some issues.
mike-fabian
commented
2 years ago Just trested 1.16.1 and I see that still it fails in that unit.
I didn’t change anything with the emoji tests yet, only with the voikko test.
I have installed in buukd env cldr-emoji-annotation. Is it what excly is needed or it is something else?
No, that is an external package. What is needed are the emoji annotations included in the ibus-typing-booster sources:
mfabian@taka:/local/mfabian/src/ibus-typing-booster (release-candidate-2.16.2 *$)
$ ls data/annotations
Makefile da.xml fr.xml ka.xml mt.xml sa.xml ti.xml
Makefile.am de.xml fr_CA.xml kab.xml my.xml sat.xml tk.xml
Makefile.in de_CH.xml ga.xml kk.xml nb.xml sc.xml to.xml
af.xml doi.xml gd.xml kl.xml ne.xml sd.xml tr.xml
am.xml dsb.xml gl.xml km.xml nl.xml si.xml tt.xml
ar.xml el.xml gu.xml kn.xml nn.xml sk.xml ug.xml
ar_SA.xml en.xml ha.xml ko.xml no.xml sl.xml uk.xml
as.xml en_001.xml ha_NE.xml kok.xml oc.xml so.xml ur.xml
ast.xml en_AU.xml he.xml ku.xml or.xml sq.xml uz.xml
az.xml en_CA.xml hi.xml ky.xml pa.xml sr.xml vi.xml
be.xml en_GB.xml hi_Latn.xml lb.xml pa_Arab.xml sr_Cyrl.xml wo.xml
bg.xml en_IN.xml hr.xml lo.xml pcm.xml sr_Cyrl_BA.xml xh.xml
bn.xml es.xml hsb.xml lt.xml pl.xml sr_Latn.xml yo.xml
br.xml es_419.xml hu.xml lv.xml ps.xml sr_Latn_BA.xml yo_BJ.xml
bs.xml es_MX.xml hy.xml mai.xml pt.xml su.xml yue.xml
ca.xml es_US.xml ia.xml mi.xml pt_PT.xml sv.xml yue_Hans.xml
ccp.xml et.xml id.xml mk.xml qu.xml sw.xml zh.xml
ceb.xml eu.xml ig.xml ml.xml rm.xml sw_KE.xml zh_Hant.xml
chr.xml fa.xml is.xml mn.xml ro.xml ta.xml zh_Hant_HK.xml
ckb.xml fi.xml it.xml mni.xml root.xml te.xml zu.xml
cs.xml fil.xml ja.xml mr.xml ru.xml tg.xml
cy.xml fo.xml jv.xml ms.xml rw.xml th.xml
mfabian@taka:/local/mfabian/src/ibus-typing-booster (release-candidate-2.16.2 *$)
$ You should have these if you build from a git checkout or a release tarball.
These emoji annotation files are searched like this:
DATADIR = os.path.join(os.path.dirname(__file__), '../data')
# USER_DATADIR will be “~/.local/share/ibus-typing-booster/data” by default
USER_DATADIR = itb_util.xdg_save_data_path('ibus-typing-booster/data')
CLDR_ANNOTATION_DIRNAMES = (
USER_DATADIR, DATADIR,
# On Fedora >= 25 there is a
# “cldr-emoji-annotation” package which has the
# .xml files here in the subdirs “annotations”
# and “annotationsDerived”:
'/usr/share/unicode/cldr/common/',
'/local/mfabian/src/cldr/common/')I.e. first ~/.local/share/ibus-typing-booster/data is checked, second ../data, third /usr/share/unicode/cldr/common/.
Your cldr-emoji-annotation package most likely installs to /usr/share/unicode/cldr/common/. Is that right?
If the emoji annotations are loaded from anywhere else but ../data, the tests should be skipped. With different versions of the emoji annotation files there will most likely be test failures and I cannot know which version is installed on the system where the test runs. The files in ../data from the ibus-typing-booster sources are always the latest versions from the CLDR git repo at the time when the release of ibus-typing-booster was done. So they are usually newer versions than those in external packages.
To save space in the ibus-typing-booster package, you may
1) gzip the .xml files
or
2) delete the .xml files from the ibus-typing-booster package and rely on loading the files from an external package.
Doing 2) is perfectly fine, it doesn’t matter much if the files with the emoji names and keywords are a bit old, still works fine. But the test cases are always adapted to the very latest versions. I.e. you should not delete the .xml files in ../data/annotations and ../data/annotations/Derived before running the test cases. When the test cases have succeeded, you may delete them in order to not include them in your package.
There is code in the emoji test cases which skips the tests if the files are loaded from other locations than ../data/. Somehow this does not seem to work in your build. The failures clearly show that different versions of these files were loaded, not those from ../data. In that case the test cases should be skipped, but that doesn’t seem to work for you.
IMO it would be better to not skip voikko test if initialisation fail because it is exectly what testsuite suppose to do which is exposing some issues.
If you don’t want to skip the test case for voikko, just install the requirements for this test case to run. I always have these requirements installed when building for Fedora and openSUSE so I run these test cases very often.
But I don’t want to make the voikko stuff a hard requirement for everybody. voikko is needed only for Finnish spellchecking. Typing Booster works fine without voikko, it only lacks spellchecking support for Finnish then.
Like it is in the subject .. test suite is failing