Missing utilities in gettext-tiny

[awilfox]

I realise it's probably very rare for these utilities to be called, hence they were not added. However, there are a few packages that use these utilities:

• [ ] msgcat Checked for by libbytesize, but never used; simple enough to remove.
• [ ] msgfilter This one actually is used by libbytesize for their Serbian translation. Our workaround has been to remove the Serbian translation from the build system.
• [ ] msgattrib Used by the gcompris educational suite for every translation. Since there is no native language, this package is actually unbuildable with gettext-tiny due to missing msgattrib.

I guess msgattrib would be the most important for Adélie, since the others we've been able to work around so far.

[xhebox]

So far, attrib is almost ignored(msgfmt will skip fuzzy marked translation). Since i assume the po files are correct. So the program wont fail even if we do not handle attrib at all, and everything will be fine.

msgattrib... Well, i guess we need to refactor the current poparser to fully support it.

What functions/options of msgattrib are used exactly, @awilfox ?

And what does @rofl0r think about: renew the poparser lib or make some workarounds/stub codes? I prefer the former because we could remove those ugly codes in msgfmt once we have poparser supporting attrib parsing. I think it's a good chance to clean the code.

If we choose to workaround the problem, we should solve this issue before the new release. Otherwise, we should tag this issue as "enchancement", it will take me sometime to finish. It's likely to be a big update on this project.

[rofl0r]

i don't know what msgattrib is doing exactly. since there's only one package using it i would suggest talking to its maintainers about the situation and suggest them to ship the output of msgattrib as part of their release tarball or convince them to stop using it. alternatively the distro could produce said output with the gnu tool and add it as a patch to their repository.

[xhebox]

Attrib are those marks, like fuzzy. What we do now is actually searching it using strstr() and skip it if we find such a flag.

In my view we should separate it out and make parsing attrib a function of poparser rather than working around(hardcoded) in the process function of poparser:

        if(lp[0] == '#') {
                char *s;
                if(s = strstr(lp, ", fuzzy")) {
                        if(fuzzymark != 0) fuzzymark++;
                        else fuzzymark=2;
                }
                inv:
                *stringstart = 0;
                return pe_invalid;
        }

Despite msgattrib, i actually wanted to separate it out at the time i introduced skipping fuzzy. If we code such a function, we could abort safely when we meet some unsupported flags.

I did not have enough time to do it before. Yet, now i have more than one months off... Do we really need to workaround it in such a way, if we can handle this using just less than 100 lines of code in poparser.c/.h, a msgattrib.c just like msgmerge.c, and maybe some improvements on code? @rofl0r

EDIT: i'll back to irc 3 or 4 days later, after checking my college admissions results.

[rofl0r]

In my view we should separate it out

yeah, parsing it properly could well be preferable.

If we code such a function, we could abort safely when we meet some unsupported flags

wouldn't that introduce regressions? like packages that can be processed RN due to skipping would then abort ?

I did not have enough time to do it before. Yet, now i have more than one months off...

sure, i have no issues with you wanting to work on that functionality and implement that tool. one thing one should keep in mind is that once it's there it needs to be maintained - that means it could cause a lot more work in the long term than initally budgeted.

as in this case it looks like almost nobody uses it, it might be worth to chat with the maintainer of that package as i suggested above (this is basically the way sabotage has always dealt with packages that use rare GLIBC functions: we lobbied them to use portable constructs, while alpine linux has always lobbied @richfelker to add yet another exotic crux to musl). what's your opinion on this, @awilfox ?

[xhebox]

wouldn't that introduce regressions? like packages that can be processed RN due to skipping would then abort ?

@rofl0r

My idea is still the same, i am not meant to really support those format-flags. But our processing on sysdep(PRIu32) is limited to c-like-format, we can abort when we met a non-c-like-format flag explicitly.

that means it could cause a lot more work in the long term than initally budgeted

Whatever it's a chance, i am not going to give up the project so quickly.

[awilfox]

as in this case it looks like almost nobody uses it, it might be worth to chat with the maintainer of that package as i suggested above (this is basically the way sabotage has always dealt with packages that use rare GLIBC functions: we lobbied them to use portable constructs, while alpine linux has always lobbied @richfelker to add yet another exotic crux to musl). what's your opinion on this, @awilfox ?

I'm not sure that's a fair assessment of either side.

I, too, have lobbied Rich to add what you may consider to be "exotic crux"es to musl – namely, things that are not present in POSIX but present on all major Unices (which for me, includes Linux, FreeBSD, NetBSD, OpenBSD, Solaris, and at least one of AIX, HP-UX, Irix, or Tru64). I'll agree with you that Alpine's hack-and-slash approach to "just make things run" isn't exactly tenable. (Adélie exists because of that fact, and that's a big reason we were excited to upstream proper fixes to them.) However, Sabotage has some hack-and-slash patches too. Hell, we even have one. It happens, especially when maintainers don't care about the packages they are hacking and slashing, or a package is too convoluted to write a correct fix in the time allotted to us as packagers.

I am not sure if you could make the argument that msgattrib is "not portable". There's only one gettext package on every Unix that is not Linux/musl, and that is GNU's. This project, while being of much higher quality and much lower bloat than GNU gettext, is not yet widely enough distributed (and likely never will be) to warrant sops in other build systems. I do however agree that the Serbian translation in libbytesize should be shipped with it and that means msgfilter shouldn't really be necessary.

I've just found another user of msgcat, however: KWave. It's only used to create new translations, though, so the fact it is required for every build is definitely a bug.

I suppose the real question is: What is the project goal of gettext-tiny? Does it intend to replace GNU gettext for packagers / distributions? Should people writing the translations install GNU gettext? Because at some point writing translations will require all these tools, and that would mean actual upstream developers would need to use GNU gettext to author them, instead of being able to use gettext-tiny.

If the goal is only to ever work for building binary packages, and not to allow authoring of translations on a musl computer, then it makes sense to not add any of those tools.

[xhebox]

The goal is to build packages, not replace GNU. But it's not simple.

Check gnu doc, msgcat, msgfilter and msgattrib is in the category of Manipulating PO Files, while msgfmt and msgunfmt is in the ctegory of Producing Binary MO Files.

But the thing is, build systems will use these tools to create po files automatically for convenience while they can actually distribute po files built by GNU/gettext directly. gettext-tiny already has some as a compromise: xgettext and msgmerge(they are too widely used).

The problem here is not if msgattrib is portable. I believe introducing it is definitely a good thing for the code of the project. And you cant say that's breaking the goal of the project if they are needed for building.

The problem is that if it's worth to implement it while there're only little packages using it. After all, they should give po files directly when they say distribution, rather than building it on the fly.

I suggest awilfox to contact the developers first, to see if then can make a difference. But if they did not reply, or they did not accept, i will try to implement these tools. How does it sound like, @rofl0r, @awilfox ? Anyway, there must be one side trying to move.

By the way, did you test #27, @awilfox ? It should be solved right now. Please verify the result and close the issue if it's OK.

[awilfox]

I'll contact upstream with gcompris. Not sure it's worth it for libbytesize since only Serbian translations are affected, at least right now while I'm under a lot of pressure to ship beta1.

27 is indeed fixed, thank you very much for your work on it!

[xhebox]

Good to see that it works :)