search

vanyossi / artscript

Artscript is a small app to easy convert from production file images(KRA,XCF,PSD,ORA,SVG) to universal formats(JPG, PNG, GIF or WEBM). If installed it uses gimp, inkscape and calligra in console mode to render PNG files in batch.
GNU General Public License v3.0
27 stars 5 forks source link

Calligraconvert fails to send eof status to artscript causing it to hang #59

Closed 0ion9 closed 10 years ago

0ion9 commented 10 years ago

I start up ./artscript2.tcl and add a single ora file for conversion (valuedrills.ora). I click the text field and type in "TEST TEXT". Then, I click Convert (leaving all other parameters at default values).

The result is saved to /tmp/valuedrills.png, and contains no text.

Here is a log of artscript's console output.

Tk drag and drop enabled Starting File extractions... Starting Convert... Extract valuedrills.ora Rendering Kriters: Calligraconvert always return errors! Compiled for arch: ::Vc::SSE2Impl Features supported: "SSE2" --- yes "SSE3" --- no "SSSE3" --- no "SSE4.1" --- no "SSE4.2" --- no "SSE4a" --- no "AVX " --- no Connecting to deprecated signal QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString) QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave. QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave. kbuildsycoca4 running... kbuildsycoca4(1469) KConfigGroup::readXdgListEntry: List entry Categories in "/usr/share/applications/gnome-screenruler.desktop" is not compliant with XDG standard (missing trailing semicolon). QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver Starting File extractions... Starting Convert... converting... valuedrills.ora Generating preview convert -quiet "/home/kau/xra/2013/Jicmu/zas/valuedrills.ora" -gravity NorthEast /tmp/artk-tmp-wtmk.png -compose dissolve -define compose:args=80 -geometry +0+0 -composite -type TrueColorMatte -define png:format=png32 -define png:compression-level=9 -define png:compresion-filter=4 show: Extract valuedrills.ora file /tmp/valuedrills.png found! File extractions finished convert: no decode delegate for this image format /home/kau/xra/2013/Jicmu/zas/valuedrills.ora' @ error/constitute.c/ReadImage/555. convert: no images definedshow:' @ error/convert.c/ConvertImageCommand/3144. some error: child process exited abnormally Convert done. Convert done.

There are a few things to note in the log:

Info on package versions:

0ion9 commented 10 years ago

Note that it does not fail for all files. For example I have another file 'shelves.ora' in the same directory, that is processed properly.

Deevad commented 10 years ago

Hey, Watching the output ( thanks for the detailed one you pasted ) I can read "convert -quiet "/home/kau/xra/2013/Jicmu/zas/valuedrills.ora" -gravity NorthEast /tmp/artk-tmp-wtmk.png " ; so it seams the script pass the *.ora directly to imagemagick ; and imagemagick is not compatible openraster ( yet ). I think the internal script should convert&flatten to PNG with calligraconverter first , then work with imagemagick from the temp PNG to the output format...

vanyossi commented 10 years ago

Hi, @0ion9

Im able to reproduce the bug on a fresh system start. This is the line that causes the error

kbuildsycoca4 running...

When kbuildsycoca4 is called calligraconvert stays idle and never finishes. pressing convert again makes a second calligraconvert call which in turns pushes the first one and makes the notify appear twice. once for the first time (which failed) and one for the second time (which succeds). The first time it appears you pressed convert, and the second one "preview" because of this line "Generating preview".

I'm still unsure on how to fix it. As it only happens the first time you call any kde service it seems. I'm not sure how calling kbuildsycoca4 interrupts calligraconvert during a piped run. This is one of the reasons I put the line "calligraconvert always returns errors" All warnings and messages are sent trough stderror which makes difficult to filter the real errors.

When you say calligra independently do you mean, without krita?

And thanks for the spelling error, I always double check strings, but this one got away, thanks! :D

0ion9 commented 10 years ago

When I say independently, I mean not as part of the script -- invoking it on the commandline manually. I do have Krita (2.7.5) installed and it appears to work normally. Since both Krita and calligraconvert seem to work normally for KRA files, that 'Rendering Kriters: Calligraconvert always return errors!' message seems like it must be wrong.

I tried invoking kbuildsycoca4 before I run artscript, and this changes the behaviour, but not in a good way:

Tk drag and drop enabled Starting File extractions... Starting Convert... Extract valuedrills.ora Rendering Kriters: Calligraconvert always return errors! Compiled for arch: ::Vc::SSE2Impl Features supported: "SSE2" --- yes "SSE3" --- no "SSSE3" --- no "SSE4.1" --- no "SSE4.2" --- no "SSE4a" --- no "AVX " --- no Connecting to deprecated signal QDBusConnectionInterface::serviceOwnerChanged(QString,QString,QString) QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave. QDBusConnection: session D-Bus connection created before QCoreApplication. Application may misbehave. kbuildsycoca4 running... QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver

And it just sits there, never completing. If I hit the 'convert' button again, it continues as before (attempting to feed the ORA directly into imagemagick, etc..)

Finally, I must point out that this does not occur for all ORA files, just (so far) this specific one. Here is a successful log for shelves.ora, which I did just after the above failure.

Starting File extractions... Starting Convert... Extract shelves.ora Rendering Kriters: Calligraconvert always return errors! Compiled for arch: ::Vc::SSE2Impl Features supported: "SSE2" --- yes "SSE3" --- no "SSSE3" --- no "SSE4.1" --- no "SSE4.2" --- no "SSE4a" --- no "AVX " --- no QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver QCoreApplication::postEvent: Unexpected null receiver file /tmp/shelves.png found! File extractions finished converting... shelves.ora convert -quiet "/tmp/shelves.png" -type TrueColorMatte -define png:format=png32 -define png:compression-level=9 -define png:compresion-filter=4 "/home/kau/xra/2013/Jicmu/zas/shelves_1.png" Convert done.

( I can see above that the 'kbuildsycoca4 running' line doesn't appear, I guess that adds weight to your theory.)

The file 'valuedrills.ora' appears to be an ordinary ORA file; MyPaint and Krita both load it okay, and independently calling calligraconverter ~/xra/2013/Jicmu/zas/valuedrills.ora /tmp/foo.png works fine. The only notable difference is that valuedrills.ora only has one layer (compared to the four layers found in shelves.ora).

EDIT: I've just conducted a few more tests and can confirm that calligraconvert successfully converts all of my test files.

Bringing these into artscript and attempting to convert them leads to the following results:

So this suggests the following workaround: Never process just one file.

vanyossi commented 10 years ago

wow! Thanks for all the detailed tests.

Im experiencing some problems randomly with the same Ora file I give. It seems calligraconverter doesn't always send the "eof" signal, also it crashes sometimes after making the convert.

Im working on a "fix" to avoid this in most cases. Gimp will be the default renderer for ora files instead of calligra. This will not be a definitive fix as ora fallbacks to calligra and kra uses calligraconverter. Im still making tests to see how to catch this particular error.

vanyossi commented 10 years ago

I created 3 ora files. One in gimp, the other in mypaint and lastly in krita.

The 3 files are singles layered Conversion worked as expected.

Then I edited the images making sure I was using a different editor than the one used to create the files. I made conversion and again it worked as expected. I'm not sure how should I save them ot make the bug happen, but calligraconverter has always given me problems as it tends to crash quite frequently. The problem is when the app doesn't crash but sits there, waiting.

I will do more tests tomorrow, in respect to ora (using 16 bits, grayscale, etc) and calligraconverter as a process.

Current version uses gimp as default for ora files hoping to reduce the users affected. I also opened a new bug concerning the problem with kbuildsycoca4 hanging artscript process.

Thanks for your reports :]

vanyossi commented 10 years ago

Closing

0ion9 commented 10 years ago

Note that GIMP is not very good at converting ORAs -- it fails to respect layer blending modes, so results for any file using modes other than 'Normal' are obviously incorrect.

For example http://imgur.com/SryKBgH,CKf9eBm#0 (first image is correct output, made by MyPaint, second image is incorrect output made by Artscript->GIMP)

This isn't really artscript's fault though -- the ORA plugin (and perhaps also the difference between MyPaint's compositing model and GIMP's compositing model) is the real source of this problem.

vanyossi commented 10 years ago

I have no problems using other modes. specifically overlay, multiply and screen. But the ORA file was generated using Krita. Gimp version is 2.8.10 Im not sure if that makes a difference.

I haven't done any special test after reading your report, but I'll make sometime tomorrow to do so, if I detect any problem Ill go back to calligraconverter. But I still haven't found a way to avoid the calligra hang bug. (without calling "calligraconverter --version &")

Deevad commented 10 years ago

Yes, this is a Gimp issue 0ion9 since long time. The *.ora plugin is still a basic first implantation made by Mypaint dev back in 2010. The usage of Blending modes is really limited to what Gimp handle ( and sometime it's even not working ). Also , Gimp never had a proper 'overlay' mode. It's a long term bug they have and never fixed ; the Gimp overlay is still a 'Soft Light' blending mode. ( test : Soft Light and Overlay are similar ).

If Krita or calligraconverter ( or artscript ) open/convert a *.ora saved with Gimp using 'overlay', it will bake the layer with a proper overlay blending mode ; because Krita has a right 'Overlay' blending mode.

0ion9 commented 10 years ago

Well, recent MyPaint versions (as of git fea54cb ) make this much easier -- they save a merged version of the layer-stack directly in the ORA! Artscript could use unzip -l $ORAFILE | fgrep mergedimage.png to detect this, and if it is present, simply extract it and use it.. a lot faster and better than GIMP or calligraconverter! :)

@vanyossi Yes, I am using 2.8.10 too; and Multiply is one of the modes used in the gallery I linked above that shows the error. Interesting that you have a different experience saving with Krita, I will have to see if it saves things in a different way to MyPaint. If initially calling calligraconverter --version works around the hang bug, then why not use it? It only has to be called a maximum of once per boot cycle, right?

If I were coding it in shell script I would just write :

CALLIGRA_UNHUNG_FLAG=/tmp/artscript-calligra-unhung
if [ ! -e "$CALLIGRA_UNHUNG_FLAG" ]; then
  calligraconverter --version &
  touch "$CALLIGRA_UNHUNG_FLAG"
fi

In the routine that handles ORA or KRA conversion (so that it is only called if strictly necessary)

@Deevad Interesting to learn; I see even in GIMP 2.9 they have not fixed this yet,

Deevad commented 10 years ago

@0ion9 , ha yes. They are not ready to fix this issue running since 10 years ( ref: https://bugzilla.gnome.org/show_bug.cgi?id=162395 , open on 2004 ). They said GEGL will fix it in 2008 and closed the bug report as 'obsolete' , lol . We are in 2014 ; next release still has it ....

Krita does also now the merged picture for both .kra and .ora since few days in master and 2.8 ( ref http://quickgit.kde.org/?p=calligra.git&a=commit&h=eb403a144c4b178a56905b5dca151db5e57033dc ) I hope it will make .ora and .kra compatible with most image viewer, and speed-up conversion. Maybe the --version flag is now mandatory to know if calligraconverter should just take the merged picture, or do the compositing of the stack himself to create a merged picture. I don't have a lot of infos about it.

vanyossi commented 10 years ago

Thanks for the discussion. I finally made some testings. Odly enough multiply works ok from mypaint to gimp., but overlay and hue modes are broken in the convert.

Calligraconvert is not perfect as it seems to break exclusion blending mode. I'm not sure if it's a bug from my calligra version (2.8) or a bug in general.

@0ion9 Thanks for the fix. I'll make the changes ASAP as this cannot be unsolved. @Deevad Sad to hear it's been 10 years of buggy behavior. I'm getting used to Krita development speed that I forget most projects move at a completely different cycle.

vanyossi commented 10 years ago

Thanks @0ion9 and @Deevad . I finally get to fix this bug. It turns out calligraconvert calls kde daemon if it is not running. but because it is a daemon it doesn't send an EOF or TERM signal to artscript which made it hang forever.

Artscript master branch is back to using calligraconvert as the main software to handle ORA files, and gimp as a fallback. I will port this bug fix to v2.1 as blending modes are a key function.

Notice: on my system, calligraconvert crashes on 4% of times when converting any file.