search

Fossy-Cats / Git-Buch_EN

English translation of "Das Git-Buch" (The Git Book)
https://git.io/gitbook
Other
1 stars 0 forks source link

Text-Alignment Problems in SVG Diagrams #12

Closed tajmone closed 3 years ago

tajmone commented 3 years ago

EDIT (TLTR) — The findings of this long thread have been resumed in a dedicated Wiki page, with practical examples:

@SicroAtGit,

I've been working on the DIA diagrams (translation, fixes, etc.) but I realize it's going to take ages. The DIA sources are messy and I often need to rebuild the whole diagram from scratch, and the whole process is going to take quite long.

But since I've decoupled the images from the HTML document (I no longer embed them via Data URIs) there's no reason preventing us from starting revising the text in PRs — as mentioned before, the best approach is to create a PR per chapter and keep revising and pushing commits, until it's ready to squash and merge into alpha-dev.

I've already translated all the figure IDs for the SVG diagrams (which I needed for preview links in the DIA TODOs list), so there are no risks that working on the diagram will create conflicts with the revision PR (at worst, we rebase).

So, in the coming week I'll start to polish a chapter and then create a PR, so maybe from the next weekend we can start revising it together — the main problem is when I wasn't sure about the shade of meaning of the original text, which might have been lost with the automatic translator, so it's better if you can check those sentences and fix them.

Also, we'll need to keep an eye open for consistency in terminology, etc.

SicroAtGit commented 3 years ago

I have installed the DIA editor and will have a look at the problems and your mentioned hacks when I have enough time free.

Yes, no problem, on the weekend we can start with pleasure with the revision.

tajmone commented 3 years ago

I have installed the DIA editor and will have a look at the problems and your mentioned hacks when I have enough time free.

Could you then please check on Linux the line-wrapping problem mentioned in the hack lists:

https://github.com/tajmone/Git-Buch_EN/blob/alpha-dev/assets_src/img/dia/README.md#leading-non-breaking-spaces-to-preserve-alignments

I start to think that the problem might be due to Windows's CRLF EOL sequence, where the extra CR might be the cause of the extra invisible character affecting text alignment. If you could just test a new Dia project using a Flow Chart box element with some centered monospaced text that wraps with a line break. Maybe under Linux the problem doesn't show up, due to the LF native EOL.

If this was the case, then it would be better to paste the LF character to wrap text, rather than pasting a leading non-breaking space at the beginning of the line, and we might also submit a Windows bug fix to the Dia project (the program could auto-fix CRLF EOLs to LF under Win OS).

SicroAtGit commented 3 years ago

Diagram

The lines 1 to 3 are shifted slightly to the left. Is that what you mean?

The screenshot is from Firefox 84.0.2. In GIMP 2.10.22, the image viewer Viewnior 1.7 and Chromium 87.0.4280.141 it is also displayed like this. All programs were run under Linux (x64).

I have tried all line break formats (CRLF, CR and LF). None of them fix the problem, except removing the line break between all </tspan> and <tspan>. After that, it looks like this:

Diagram_fixed

tajmone commented 3 years ago

I have tried all line break formats (CRLF, CRand LF). None of them fix the problem

Then the current hack of inserting a leading non-breaking space before any line that wraps is the only solution.

I wonder why this happens, but it's not the first time I had to resort to this hack. Having to use non-visible characters is not an ideal solution, but I couldn't come up with any other workaround to this. Unfortunately, with monospaced fonts the difference in alignment stands out quite a lot.

SicroAtGit commented 3 years ago

Then the current hack of inserting a leading non-breaking space before any line that wraps is the only solution.

As I wrote above, the line break between </tspan> and <tspan> is the problem. The SVG file output from DIA editor is incorrect.

Wrong:

<tspan></tspan>
<tspan></tspan>

Correct:

<tspan></tspan><tspan></tspan>

I wonder why this happens, but it's not the first time I had to resort to this hack.

The line break after <\tspan> is interpreted as a space: https://stackoverflow.com/questions/41364908/issue-with-svg-tspans-not-aligning-with-text-anchor-end

tajmone commented 3 years ago

If this is the case then, passing the created SVG files to the SVGO optimizer should solve the issue (at least in theory).

I need to test this locally, because in all my AsciiDoc works I've always used SVGO to optimize the Dia generated images, but as far as I can remember the problem was still there. Still, it's worth a try (I only need to add a line of code to the SBV build script, to optimize the results directly in their source folder, before deploying them to the images folder).

SicroAtGit commented 3 years ago

If this is the case then, passing the created SVG files to the SVGO optimizer should solve the issue (at least in theory).

I just tried SVGO (https://github.com/svg/svgo) and it does not fix the error in the SVG file. Extract of the output file:

</tspan> <tspan>

Notice the problematic space character between </tspan> and <tspan>.

We apparently need to write a script ourselves that removes the space characters and line breaks between </tspan> and <tspan>.

In addition, we should write a bug report at the DIA editor repository so that we don't need the fix script anymore in the near future.

tajmone commented 3 years ago

We apparently need to write a script ourselves that removes the space characters and line breaks between </tspan> and <tspan>.

Yes, it's better than having to resort to those horrible hacks with non-breakng spaces, especially since this problem only affects SVG images, so these hacks would make the raster images look bad.

Tomorrow I'll do some testing to see whether we should use a script or maybe even create a custom PureBasic tool for this. The advantage of a PB tool would be that we could also integrate other features like batch color values fixing (which in this project we'll be needing due to the low accessibility color palette).

The upstream repository did have a shell script that could replace color values, and it would even unpack and repack the Dia project file before editing. A PB tool would be more handy when it comes to handling projects which are in compressed format — but I've noticed that it's also possible to configure Dia to not pack the project files, which probably would be a better solution for a version controlled project like this.

In addition, we should write a bug report at the DIA editor repository so that we don't need the fix script anymore in the near future.

Yes, we should. The source repository is hosted at SourceForge, so I'm not sure how reporting works there, and if I need to create a user account in order to do so. Maybe there's also a Dia community forum. I'll update you here if and when I manage to file a bug report for this.

SicroAtGit commented 3 years ago

The source repository is hosted at SourceForge

It looks like the last one of the above repositories is the official repository. The page is also mentioned as the tool's website in the Wikipedia article (https://en.wikipedia.org/wiki/Dia_(software)).

tajmone commented 3 years ago

I've began investigating the problem and carry out some tests.

Cairo SVG Export Format Unaffected by Alignment Issues

The first thing I'm noticing is that SVG images generated by exporting to the Cairo SVG format don't suffer from this alignment problem at all.

Added Bonus: Cairo SVG Does Not Depend on Fonts!

Mainly, this is due to the fact that the Cairo SVG format actually converts all text to vector shapes, which is also very good since it frees SVG images from fonts dependencies — currently (with the default SVG export format) to ensure correct rendering across all machines and OSs we have to stick to safe fonts like Times New Roman, Arial, Courier New, etc., which are present on all systems (including smart phones). This problem has affected various AsciiDoc projects I've been working on, limiting the font choices in the name of safe rendering, at the cost of sacrificing elegance.

I've tested creating SVG images (via Dia) containing text in odd fonts, and when viewed on machines that didn't have that specific font they looked really ugly (text spilling out from the bounding shape, alignments all wrong).

Surely, diagrams where text is converted to vector shapes are going to be bigger in size, but this isn't really going to be a problem if we optimize the generated SVGs using SVGO with all the standard optimizing options plus Zip compression (i.e. create .svgz images). Also, these diagrams don't contain much text really.

What do you think about adopting the Cairo SVG format instead?

We'd not only get rid of the alignment problem (and all the horrible hacks to fix it via special space entities in the source text), but also (and most important) we'll be free to use any font of our liking and still be sure that the diagrams will look as intended on all devices and OSs.

NOTE — I still need to do some extensive tests though, because I've noticed that images generated via Cairo SVG are not 100% faithful to the original design — e.g. 0-width borders are being rendered as thin lines, which is a problem in our case since we need invisible background boxes to enforce padding as well as force grouped images to have the a constant width (i.e. by making their BG box always as wide as the larger diagram's BG box).

tajmone commented 3 years ago

CLI Option to Export to Cairo SVG

Exporting to the Cairo SVG format via command line can be done via the -t cairo-svg option (which, for some reason, is not documented neither in the Dia Manual nor the CLI help text).

In my local tests, I've been using the following Bash script to convert a Dia project file to both Cairo SVG and the default SVG format, in order to quickly compare results on batch test images:

#!/bin/bash
for f in *.dia ; do
    fname="${f%.dia}"
    # ================
    # Cairo SVG format
    # ================
    dia -n -t cairo-svg $f
    mv "${fname}.svg" "${fname}_cairo.svg"
    # ==================
    # Default SVG format
    # ==================
    dia -n -t svg $f
done

The script will add the _cairo suffix to the generated SVGs in Cairo-SVG format, in order to preserve both formats for quick comparison.

tajmone commented 3 years ago

I've been attempting to preserve the invisible background box in Cairo-SVG, but without success...

So it's going to be tricky to conciliate these limitations on both Dia's side as well as Cairo's export end.

I've tried using a custom embedded SVG image (a transparent borderless box), but Dia didn't recognize it as a valid SVG, possibly due to unsupported SVG elements.

I've started digging into the documentation, to see if I could create a custom ad hoc Dia shape (SVG module) to insert a borderless rectangle:

But the core problem is that Dia supports only a very limited subset of SVG elements, properties and attributes — and it doesn't seem to support stroke: none.

I've also tried to manually edit the Dia source projects to insert custom attributes (something a script could do if Dia projects are saved uncompressed, as pure XML), but at export time Dia simply replaces unknown attributes with default values, so the generated image is the same.

But it seems that custom shapes do support the none value as a valid line color or width — but I'll need to find out if Dia will interpret it as a default value at build time, or whether Cairo supports it or applies its own defaults.

The only reason why we need this invisible box is to enforce image padding across all output formats, otherwise the default template will place captions right next to the diagrams.

Another use of this box relates to those diagrams which are part of a series, showing a branch state across various operations, where each diagram is larger than the preceding one; in these cases we need a same-sized background box to ensure that all these related images stretch to the same width, to prevent inconsistent elements and text sizes from one diagram to the next.

Chances are that it might be possible to obtain the same effect by setting the image (or viewPort) size explicitly. I'm not sure if Dia allows this (the default behavior is to crop the generated image to the actual contents).

Anyhow, if there's a way other than the invisible box to enforce padding or a fixed width in the generated SVGs, the above mentioned line-width/stroke problems will no longer apply. I'll dig more into it, by reading the full Dia docs and/or testing some post-processing scripted workarounds.

But for today I've had enough of fiddling with Dia and SVG XML output.

SicroAtGit commented 3 years ago

What do you think about adopting the Cairo SVG format instead?

Good idea. The fact that the display is independent of the installed fonts is also very important to me.

* Dia doesn't support `stroke: none`

When I export the above diagram as Cairo SVG format, I see stroke:none several times in the output file:

My Dia version:

$ dia --version
Dia version 0.97.3, compiled 13:19:18 Jul  7 2020

Possibly this was added in this version. Here you can find the changelog of Dia: https://gitlab.gnome.org/GNOME/dia/blob/master/NEWS

The Windows version of Dia that can be downloaded from http://dia-installer.de is 0.97.2.

tajmone commented 3 years ago

Indeed, v0.97.3 provides a solution to this, just by this feature:

  * Transparency (aka. Alpha support)
    Almost every place where a color can be selected now also allows to
    choose the transparency value. If the the plug-in/importer/exporter does
    not support transparency, it will try to emulate (GDK renderer) or just
    ignore the transparency.

But since there's a matching Windows release I can't do much except waiting for it to be released.

It won't take that long probably, so we might just postpone the images fixes and carry on with the text, and probably by the time we finished the first draft revision it will be out.

I think that it's worth waiting for it, for it will get rid of the horrible hacks and fonts-dependencies, while still being a cross-platform solution.

tajmone commented 3 years ago

Found Dia 0.97.3 for Windows!

@SicroAtGit, I've managed to find the Windows binaries of of 0.97-pre3 (dia-setup-0.97-pre3.exe) on SourceForge:

A couple of quick tests confirmed that with this new version zero-width borders are preserved when exporting to Cairo SVG! So it looks all good, I'll just be making some more tests before actually committing diagrams changes to the repository (I also need to undo all the NBSP hacks from the latest commits).

During the weekend I'll try to fix the diagrams and the conversion toolchain, using Cairo SVG, and if the results are as expected I'll commit the changes.

Confusion Around Dia 0.97.3

There's a bit of confusion around v0.97.3, for it's not mentioned as an official release on neither of the two websites, but only on the GitLab GNOME repository, where it's marked as a stable release:

dia-0.97.3: 2014-09-05

 * Fix double free with some SVG rendering (regression from Dia 0.97.2)
 * Fixes to cope better with updated versions of Dia's dependencies:
   - don't crash at start-up with ABI breaking GLib 2-36
   - don't assert in cairo 1.12 with invalid arc parameters
   - avoid kerning problems (character overlap) for all Pango versions
   - fix image files to be loadable by libpng16
 * Backport fixes for some seldom crashes and other annoyances, see:
   http://bugzilla.gnome.org/buglist.cgi?product=dia&target_milestone=0.97.3
 * Update translations for Brazilian Portuguese, German, Hungarian,
   Polish, Romanian, Serbian and Slovenian

But the Wikipedia Dia page mentions v0.97.3 as being a preview release only, so its existence is not quite obvious to casual user.

Furthermore, it's been now six years since the Linux 0.97.3 release, so I'm not quite sure whether it's going to make it anywhere soon into the official releases of the Dia websites:

The former website is for Linux only (and, at it's own admission, quite outdated), the latter offers Win and Mac binaries and doesn't mention the 0.97.3 preview release at all.

Frankly, the fact that the Dia project is spread across two different websites and repositories (GNOME and SourceForge) doesn't seem to make it's maintenance easy. Sure, the double standard is due to the Linux edition being developed separately from Win and Mac, but still ... finding the 0.97.3 pre-release took me some time digging around, even if I've been using Dia for some years now.

So, I'll add some instructions on how to get Dia 0.97.3 for Windows in the diagrams README file. I'll also add a few notes on how to convert the Dia installer for Windows into a stand-alone version, since some end users might prefer to keep the stable version as the main Dia installed on their system.

tajmone commented 3 years ago

Switched to Dia 0.97.3 and Cairo SVG!

@SicroAtGit, it really looks like Dia 0.97.3 solved all of our problems!

After having removed the non-breaking space hacks and rebuild all the SVGs using Cairo, the generated images look just great! (and the invisible box's borders are not showing up now).

The results are so good that I just committed the changes right away — after all, we must switch to Cairo and get rid of fonts dependencies, and since all the minor problems that were holding me back are now solved with the new Dia version, I see no further reason not to make the move.

I've added to the Diagram's README some notes on why we switched to Dia 0.97.3 and using Cairo-SVG instead of the native SVG format, and also provided some instruction on how to actually get hold of Dia 0.97.3:

https://github.com/tajmone/Git-Buch_EN/blob/alpha-dev/assets_src/img/dia/README.md

I didn't really say much on the Linux version, beside providing a link to the Dia's Git repository at GNOME, and focused instead on how to fetch the Windows pre-release binaries.

Since the official websites don't mention Dia 0.97.3 at all, not even for Linux, I'm not sure if we might need to add some instructions for Linux too.

Now I only need to carry on with the Diagrams translations, adaptations and fixes which I had interrupted due to the problems that now are solved.

I can't really thank you enough for your support on this because, quite frankly, I don't think I would have ever come up with a solution if it wasn't for your discoveries (I'd be probably still stuck with Dia 0.97.2, waiting for ever). Since I use Dia for diagrams also in other documentation projects, these improvements are going to affect those projects too.

SicroAtGit commented 3 years ago

Yes, the two repositories of Dia cause a lot of confusion. I think the GitLab repository is the new repository.

Here you can read how to build a Windows version of Dia: https://gitlab.gnome.org/GNOME/dia/-/blob/master/BUILDING.md#windows

https://sourceforge.net/projects/dia-installer/files/dia-win32-installer/0.97-pre3/

Are you sure this is version 0.97.3? When I read 0.97-pre3, I interpret it as 3rd pre-release before the final version 0.97, which is something different than a version 0.97.3. Also, if you look at this version list you can see that version 0.97.2 has a newer modification date than version 0.97-pre3. So I doubt that the version 0.97-pre3 is the version 0.97.3.

Here someone has built a portable Windows version of 0.97.3: https://gitlab.gnome.org/GNOME/dia/-/issues/39

tajmone commented 3 years ago

Are you sure this is version 0.97.3? When I read 0.97-pre3, I interpret it as 3rd pre-release before the final version 0.97, which is something different than a version 0.97.3.

Hard to tell, especially with the double-repository standard which uses different dates; but at least it does solve our problem. Even if might not include all the new features of stable 0.9.73, projects created with it should work fine with the stable release.

Also, seeing the slow release times of Dia (6 years have passed already!) I'm not sure we can hope on an official Win32 0.97.3 release any time soon...

Here someone has built a portable Windows version of 0.97.3:

I'll have a look at it. Unfortunately it's a ".paf" portable version, which is not as portable as simply unpacking the installer contents via 7-Zip, but it's definitely worth trying it (it's been a while since I've handled ".paf" portables, so I don't remember exactly how to tweak their contents. I'll let you know.

tajmone commented 3 years ago

Ok, I've downloaded the portable version. But I'm afraid it won't work as expected because they've renamed the binary to diaw.exe, which won't match the build script invocation — I could create a symlink to redirect to it, but I'm not quite convinced that a portable app will work as expected via the command line (especially from Bash) since they adopt a virtualization system to conform to PortableApps, which might be isolating parts of the application (not just the registry).

I'll try to understand if it really contains new features compared to the version I've found, and if so I'll try to see if I can patch up a version that works for the command line (or maybe just try and build it from sources, the normal way).

You're probably right about 0.97-pre3 being an older version, I checked the dates on SourceForge and it's dated 2009, whereas 0.97.2 is dated 2012. So, it turns out that older versions are better at handling Cairo SVG features? Chances are that the 0-width line problem then is still there with 0.97.3.

Ahhh ... what a mess, I'll need to find a way to build Dia 0.97.3 under Windows, hope it works as expected and if yes then store our own package somewhere...

tajmone commented 3 years ago

Dia Incompatiblity Solved!

@SicroAtGit, after running in circles for hours I think I've finally solved the Dia/CAIRO SVG incompatibility problems.

First of all, the Portable edition of Dia 0.97.3 isn't usable via the command line (as I suspected). The standard Dia for Windows edition has two binaries, dia.exe and diaw.exe, in the Portable edition the former is absent, so it won't work via command line (also, paths are virtualized, which means that the app doesn't even find dependencies when run without the PAF startup binary).

In any case, I've tried to manually export to Cairo SVG a sample diagram using Dia 0.97.3 Portable, and the 0-width borders are still there. Also, I've checked the line options and it doesn't contain any new features like invisible line color, don't draw lines, etc. So even if I were to build it from the GitLab sources it wouldn't solve our problem.

Luckily I mistakenly downloaded the 0.97-pre3 release, which worked fine, although it was actually a pre-release for version 0.97. So I started testing with every stable release above 0.97, and it turned out that any Dia version above 0.97 will draw the zero-width lines/borders in Cairo SVG.

Maybe the problem has to do with the Cairo library version used by Dia releases from 0.97.1 upward, which don't accept zero-width lines.

So, in the end I'll be using Dia 0.97 for this project. I'll also tweak the build.sh script so under Windows it will check for the presence of a Dia/ standalone folder in the diagrams folder, and if found use its dia/bin/dia.exe binary — this will prevent conflicts in case end users have installed a later version of Dia on their machines.

Linux Tests Needed

I'm not sure how Dia setup works on Linux, and if we should add similar checks for Linux in build.sh.

As a last thing, can you confirm me that under Linux the zero-width lines problem for Cairo SVG is the same?

Could you please try converting the diagrams under Linux and check if the following diagram will have invisible borders in the final SVG:

I'd like to know if under Linux Dia 0.97 doesn't draw the border, and if version from 0.97.1 upward do. Since you've installed v0.97.3, you could start testing with that one — it should draw the border.

Maybe it's a Windows only problem. I just need to provide the correct instructions in the documentation, regarding which Dia version to use under Win and Linux. We need to be sure that the generated SVGs are the same, regardless of the platform under which they're built, otherwise we'll be facing unneeded changed files showing up in Git's working area.

Thanks.

SicroAtGit commented 3 years ago

Very good work! Also, the Dia downloader is a good idea.

I converted the diagram assets_src/img/dia/integration-manager-workflow.dia to Cairo-SVG with my Dia version 0.97.3 under Linux and there the outer frame is also visible.

Installing old programs under Linux is problematic. Either the old programs are not in the official repositories anymore or they need older dependencies of the already installed dependencies. Mostly not several versions of a dependency can be installed at the same time and if I install the older version, then already installed programs, which need the dependency in the new version, would not work anymore. Old program versions are removed after a while so that the repository space consumption does not grow too much. Currently, there are only 0.97.3-x versions in the archive repository.

I tried to build an old version of Dia from the git repository by checking out the tag DIA_0_97 locally and trying to follow the build instructions. However, I had no success in doing so. Either the required versions of the dependencies for building were no longer in the official repositories or something else.

It is certainly possible to get the Windows version of Dia to work with WINE on Linux, but then there is another problem as described below.

We need to be sure that the generated SVGs are the same, regardless of the platform under which they're built, otherwise we'll be facing unneeded changed files showing up in Git's working area.

Since we switched to Cairo-SVG, the texts are saved rendered in the defined font. As I have seen, the font monospace is defined, but what the font is, the OS user can change or it differs between different OS versions. Anyway, the font monospace is not identical between Windows and Linux. We would have to pick a free monospace font, set all diagrams to that font, add the font file to the repository and everyone working on the project and editing diagrams would have to install that font in their operating system.

The easiest solution would be if only you do the conversion of the Dia files to Cairo-SVG files.

tajmone commented 3 years ago

Installing old programs under Linux is problematic. [...]

This is why I prefer to do all the builds work under Windows. Besides being able to run a Linux kernel and distro in the background via WSL2, which I can interface to native Win apps, I also can run both 64- and 32-bit applications, without dodgy bitness restrictions on x86 architectures. Although the personal freedom aspect offered by Linux is cool for personal use, when it comes to collaborative work it can often lead to complications due to distro differences and incompatibilities.

Since we switched to Cairo-SVG, the texts are saved rendered in the defined font. As I have seen, the font monospace is defined, but what the font is, the OS user can change or it differs between different OS versions. Anyway,

Yes, I'm aware of that, but I'm postponing editing the current font with a real one to the last moment, because it will most likely break up all proportions and force me to re-edit all the diagrams, one by one (boxes too large, or too small, etc.). First I'd like to focus on the translation, adding a box for padding and consistent widths among similar images of a series, and then fix the color palette, and then as a last step enforce an open source font for both the monospace an san-serif texts — bear in mind that for many diagrams we don't yet know if we'll translate the German text referring to commit messages.

I'll also include the TTF font in the repository assets, with license and all, to avoid problems in the future (broken links, etc.).

The easiest solution would be if only you do the conversion of the Dia files to Cairo-SVG files.

For the time being that would be fine, but we should also think of the project's future. Either whoever takes on the project will simply rebuild all the diagrams with his/her own system (OS, Dia version, output format, etc.) or they can always use a Windows Server machine via Travis CI to auto-build the diagrams.

In any case, right now the important thing is to get them done right, using the best method at our disposal.

SicroAtGit commented 3 years ago

WSL2 in Windows 10 is certainly very useful if you mainly use Windows, but also want to be able to use Linux tools. But everyone has chosen their suitable operating system. I don't think we can expect everyone to have Windows 10 installed on their PC today, who wants help on this project or pick it up and continue the work in the future. There are certainly also many Linux users on GitHub, because development tools are mostly easier to install under Linux (many distributions have build tools already installed out-of-the-box).

I think the problem is not so much the operating system, but the tools that are chosen for the collaborative work. In this project we use a very old version of Dia, because the experimental plugin Cairo does not give the desired Cairo-SVG output file with the current Dia version. In addition, the very old Dia version is only available for Windows in a precompiled form, but not for Linux or MacOS. If it were the other way around, you would have the problem on Windows and the struggle to compile the source code.

If no other option can be found, I could fall back to my Windows 7 that I have in Virtual Box to do little Windows things. But it would be nice if it didn't become a Windows-only development project.

Yes, I'm aware of that, but I'm postponing editing the current font with a real one to the last moment, because it will most likely break up all proportions and force me to re-edit all the diagrams

That is not what I wanted to achieve. I mean, it would be good to know which font is currently used when using monospace in your Dia when the texts are rendered to Cairo-SVG. Without this information it is difficult to create identical Cairo-SVG files in other operating systems.

For the time being that would be fine, but we should also think of the project's future. In any case, right now the important thing is to get them done right, using the best method at our disposal.

Sure. It should be understood only as a temporary solution. I have not stopped looking for cross-platform solutions. I have been looking at the Inkscape tool as an alternative to Dia. So we could do away with the .dia files and switch to .svg files only. However, I think the Dia tool is more suitable for this project because it is specialized for diagrams.

So maybe the tool CairoSVG is a better alternative. It can also convert SVG files to Cairo SVG files. We would then no longer need to use the experimental Cairo plugin in Dia to create the Cairo SVG files. This means that we then take the normal SVG output format again in Dia and the CairoSVG tool performs the conversion from SVG to CairoSVG afterwards.

When using the CairoSVG tool instead of the experimental Dia plugin, we would have several advantages:

How the conversion is performed:

$ cairosvg normal_svg_file.svg -o cairo_svg_file.svg
tajmone commented 3 years ago

WSL2 in Windows 10 is certainly very useful if you mainly use Windows, but also want to be able to use Linux tools. But everyone has chosen their suitable operating system. I don't think we can expect everyone to have Windows 10 installed on their PC today, who wants help on this project or pick it up and continue the work in the future. There are certainly also many Linux users on GitHub, because development tools are mostly easier to install under Linux (many distributions have build tools already installed out-of-the-box).

Surely, I agree, it's just that in general I've started to move away from macOS (entirely) and Linux too, mostly due them dropping 32-bit support, and because on most CI services only 64-bit versions of the OSs are available — and currently on Windows can run both 32- and 64-bit apps.

Keep in mind that I work with many legacy tools in my projects, especially relating to Interactive Fiction authoring languages and interpreters, which are all available only as 32-bit. So it's not that I have much choice there, since many of these tools are simply unusable on Linux or macOS, except by installing Wine, or some special tools for 32-bit support (often commercial, for macOS). It seems to me that now roles are being reversed, and Windows is actually becoming more flexible under many aspects.

That is not what I wanted to achieve. I mean, it would be good to know which font is currently used when using monospace in your Dia when the texts are rendered to Cairo-SVG. Without this information it is difficult to create identical Cairo-SVG files in other operating systems.

It's hard to tell. I've tried looking into it, but the fact is that my understanding is that each Windows version might use a different fallback font for generic definitions like "monospace", "sans-serif" and "serif", because MS takes pride in introducing new fonts with each major Win version. Also, I believe there's a user setting somewhere which would allow to change the fallback fonts either at the User or Machine level, but don't know where this setting might be.

What I've done instead in the past days was to pick some FOSS fonts that we'll use for monospace and san-serif text in the diagrams. And, since some of these fonts were already present on my machine, I've installed them in my User fonts folder, to ensure that I'll be using the FOSS version, instead of the native one — they might differ slighlty, although they look identical, because they are from different authors, and have different licenses.

Also, I've picked the common TTF variants (regular, bold, italic) which we'll be actually using, since these fonts have many text style variations, and some of them were also available in the new variable font format (which I doubt is supported by Dia).

I've also prepared the ground for some in-project testing — i.e. picking some diagrams and using the picked fonts instead of "monospace" — but before that I wanted to finish creating the new color palette/scheme for the diagrams, since it makes sense to fix all the colors whenever I edit a diagram. Right now, the diagrams palette is a bit messy, many variations of similar (but not identical) colors are being used, some of them too dark in terms of accessibility; and I'd like to use a consistent palette throughout all diagrams, with slightly more readable color contrasts.

So maybe the tool CairoSVG is a better alternative.

I wasn't aware of this tool, I only knew of the Cairo 2D library, which I think is what Dia uses internally.

The problem with other SVG solution (CairoSBG, Inkscape, etc.) is that they don't provide specific shapes for the task, like Dia. The shapes library of Dia is really handy to work with, for it allows to work on single items (e.g. the text is part of a diagram box, or a flow chart element). Editing vector projects with tools like InkScape can be quite an overkill; it has too many features which we wouldn't need, and it involves a lot of work keeping text together with its box, centering lines that join boxes (which Dia does automatically, in a smart way via drag-n-drop).

If I were to drop Dia it would be in favor of some ASCII to SVG diagram generator. But having tried many of them out, I doubt we could find one that could do all that Dia allows us to (especially for Git graphs). Also, ASCII "drawn" diagrams become soon very hard to deal with, when you have big diagrams (and this books has some rather big ones, like the Git Flow diagram).

I personally like Dia very much, even if it has some limitations, and the project is currently a bit in a mess state; but I trust that it will keep growing in time, and the using it is a way of promoting it. And what I like most about Dia is that it's extensible — you can quite easily create new shapes libraries, and plug-in (in Python).

The resolve the Dia incompatiblity and cross-platform issues, we could focus on compiling just the command line tool — most of Dia's package files are for the actual GUI editor, so focusing on the CLI tool should reduce also the dependencies issues, and produce a standalone binary that should be usable regardless of any latest version installed system wide. Probably it's enough to just get hold of a full pre-compiled package and strip it down, and then we could park a Zip file somewhere.

SicroAtGit commented 3 years ago

Windows can run both 32- and 64-bit apps. Keep in mind that I work with many legacy tools in my projects, especially relating to Interactive Fiction authoring languages and interpreters, which are all available only as 32-bit. So it's not that I have much choice there

As far as I know, this is also possible under Linux if the 32-bit libraries are installed in the 64-bit OS. But yes, 32-bit support is becoming more and more difficult and will surely be dropped from all OS in the near future.

I don't want to make you switch back to Linux. You use the operating system that meets your needs and all is well.

I've tried looking into it, but the fact is that my understanding is that each Windows version might use a different fallback font for generic definitions like "monospace", "sans-serif" and "serif", because MS takes pride in introducing new fonts with each major Win version.

I also think so. But it's more about the monospace font used in your operating system to achieve the identical Cairo SVG output files in other operating systems. I will try the Courier New font tomorrow in my virtualized Windows 7 with Dia 0.97 and check if the SVG result files are the same as in the repository. Probably this is the new monospace font in Windows 10. If the font can change with Windows 10 updates, we should switch to a fixed font soon to avoid unnecessary changes in the git working area.

The problem with other SVG solution (CairoSBG, Inkscape, etc.) is that they don't provide specific shapes for the task, like Dia.

The CairoSVG tool is not a replacement for Dia, but would only handle the conversion from SVG to Cairo-SVG in our case, as written above. Therefore, I don't see any problems with specific shapes with this solution.

The resolve the Dia incompatiblity and cross-platform issues, we could focus on compiling just the command line tool

In the build instructions from the repository I didn't see a build flag to compile only the command line version. Besides, under Linux there is only one executable that is both the GUI and command-line version. I will have a look at it again tomorrow ...

Probably it's enough to just get hold of a full pre-compiled package and strip it down, and then we could park a Zip file somewhere.

There is no pre-compiled Linux package of version 0.97. The package dia_0.97-1_i386.deb on https://sourceforge.net/projects/dia-installer/files/dia/0%2C97/ is corrupt. I installed it on a virtual Xubuntu 32-bit and Dia would not start.

tajmone commented 3 years ago

In the build instructions from the repository I didn't see a build flag to compile only the command line version.

We should request this feature, it would be very useful to have a trimmed down CLI version for Continuous Integration services (Travis CI, GitHub actions, etc.), possibly as a standalone binary or package that can be just built or unpacked into any folder and added to the sys path.

Maybe now it's possible to extract selected files from the official Dia app and create a Zip archive (it will have to include all the shapes libraries, and some plug-ins and export filters though). Dia for Windows contains tons of DLLs, so working out which are needed and which not could be a long trial-and-error process.

Besides, under Linux there is only one executable that is both the GUI and command-line version.

For automation purposes, this should be fine, even if trying to launch it as a GUI would fail, since we'd be only invoking it via command line switches.

But all this seems a lot of work.

The CairoSVG tool is not a replacement for Dia, but would only handle the conversion from SVG to Cairo-SVG in our case, as written above. Therefore, I don't see any problems with specific shapes with this solution.

But this wouldn't solve our original problem, i.e. creating good SVGs out of Dia in the first place. The CairoSVG tool won't be able to fix the problems in the source SVGs. Also, SVG is a set of standards, not just one, and Dia leverages the most common one, which is granted to work in all browsers; whereas tools like InkScape or CairoSVG are probably targeting also the newer features from the latest SVG format, which are not well supported by most browsers.

This is another reason why I would have avoided using tools like InkScape for the diagrams project, because future contributors and forkers might inadvertently use some advanced features that might force InkScape to use the new SVG formats. Right now, there's a lot of confusion when it comes to using SVG on the web or HTML based tools, due to the different SVG formats available (the latest being still WIP) and the discrepancies on how browsers support them.

SicroAtGit commented 3 years ago

I tried Dia 0.97 today on my Windows 7 in VirtualBox and it creates an identical Cairo SVG file to the one in the repository. Courier New is also present in the virtualized Windows 7.

Then I installed Dia 0.97 in my Linux with WINE and had it create the Cairo SVG file there as well. However, the file was no longer identical there. Surely because WINE includes my installed fonts in my Linux as well and there are several monospace fonts, which causes Dia in WINE to pick another monospace font when creating the Cairo SVG file instead of Courier New. So in order to create an identical Cairo SVG file as in the repository, I had to change the font of the diagram with Dia from monospace to Courier New.

Is it ok for you, if I change the font from monospace to Courier New in all diagrams tomorrow? For the Cairo SVG diagram files already in the repository, this shouldn't trigger any changes, because Dia already uses Courier New font for monospace in your OS.

We should request this feature, it would be very useful to have a trimmed down CLI version for Continuous Integration services (Travis CI, GitHub actions, etc.), possibly as a standalone binary or package that can be just built or unpacked into any folder and added to the sys path.

I agree. On Windows there are dia.exe, diaw.exe and dia-win-remote.exe. In my virtualized Windows 7 diaw.exe opens the GUI of Dia, dia.exe also opens the GUI, but creates an empty console window behind it, which closes only when Dia is closed. I have not tried dia-win-remote.exe.

I will create two issues in the official Dia repository tomorrow, addressing the text alignment problem with multi-line texts and the wish for a build flag for creating a CLI-only version.

But this wouldn't solve our original problem, i.e. creating good SVGs out of Dia in the first place. The CairoSVG tool won't be able to fix the problems in the source SVGs.

Yes, it only fixes the errors (for example, the text alignment problem and text rendering) in the generated Cairo SVG output file. So we would always depend on the CairoSVG tool. It also requires several dependencies, which is why it is not as portable as Dia and will probably not work without an installation.

Also, SVG is a set of standards, not just one, and Dia leverages the most common one, which is granted to work in all browsers; whereas tools like InkScape or CairoSVG are probably targeting also the newer features from the latest SVG format, which are not well supported by most browsers.

Yes, it may be that CairoSVG uses features that are not currently supported by most browsers. Also, installing this tool on Windows is probably not that easy.

I suggest, we stick with the WINE Dia variant on Linux for now, which requires me to change the font on all diagrams from monospace to Courier New, but that's ok for me.

tajmone commented 3 years ago

Is it ok for you, if I change the font from monospace to Courier New in all diagrams tomorrow?

Please don't. I've invested many hours on planning how to fix all the diagrams, and had chosen the "Consolas" font for monospace font ("Courier" is really ugly), but I'm aware that switching fonts will break all alignments, because Consolas is slimmer than Courier, so every diagram will have be rebuilt (a bit of a disaster, really, but I knew this was coming from before I started working on them, because by experience I know that tweaking diagrams is always a mess).

It's just that right now I'm still trying to fine-tune the color palette, and work out how to replace the current colors (a really messy situation, it should be an eigth max ten colors palette, but there are more than 20 colors that I've counted so far, mostly are probably accidental variations).

I'll need to create all the required shades and tints of each hue, and then test them for color-blind accessibility — this being the time consuming part, for each time a darker shade rates poorly I have to rebuilt all of the base hues trying to work out a new color harmony for the scheme.

Once I have the palette, I'll just be tweaking the various diagrams one after the other (well, some are groups really). Hopefully, this time each editing will be the final one, for I'll handle everything, from rebuilding the diagram with the right shapes, to changing fonts and colors, adding a layer with the BG box, etc.

I just realized that keeping editing them over and over, once for the font, another to translate, and then again for colors, it's just too time consuming and harder to track.

SicroAtGit commented 3 years ago

Sure, the Consolas font is much nicer. The Courier New font is just the font currently used in the Cairo SVG files. Ok, I will not make the changes.

Sounds very time-consuming, indeed. Let me know if I can help you again.

SicroAtGit commented 3 years ago

I have now created the issues in the official Dia repository:

tajmone commented 3 years ago

Dedicate Wiki Page for these Dia Issues

I've created a Wiki page at the ALAN Docs repository, which resumes all the issues uncovered in this discussion, providing real examples and solutions:

https://github.com/alan-if/alan-docs/wiki/Dia-Diagrams

SicroAtGit commented 3 years ago

Very good!