Image paths (sometimes) go wrong with imagesdir variables

[lurch]

Disclaimer: I'm a bit of an asciidoctor novice (only started using it a few days ago), so apologies in advance if I'm doing something stupid... :wink:

I've got some existing documentation that I'd like to convert into asciidoc format - everything seemed to be going fine until I started using asciidoctor-mathematical.

I've got my folder-structure setup like this:

.
├── book.adoc
└── chapters
    ├── conclusion
    │   ├── chapter.adoc
    │   └── images
    └── intro
        ├── chapter.adoc
        └── images

so that each chapter has it's own self-contained directory (into which I can place images, CSV files, etc. etc.). Some of the chapters may be re-used in other documentation later, which is why I want to keep them self-contained.

I've (eventually!) managed to narrow down the problem I'm having to a minimal test-case. Using the file-structure above, book.adoc contains:

= Math book
:doctype: book
:math:
:imagesoutdir: generated_images

include::chapters/intro/chapter.adoc[]

include::chapters/conclusion/chapter.adoc[]

chapters/intro/chapter.adoc contains:

:imagesdir: chapters/intro/images

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

and chapters/conclusion/chapter.adoc contains:

:imagesdir: chapters/conclusion/images

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

If I run asciidoctor-pdf -r asciidoctor-mathematical book.adoc then it all works okay, generated_images/stem-06cc0e32aefea03545983a9c9db6f8bc.png & generated_images/stem-eec5713a36c954c2630d087e912385e3.png get created, and I get both equations nicely displayed in book.pdf.

However if I then delete book.pdf and generated_images and change chapters/intro/chapter.adoc to:

:imagesdir: chapters/intro/images

Introduction

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

(i.e. all I've done is add the word Introduction) then when I run asciidoctor-pdf -r asciidoctor-mathematical book.adoc it now says:

asciidoctor: WARNING: image to embed not found or not readable: /home/andrew/asciidoc-test/chapters/intro/images/generated_images/stem-eec5713a36c954c2630d087e912385e3.png
asciidoctor: WARNING: image to embed not found or not readable: /home/andrew/asciidoc-test/chapters/intro/images/generated_images/stem-06cc0e32aefea03545983a9c9db6f8bc.png

and I get

Introduction
[$$a^2 + b^2 = c^2$$] | generated_images/stem-eec5713a36c954c2630d087e912385e3.png
Equation 1
[$$d^3 + e^3 = f^3$$] | generated_images/stem-06cc0e32aefea03545983a9c9db6f8bc.png
Equation 2

displayed in book.pdf :sob:

Looking in /home/andrew/asciidoc-test/generated_images the stem-06cc0e32aefea03545983a9c9db6f8bc.png and stem-eec5713a36c954c2630d087e912385e3.png files have been created directly, they just haven't been included into the PDF properly.

Until this is fixed, or unless there's a workaround because I've been doing something wrong, this is a bit of a show-stopper for me.

[lurch]

I've been trying to dig into this by creating small test-cases, and sometimes it works, and sometimes it doesn't, and I can't for the life of me figure out a consistent reasoning for which examples work and which examples fail :sob:

I've got some existing documentation that I'd like to convert into asciidoc format...

I've now completed that phase, so this is starting to become a major sticking point :confused: I can't "flatten" all my chapters down into a single directory, because the image-names in one chapter are the same as image-names in another chapter.

@mojavelinux Apologies for the ping, but is this a bug specifically with asciidoctor-mathematical or a wider bug with asciidoctor itself? Unfortunately Ruby is one of the languages I'm not familiar with, so I can't really dig into the problem myself (yet).

[mojavelinux]

There is no correlation between the imagesdir and the include directive. The include directive coalesce documents together into a single document, which is then parsed. (The only exception is that the include directive is aware of its own context).

This is explained in this issue: https://github.com/asciidoctor/asciidoctor/issues/650

There's an extension linked in that issue you may find interesting.

[lurch]

Ahhh, thanks for the pointers @mojavelinux :+1:

So based on your feedback I've now verified that this bug is somehow related to the imagesdir variable, and actually has nothing to do with the included files (and I've edited this issue's title).

It's still very strange behaviour though, and it would be great to see it fixed (due to my use of different per-chapter image folders as described earlier).

I've tried to create some minimal working / non-working examples, and these are what I came up with...

These work: (both equations get displayed in the output pdf)

= Math tests
:stem: latexmath

:imagesdir: images1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

:imagesdir: images2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

= Math tests
:stem: latexmath

Text 1

:imagesdir: images1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

Text 2

:imagesdir: images2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

These don't work: (asciidoctor-pdf says asciidoctor: WARNING: image to embed not found or not readable: /home/andrew/asciidoc-test/images2/stem-06cc0e32aefea03545983a9c9db6f8bc.png and only the first equation gets displayed properly in the pdf)

= Math tests
:stem: latexmath

Text 1

:imagesdir: images1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

:imagesdir: images2

Text 2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

= Math tests
:stem: latexmath

Text 1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

:imagesdir: images2

Text 2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

These don't work: (asciidoctor-pdf says asciidoctor: WARNING: image to embed not found or not readable: /home/andrew/asciidoc-test/images1/stem-eec5713a36c954c2630d087e912385e3.png asciidoctor: WARNING: image to embed not found or not readable: /home/andrew/asciidoc-test/images1/stem-06cc0e32aefea03545983a9c9db6f8bc.png and neither equation gets displayed properly in the pdf)

= Math tests
:stem: latexmath

:imagesdir: images1

Text 1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

Text 2

:imagesdir: images2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

= Math tests
:stem: latexmath

:imagesdir: images1

Text 1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

:imagesdir: images2

Text 2

.Equation 2
[latexmath]
++++
d^3 + e^3 = f^3
++++

And I get the identical behaviour and error messages regardless of whether the images1 and images2 directories exist or not. If I add :imagesoutdir: generated_images to the document-header, I still get the same working / non-working behaviour as described above (but the error messages are slightly different).

Anything else I can do to help, please let me know!

[mojavelinux]

It's not a bug, per se. It's just something tricky you need to grasp in AsciiDoc / Asciidoctor. To start, not all converters use the imagesdir value in the same way, so sometimes you need different values for different backends. And then you have to consider where generated images are going and how those get read in (pdf) or linked to (html). Unfortunately, you just have to keep playing around with it until you find the combination that works.

We have addressed this many times. Since asciidoctor-mathematical is generating images, there's a possibility it's not doing it in the same way as asciidoctor-diagram, which has solved this issue. See https://github.com/asciidoctor/asciidoctor-diagram/issues/59

[lurch]

Thanks @mojavelinux

I'm still kinda confused though, that even when I use imagesoutdir, this works:

= Math tests
:stem: latexmath
:imagesoutdir: generated_images

Text 1

:imagesdir: images1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

but this fails:

= Math tests
:stem: latexmath
:imagesoutdir: generated_images

:imagesdir: images1

Text 1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

The only difference is whether Text 1 comes before or after :imagesdir: images1 :confused:

[mojavelinux]

That's because it depends on how the library is accessing imagesdir to resolve where to put the files. In the first case, imagesdir is not a document header attribute. In the second case it is. These are different things in AsciiDoc.

AsciiDoc really wasn't designed for multiple image directories. If you can get it to work, you can certainly do it. But you're stretching the behavior of the language / processor.

[mojavelinux]

I should add that it works fine to change the imagesdir when making references to images (such as HTML). But it just doesn't work well when you are also generating images. It may be something we need to add to AsciiDoc, but that doesn't change the fact that it wasn't there to start with.

[lurch]

In the first case, imagesdir is not a document header attribute. In the second case it is. These are different things in AsciiDoc.

Hmmm, does that explain why this works:

= Math tests
:stem: latexmath
:imagesoutdir: generated_images

Text 0

:imagesdir: images1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

but this doesn't?

= Math tests
:stem: latexmath
:imagesoutdir: generated_images

Text 0

:imagesdir: images1

Text 1

.Equation 1
[latexmath]
++++
a^2 + b^2 = c^2
++++

Apologies if I'm beginning to sound like a broken record! :laughing:

AsciiDoc really wasn't designed for multiple image directories. ... But it just doesn't work well when you are also generating images.

Fair enough. Instead of having my per-chapter asciidoc files referencing images as image.png (and changing {imagesdir} before including each chapter), maybe I need to instead change my per-chapter asciidoc files to reference {chaptername}/image.png (without changing {imagesdir}), and then have a Makefile rule which copies chapter/$CHAPTERNAME/images/* to images/$CHAPTERNAME/ before running asciidoctor-pdf. I'll give that a go today and let you know if it works out. And/or maybe I should look at https://github.com/asciidoctor/asciidoctor/issues/650 more closely :slightly_smiling_face:

[mojavelinux]

Hmmm, does that explain why this works:

Perhaps. I don't have time to explore in depth, but messing with the imagesoutdir and imagesdir is just asking for trouble. Your second proposal is likely a more sound approach.

I have set up projects that use images per chapter. But I don't think I've done it when images are also being generated. Usually what authors do is put all images in single directory and mirror the chapter structure under that. That way, you get a single base images folder, but you can still subdivide your images by chapter. In other words, two parallel hierarchies.

[lurch]

I managed to get my "workaround strategy" (that I outlined earlier) working fine :smiley: (I've got a Python script which scans chapters/$CHAPTER/$CHAPTER.adoc for image::{chapter}/image.png lines, adds any found image files as a images/$CHAPTER/image.png pre-requisite, which then matches a images/$CHAPTER/%: chapters/$CHAPTER/images/% wildcard rule in the Makefile.) Maybe a bit over-complicated, but it works great, and all my autogenerated latexmath equations are showing up properly in my output PDF file now.

Thanks for your help and advice @mojavelinux

(I'll leave this issue open though, because I still think the difference between the last two example-files I mentioned warrants further investigation)

[krOoze]

Isn't this dup of #43?

The problem is the _ in the :imagesoutdir: generated_images combined with another _ in the asciidoc document following the math.

[mojavelinux]

I could be wrong, but I don't think that's what's happening in this case. These are block images, so no formatting should be happening.

[krOoze]

Well, it is what I found is causing this, as shown in repro in #43. In pdf output it instead shows the warnings quoted in OP here.

[mojavelinux]

That's because you are using an inline stem (and thus an inline image). So there are two different things going on, it would seem. But there are so many scenarios being thrown around here, that there's likely no one explanation for why things don't work.

[krOoze]

Yea, this is something different. What happens here is that :imagesoutdir: and :imagesdir: get somehow convoluted together in a weird way.

It is interesting to see which image paths it generates in HTML:

Swapping :imagesoutdir: and :imagesdir: order seems to matter too (different order creates another wrong path).

Putting = Math book after :imagesoutdir: seems to fix things. Weird behavior indeed.