AsciiDoc filters generate images on disk (even when embedded)

[dagwieers]

Currently AsciiDoc filters generate images on disk, but for flat XML files and HTML files that use embedded images, this is unneeded and undesirable. It would be nice if AsciiDoc could either prevent any files to be written on disk, but instead use base64 immediately, or instead clean up the files on disk after embedding.

[elextr]

On 14 February 2012 23:43, Dag Wieërs reply@reply.github.com wrote:

Currently AsciiDoc filters generate images on disk, but for flat XML files and HTML files that use embedded images, this is unneeded and undesirable. It would be nice if AsciiDoc could either prevent any files to be written on disk, but instead use base64 immediately, or instead clean up the files on disk after embedding.

Asciidoc does not know what a filter does or when its output may be used, this is in the filter configuration. Filter output may not be used until later parts of the toolchain. So there is no way for asciidoc to know when it is safe to delete files or even what files to delete, and the filter which does know is long gone. And not all toolchains support embedded images so support for files is still needed.

---

Reply to this email directly or view it on GitHub: https://github.com/dagwieers/asciidoc-odf/issues/29

[dagwieers]

Welcome back !

Yes, I don't want to get rid of support for files on disk, but currently when you have inline ascii-diagrams, resulting in embedded diagrams. Your directory is full of unwanted images, and I think we should find a way to fix that.

[elextr]

On 15 February 2012 01:14, Dag Wieërs reply@reply.github.com wrote:

Welcome back !

Yes, I don't want to get rid of support for files on disk, but currently when you have inline ascii-diagrams, resulting in embedded diagrams. Your directory is full of unwanted images, and I think we should find a way to fix that.

Well, to delete "unwanted" images we need to know the definition of an unwanted image.

Since the filter's images go in the same directory as user supplied images we can't just delete all image files.

Somehow a2x has to find out:

1. the image files that are generated as part of processing the asciidoc file (since they can be re-created)
2. which of those files the backend embedded

If a file meets both requirements then they are no longer needed and can be deleted.

Filters can probably be made to write a list of files they generate for item 1. but how do we get backends to say which ones are embedded?

Note that you can put images in a subdirectory so they don't clutter the document directory.

Cheers Lex

---

Reply to this email directly or view it on GitHub: https://github.com/dagwieers/asciidoc-odf/issues/29#issuecomment-3961436

[dagwieers]

You are now specifically discussing a2x, while I was referring to asciidoc.

But regardless of how it is implemented today, let's think about how it would look if we would implement it with the requirement to support embedded images. I can imagine a mechanism where asciidoc instructs the filter to get base64, or the image binary data, rather than a reference to a file. So if uri-data is enabled, the filters behave differently in that respect. Of course asciidoc would also need to make a clear distinction between embedded image blocks and normal image blocks. The current way of doing that really clutters up the backend file. If only I could reclaim the hours spent ;-)

[elextr]

On 15 February 2012 19:40, Dag Wieërs reply@reply.github.com wrote:

You are now specifically discussing a2x, while I was referring to asciidoc.

But regardless of how it is implemented today, let's think about how it would look if we would implement it with the requirement to support embedded images. I can imagine a mechanism where asciidoc instructs the filter to get base64, or the image binary data, rather than a reference to a file. So if uri-data is enabled, the filters behave differently in that respect. Of course asciidoc would also need to make a clear distinction between embedded image blocks and normal image blocks. The current way of doing that really clutters up the backend file. If only I could reclaim the hours spent ;-)

Well asciidoc actually expects filters to pass the output directly to them, its the filters themselves that save stuff to files. For example the source-highlight filter doesn't leave a file hanging around because source-highlight and pygmentize both send their output to stdout.

"All" you have to do is re-write all filters to return base64 encoded text via stdout if the data-uri attribute is set, otherwise to leave it in files for the backend toolchains to use, easy :)

But since filters are external plugins they are going to do what they want, so you can only control the four built-in ones.

Hence my suggestion that you may have to use a2x to clean up the mess after the toolchain has been run, although as I said it isn't that easy.

---

Reply to this email directly or view it on GitHub: https://github.com/dagwieers/asciidoc-odf/issues/29#issuecomment-3976799

[dagwieers]

The mechanism I was talking about was more a standard interface plugins should adhere to by the AsciiDoc project. Do you think it is worthwhile to discuss on the AsciiDoc mailinglist ?

Also, the a2x currently has a problem slightly related to this, when parsing README.odt:

[dag@moria asciidoc-odf]$ a2x -v -b odt --backend-opts="--schema=relaxng/OpenDocument-v1.2-os-schema.rng" -a icons -a iconsdir=/etc/asciidoc/images/icons README.asciidoc
-snip-
asciidoc: README.asciidoc: line 248: filtering: "/etc/asciidoc/filters/ditaa/ditaa2img.py" -v -o "/tmp/tmpYmkr3D/README__1.png" --scale 4 -
Runing filter script /etc/asciidoc/filters/ditaa/ditaa2img.py
Temporary input file is /tmp/tmps3Qth9
Exec: java -jar "/etc/asciidoc/filters/ditaa/ditaa0_9.jar" "/tmp/tmps3Qth9" "/tmp/tmpYmkr3D/README__1.png" --overwrite -v --scale 4.000000 >&2

ditaa version 0.9, Copyright (C) 2004--2009  Efstathios (Stathis) Sideris

Running with options:
overwrite
verbose
scale = 4.000000
Reading file: /tmp/tmps3Qth9
Locale: en_US
Dialog.bold
Rendering to file: /tmp/tmpYmkr3D/README__1.png
Done in 1sec
asciidoc: README.asciidoc: line 248: evaluating: {counter2:image-number}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.abspath("README__1.png")}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README__1.png')[1]}
asciidoc: README.asciidoc: line 296: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 343: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 367: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 398: evaluating: {counter:admonition}
asciidoc: README.asciidoc: line 419: evaluating: {counter:admonition}

Traceback (most recent call last):
  File "/usr/bin/a2x", line 934, in <module>
    a2x.execute()
  File "/usr/bin/a2x", line 356, in execute
    self.to_backend()
  File "/usr/bin/a2x", line 608, in to_backend
    eval('to_%s(self)' % self.backend)
  File "<string>", line 1, in <module>
  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 87, in to_odt
    a.make_archive(opts.base_doc, self.asciidoc_file, odf_file)
  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 57, in make_archive
    shutil.copyfile(mc.group(1), os.path.join(td, mc.group(2)))
  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile
    with open(src, 'rb') as fsrc:
IOError: [Errno 2] No such file or directory: '/home/dag/home-made/asciidoc-odf/README__1.png'

[elextr]

[...]

On 16 February 2012 06:21, Dag Wieërs reply@reply.github.com wrote:

The mechanism I was talking about was more a standard interface plugins should adhere to by the AsciiDoc project. Do you think it is worthwhile to discuss on the AsciiDoc mailinglist ?

Can't hurt, but remember that filters are external programs and asciidoc would have to continue to support the current interface (even if deprecated) for some time.

Currently filters don't know if their output is going to be linked or embedded, so it would be a big change for them to all be written to be able to save a file or send to asciidoc depending on some option, especially if they depend on some external program like ditaa.

Also, the a2x currently has a problem slightly related to this, when parsing README.odt:

---

[dag@moria asciidoc-odf]$ a2x -v -b odt --backend-opts="--schema=relaxng/OpenDocument-v1.2-os-schema.rng" -a icons -a iconsdir=/etc/asciidoc/images/icons README.asciidoc -snip- asciidoc: README.asciidoc: line 248: filtering: "/etc/asciidoc/filters/ditaa/ditaa2img.py" -v -o "/tmp/tmpYmkr3D/README__1.png" --scale 4 - Runing filter script /etc/asciidoc/filters/ditaa/ditaa2img.py Temporary input file is /tmp/tmps3Qth9 Exec: java -jar "/etc/asciidoc/filters/ditaa/ditaa0_9.jar" "/tmp/tmps3Qth9" "/tmp/tmpYmkr3D/README__1.png" --overwrite -v --scale 4.000000 >&2

ditaa version 0.9, Copyright (C) 2004--2009  Efstathios (Stathis) Sideris

Running with options: overwrite verbose scale = 4.000000 Reading file: /tmp/tmps3Qth9 Locale: en_US Dialog.bold Rendering to file: /tmp/tmpYmkr3D/README1.png Done in 1sec asciidoc: README.asciidoc: line 248: evaluating: {counter2:image-number} asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.abspath("README1.png")} asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README1.png')[1]} asciidoc: README.asciidoc: line 248: evaluating: {eval:os.path.splitext('README1.png')[1]} asciidoc: README.asciidoc: line 296: evaluating: {counter:admonition} asciidoc: README.asciidoc: line 343: evaluating: {counter:admonition} asciidoc: README.asciidoc: line 367: evaluating: {counter:admonition} asciidoc: README.asciidoc: line 398: evaluating: {counter:admonition} asciidoc: README.asciidoc: line 419: evaluating: {counter:admonition}

Traceback (most recent call last):  File "/usr/bin/a2x", line 934, in    a2x.execute()  File "/usr/bin/a2x", line 356, in execute    self.to_backend()  File "/usr/bin/a2x", line 608, in tobackend    eval('to%s(self)' % self.backend)  File "", line 1, in  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 87, in to_odt    a.make_archive(opts.base_doc, self.asciidoc_file, odf_file)  File "/etc/asciidoc/backends/odt/a2x-backend.py", line 57, in make_archive    shutil.copyfile(mc.group(1), os.path.join(td, mc.group(2)))  File "/usr/lib64/python2.6/shutil.py", line 50, in copyfile    with open(src, 'rb') as fsrc: IOError: [Errno 2] No such file or directory: '/home/dag/home-made/asciidoc-odf/README__1.png'

The proper path wasn't being recorded, should be fixed in git (remember I can't test ATM)

Cheers Lex

---

---

Reply to this email directly or view it on GitHub: https://github.com/dagwieers/asciidoc-odf/issues/29#issuecomment-3987003