Possible bug not respecting org-mode "exclude tags" in the exporter

[jchkoch]

Thanks for developing such a great jupyter notebook exporter from org-mode. I tend to generate both jupyter notebooks and latex (also beamer) files from a single org-mode source file which sometimes requires me to use tags to selectively choose which parts of the org-mode file gets exported to the jupyter notebook and latex/pdf file, respectively. In the system I have set up I use two different tags: noexport for org-mode blocks which should never be exported regardless of the export format and beamer for org-mode blocks which should not appear in jupyter notebooks. The issue that I have come across is that when including the following preamble in org-mode and exporting to a notebook, I seem to get an error.

#+TAGS: noexport(n) beamer(b)
# ' Comment this out to get Beamer only
# '+EXCLUDE_TAGS: beamer

The error backtrace that I get when I call the ox-ipynb exporter (toggle-debug-on-error is enabled in emacs) is appended at the end of this post describing the issue. However, by enabling the org-mode line above, #+EXCLUDE_TAG: beamer, (i.e. uncommenting this line) then the error disappears and ox-ipynb export works perfectly again. Could this possibly be a bug which is fixable in ox-ipynb or is this related to the discussion in issue #15 where ox-ipynb is referred to as not a "regular kind of exporter" not supporting the things the usual ones do? Would be interested in hearing your thoughts regardless and still a great package.

Debugger entered--Lisp error: (args-out-of-range #<buffer L04.org<2>> 28837 29106)
cl--set-buffer-substring(28837 29106 "")

(let* ((v (org-element-property :begin hl)) (v (org-element-property :end hl))) (cl--set-buffer-substring v v ""))

(while (consp --cl-var--) (setq hl (car --cl-var--)) (let* ((v (org-element-property :begin hl)) (v (org-element-property :end hl))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--)))

(let* ((--cl-var-- (reverse (org-element-map (org-element-parse-buffer) 'headline #'(lambda (hl) (if ... ...))))) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* ((v (org-element-property :begin hl)) (v (org-element-property :end hl))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil)

(let ((exclude-tags (or (plist-get (org-export--get-inbuffer-options) :exclude-tags) org-export-exclude-tags))) (let* ((--cl-var-- (reverse (org-element-map (org-element-parse-buffer) 'headline #'(lambda ... ...)))) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* ((v (org-element-property :begin hl)) (v (org-element-property :end hl))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil))

(progn (let ((exclude-tags (or (plist-get (org-export--get-inbuffer-options) :exclude-tags) org-export-exclude-tags))) (let* ((--cl-var-- (reverse (org-element-map (org-element-parse-buffer) 'headline #'...))) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* ((v (org-element-property :begin hl)) (v (org-element-property :end hl))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil)) (let* ((select-tags (or (plist-get (org-export--get-inbuffer-options) :select-tags) org-export-select-tags)) (found nil) (hls (reverse (org-element-map (org-element-parse-buffer) 'headline #'(lambda ... ... ...))))) (if found (progn (let* ((--cl-var-- hls) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* (... ...) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil)))) (let* ((--cl-var-- (reverse (org-element-map (org-element-parse-buffer) 'src-block #'(lambda ... ...)))) (src nil)) (while (consp --cl-var--) (setq src (car --cl-var--)) (goto-char (org-element-property :begin src)) (org-babel-remove-result) (let* ((v (org-element-property :begin src)) (v (org-element-property :end src))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil) (let* ((--cl-var-- ox-ipynb-preprocess-hook) (func nil)) (while (consp --cl-var--) (setq func (car --cl-var--)) (funcall func) (setq --cl-var-- (cdr --cl-var--))) nil) (let ((data (ox-ipynb-export-to-buffer-data)) (ipynb (ox-ipynb-notebook-filename))) (save-current-buffer (set-buffer (get-buffer-create "*ox-ipynb*")) (erase-buffer) (insert (json-encode data))) (switch-to-buffer "*ox-ipynb*") (set (make-local-variable 'export-file-name) ipynb) (get-buffer "*ox-ipynb*")))

(save-current-buffer (set-buffer --buf-copy) (goto-char (point-min)) (progn (let ((exclude-tags (or (plist-get (org-export--get-inbuffer-options) :exclude-tags) org-export-exclude-tags))) (let* ((--cl-var-- (reverse (org-element-map ... ... ...))) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* ((v ...) (v ...)) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil)) (let* ((select-tags (or (plist-get (org-export--get-inbuffer-options) :select-tags) org-export-select-tags)) (found nil) (hls (reverse (org-element-map (org-element-parse-buffer) 'headline #'...)))) (if found (progn (let* ((--cl-var-- hls) (hl nil)) (while (consp --cl-var--) (setq hl ...) (let* ... ...) (setq --cl-var-- ...)) nil)))) (let* ((--cl-var-- (reverse (org-element-map (org-element-parse-buffer) 'src-block #'...))) (src nil)) (while (consp --cl-var--) (setq src (car --cl-var--)) (goto-char (org-element-property :begin src)) (org-babel-remove-result) (let* ((v (org-element-property :begin src)) (v (org-element-property :end src))) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil) (let* ((--cl-var-- ox-ipynb-preprocess-hook) (func nil)) (while (consp --cl-var--) (setq func (car --cl-var--)) (funcall func) (setq --cl-var-- (cdr --cl-var--))) nil) (let ((data (ox-ipynb-export-to-buffer-data)) (ipynb (ox-ipynb-notebook-filename))) (save-current-buffer (set-buffer (get-buffer-create "*ox-ipynb*")) (erase-buffer) (insert (json-encode data))) (switch-to-buffer "*ox-ipynb*") (set (make-local-variable 'export-file-name) ipynb) (get-buffer "*ox-ipynb*"))))

(unwind-protect (save-current-buffer (set-buffer --buf-copy) (goto-char (point-min)) (progn (let ((exclude-tags (or (plist-get ... :exclude-tags) org-export-exclude-tags))) (let* ((--cl-var-- (reverse ...)) (hl nil)) (while (consp --cl-var--) (setq hl (car --cl-var--)) (let* (... ...) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil)) (let* ((select-tags (or (plist-get ... :select-tags) org-export-select-tags)) (found nil) (hls (reverse (org-element-map ... ... ...)))) (if found (progn (let* (... ...) (while ... ... ... ...) nil)))) (let* ((--cl-var-- (reverse (org-element-map ... ... ...))) (src nil)) (while (consp --cl-var--) (setq src (car --cl-var--)) (goto-char (org-element-property :begin src)) (org-babel-remove-result) (let* ((v ...) (v ...)) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil) (let* ((--cl-var-- ox-ipynb-preprocess-hook) (func nil)) (while (consp --cl-var--) (setq func (car --cl-var--)) (funcall func) (setq --cl-var-- (cdr --cl-var--))) nil) (let ((data (ox-ipynb-export-to-buffer-data)) (ipynb (ox-ipynb-notebook-filename))) (save-current-buffer (set-buffer (get-buffer-create "*ox-ipynb*")) (erase-buffer) (insert (json-encode data))) (switch-to-buffer "*ox-ipynb*") (set (make-local-variable 'export-file-name) ipynb) (get-buffer "*ox-ipynb*")))) (and (buffer-live-p --buf-copy) (progn (save-current-buffer (set-buffer --buf-copy) (restore-buffer-modified-p nil)) (kill-buffer --buf-copy))))

(let ((--buf-copy (org-export-copy-buffer))) (unwind-protect (save-current-buffer (set-buffer --buf-copy) (goto-char (point-min)) (progn (let ((exclude-tags (or ... org-export-exclude-tags))) (let* ((--cl-var-- ...) (hl nil)) (while (consp --cl-var--) (setq hl ...) (let* ... ...) (setq --cl-var-- ...)) nil)) (let* ((select-tags (or ... org-export-select-tags)) (found nil) (hls (reverse ...))) (if found (progn (let* ... ... nil)))) (let* ((--cl-var-- (reverse ...)) (src nil)) (while (consp --cl-var--) (setq src (car --cl-var--)) (goto-char (org-element-property :begin src)) (org-babel-remove-result) (let* (... ...) (cl--set-buffer-substring v v "")) (setq --cl-var-- (cdr --cl-var--))) nil) (let* ((--cl-var-- ox-ipynb-preprocess-hook) (func nil)) (while (consp --cl-var--) (setq func (car --cl-var--)) (funcall func) (setq --cl-var-- (cdr --cl-var--))) nil) (let ((data (ox-ipynb-export-to-buffer-data)) (ipynb (ox-ipynb-notebook-filename))) (save-current-buffer (set-buffer (get-buffer-create "*ox-ipynb*")) (erase-buffer) (insert (json-encode data))) (switch-to-buffer "*ox-ipynb*") (set (make-local-variable 'export-file-name) ipynb) (get-buffer "*ox-ipynb*")))) (and (buffer-live-p --buf-copy) (progn (save-current-buffer
(set-buffer --buf-copy) (restore-buffer-modified-p nil)) (kill-buffer --buf-copy)))))

ox-ipynb-export-to-buffer()

(setq buf (ox-ipynb-export-to-buffer))

(let ((ipynb (ox-ipynb-notebook-filename)) (content (buffer-string)) buf) (let ((temp-buffer (generate-new-buffer " *temp*"))) (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (insert content) (org-mode) (set (make-local-variable 'export-file-name) ipynb) (let ((links ...)) (let* (... ...) (while ... ... ... ...) nil)) (let ((bib-entries ...)) (let* (... ...) (while ... ... ... ...) nil))) (and (buffer-name temp-buffer) (kill-buffer temp-buffer))))) (setq buf (ox-ipynb-export-to-buffer)) (save-current-buffer (set-buffer buf) (set (make-local-variable 'export-file-name) ipynb)) buf)

ox-ipynb-export-to-ipynb-buffer(nil nil nil nil nil)

(set-buffer (ox-ipynb-export-to-ipynb-buffer async subtreep visible-only body-only info))

(save-current-buffer (set-buffer (ox-ipynb-export-to-ipynb-buffer async subtreep visible-only body-only info)) (let* ((efn export-file-name) (buf (find-file-noselect efn))) (write-file efn) (save-current-buffer (set-buffer buf) (set (make-local-variable 'export-file-name) efn)) (kill-buffer buf) efn))

ox-ipynb-export-to-ipynb-file(nil nil nil nil)

org-export-dispatch(nil)

funcall-interactively(org-export-dispatch nil)

call-interactively(org-export-dispatch record nil)

command-execute(org-export-dispatch record)

execute-extended-command(nil "org-export-dispatch")

smex-read-and-run(("toggle-debug-on-error" "org-export-dispatch" "org-version" "package-list-packages" "cd" "5x5" "arp" "dbx" "dig" "erc" "ert" "eww" "ftp" "gdb" "irc" "jdb" "man" "mpc" "pdb" "pwd" "rsh" "sdb" "xdb" "calc" "diff" "dirs" "ffap" "gnus" "grep" "help" "ielm" "info" "life" "mail" "mpuz" "ping" "pong" "smex" "talk" "term" "undo" "yank" "zone" "align" "chmod" "debug" "diary" "dired" "ediff" "edirs" ...))

smex()

funcall-interactively(smex)

call-interactively(smex nil nil)

command-execute(smex)

[jkitchin]

Off the top of my head I am not sure how easy it is to use :exclude: tags.

One way might be to export the org file as an org file, and then export that to an ipynb. The downside of that is you can lose some features of the ipynb markup like keywords.

Another option would be to use a preprocessing hook that deletes the exclude headings.

this exporter is not like regular org exporters, and doesn't work on the the parse tree, so it doesn't always support all the usual export features.

[jchkoch]

Thanks for the quick reply. I thought I would just open a issue more to document what works and what doesn't work with :exclude: tags in the notebook exporter. Upon reflecting on this error message I occasionally get when exporting my org notebooks to jupyter notebooks, I realized the only condition where I get this error is when exporting to a jupyter notebook using your exporter when I have :exclude: tags (i.e. "beamer" tag set on a header in org mode) and the org-mode line to :exclude: tags is commented out (like this # '+EXCLUDE_TAGS: beamer). This then tells me that I'm doing something wrong because this line should not be commented out (in my workflow) when exporting to jupyter notebooks using ox-ipynb. It would then appear to me that perhaps it would be possible for ox-ipynb to either (since it appears that ox-ipynb does respect the "normal" exporter rules regarding :exclude: tags):

1. fail more elegantly if there are undefined/unknown tags on headers in org-mode or
2. export all headers if there is no #+EXCLUDE_TAGS: line in the org-file.

I have also created a MWE of this behaviour as an example org file which is attached here which may better illustrate my workflow of using :exclude: tags with ox-ipynb. Anyway, thanks for the awesome package and when I get some time I may dive into seeing if I can add some elisp code to fix this myself.

[gmoutso]

Do you mean exclude_tags instead of exclude_tag? I still get the error as long as there is a tag that is excluded with exclude_tags. So to export to ipynb I remove the exclude_tags option (which is faster than removing the tags from each heading).

[jchkoch]

@gmoutso I'm not sure quite what you mean but yes I mean exclude_tags (the one with a "s"; I have edited my previous comment to include this). The MWE from my last comment shows describes my workflow which works for me to export the same org source file to 1) a Jupyter Notebook and 2) a PDF through LaTeX.

[gmoutso]

@jchkoch Thanks and apologies for the confusion. Your MWE works. If I use exclude_tags to limit export to ipynb, then the following export to ipynb works

#+exclude_tags: setup
* Summary
Hello
* Setup           :setup:
Do not show

but this does not

#+exclude_tags: setup
* Summary
Hello
* Setup             :setup:
Do not show
** Bad bad subsection
Hello

On the other hand, exporting to pdf exports both examples correctly (the subsection is not meant to be exported in the second example).

This actually does not affect me. I actually have a different use case to yours. I want to export to ipynb everything (to share implementation and results) and export to pdf only a subset (the results). If you know of a way to ignore exclude_tags you would make me very happy! And also, many thanks @jkitchin for the package.

[jchkoch]

@gmoutso I think I understand your use case better now. If I understand correctly, you want to export to ipynb everything (and only publish a subset to PDF). In that case your second example should work if you simply remove the word setup in the line #+exclude_tags: setup. As far as my use case is concerned it becomes more difficult because ox-ipynb is not as @jkitchin has mentioned a "normal" exporter and so does not respect tag inheritance (which seems difficult to enable with a notebook exporter for org). So for now, I specifically tag every heading (including sub-headings of headings which themselves are tagged with "beamer" tag) with a "beamer" tag. I hope that makes some sense.