Error parsing diatheke output on Ubuntu Trusty

[alphapapa]

Hi there,

First, this is very interesting! I'm glad you found my sword-converter and sword-to-org projects. Have we spoken before? (I'm not sure who dtk01 is.)

This bug report is about a problem parsing the output of diatheke on Ubuntu Trusty. There's some dumb gnome-keyring bug that causes this line to be output for certain programs that use Gtk or GNOME stuff:

couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory

I don't remember the details of the bug, but in this case, it causes output parsing to fail with this backtrace:

Debugger entered--Lisp error: (wrong-type-argument consp nil)
  setcar(nil (("p11-ki" "couldn't load module: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: /usr/lib/x86_64-linux-gnu/pkcs11/gnome-keyring-pkcs11.so: cannot open shared object file: No such file or directory")))
  (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description)))))
  (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description))))))
  (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description)))))))
  (let ((colon-position (seq-position x 58))) (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description))))))))
  (if (= (aref x (1- (length x))) 58) (setq modules-by-category (cons (list (seq-subseq x 0 (1- (length x)))) modules-by-category)) (let ((colon-position (seq-position x 58))) (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description)))))))))
  (while --dolist-tail-- (setq x (car --dolist-tail--)) (if (= (aref x (1- (length x))) 58) (setq modules-by-category (cons (list (seq-subseq x 0 (1- (length x)))) modules-by-category)) (let ((colon-position (seq-position x 58))) (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description))))))))) (setq --dolist-tail-- (cdr --dolist-tail--)))
  (let ((--dolist-tail-- modulelist-strings) x) (while --dolist-tail-- (setq x (car --dolist-tail--)) (if (= (aref x (1- (length x))) 58) (setq modules-by-category (cons (list (seq-subseq x 0 (1- (length x)))) modules-by-category)) (let ((colon-position (seq-position x 58))) (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description))))))))) (setq --dolist-tail-- (cdr --dolist-tail--))))
  (let ((modulelist-strings (process-lines "diatheke" "-b" "system" "-k" "modulelist")) (modules-by-category nil)) (let ((--dolist-tail-- modulelist-strings) x) (while --dolist-tail-- (setq x (car --dolist-tail--)) (if (= (aref x (1- (length x))) 58) (setq modules-by-category (cons (list (seq-subseq x 0 (1- (length x)))) modules-by-category)) (let ((colon-position (seq-position x 58))) (let ((modulename (seq-subseq x 0 (1- colon-position))) (module-description (seq-subseq x (+ 2 colon-position)))) (let* ((v modules-by-category)) (if (listp v) (setcar (nthcdr 0 v) (append (elt modules-by-category 0) (list (list modulename module-description)))) (aset v 0 (append (elt modules-by-category 0) (list (list modulename module-description))))))))) (setq --dolist-tail-- (cdr --dolist-tail--)))) modules-by-category)
  dtk-modulelist()
  (assoc category (dtk-modulelist))
  dtk-module-category("Biblical Texts")
  (cdr (dtk-module-category category))
  (let ((biblical-text-modules (cdr (dtk-module-category category)))) (mapcar (function (lambda (modulename-description) (elt modulename-description 0))) biblical-text-modules))
  dtk-modules-in-category("Biblical Texts")
  dtk-biblical-texts()
  (if (dtk-biblical-texts) (if (not (dtk-go-to)) (let ((dtk-buffer (dtk-ensure-dtk-buffer-exists))) (dtk-switch-to-dtk-buffer) (dtk-mode))) (message "Biblical texts are not presently available via diatheke. Consider installing the desired texts."))
  (if (dtk-buffer-exists-p) (switch-to-buffer-other-window dtk-buffer-name) (if (dtk-biblical-texts) (if (not (dtk-go-to)) (let ((dtk-buffer (dtk-ensure-dtk-buffer-exists))) (dtk-switch-to-dtk-buffer) (dtk-mode))) (message "Biblical texts are not presently available via diatheke. Consider installing the desired texts.")))
  dtk()
  funcall-interactively(dtk)
  call-interactively(dtk record nil)
  command-execute(dtk record)
  helm-M-x(nil #("dtk" 0 3 (match-part "dtk")))
  funcall-interactively(helm-M-x nil #("dtk" 0 3 (match-part "dtk")))
  call-interactively(helm-M-x nil nil)
  command-execute(helm-M-x)

I'm guessing you might be able to fix it (if you want to) by not capturing STDERR from diatheke's output.

Looking forward to seeing this on MELPA!

[alphapapa]

BTW, you might prefer to use the Diatheke-output-parsing code in sword-to-org, e.g.:

https://github.com/alphapapa/sword-to-org/blob/master/sword-to-org.el#L127

You can get a list of verse plists, like this:

(sword-to-org--diatheke-parse-text
 (sword-to-org--diatheke-get-text "ESV" "John 1:1-3"))
;; =>
;; ((:book "John" :chapter 1 :verse 1 :text "In the beginning was the Word, and the Word was with God, and the Word was God.")
;;  (:book "John" :chapter 1 :verse 2 :text "He was in the beginning with God.")
;;  (:book "John" :chapter 1 :verse 3 :text "All things were made through him, and without him was not any thing made that was made."))

I've been thinking about renaming the package org-sword, and maybe functions like these could be turned into a library, as well. What do you think?

[alphapapa]

Also see https://github.com/alphapapa/sword-to-org/issues/3 :)

[dtk01]

Hi! Glad you find it interesting. Not sure if we've spoken before (I don't think so).

1. I went ahead and took a look at your parsing code (thanks for the invitation) and took a quick stab at integrating it -- see https://github.com/dtk01/dtk/commit/7a7d963226484aad784bc3cfd4a27972451e8ed5. You noted the possibility of a library. From my perspective, it certainly makes sense not to reinvent the wheel, especially if you're open to moving parsing functionality into a distinct library. However, I'm wondering if we may be better served, in the long run, by using the XML (OSIS) output of diatheke. Have you given any thought to this?

2. I'll take a look at the gnome-keyring problem. Would you be willing to post the output of (process-lines "diatheke" "-b" "system" "-k" "modulelist") or, even just from the shell, what you see with diatheke -b system -k modulelist? Thanks...

[alphapapa]

However, I'm wondering if we may be better served, in the long run, by using the XML (OSIS) output of diatheke. Have you given any thought to this?

That's a good point. I haven't worked on this project for some time, so I don't remember if I considered that already. I'm guessing that I decided to go with plain-text at the time because I didn't need the extra information in the XML output. Maybe it wouldn't be too difficult to parse the XML in Emacs and render it with...something else in Emacs, haha.

I'll take a look at the gnome-keyring problem. Would you be willing to post the output of (process-lines "diatheke" "-b" "system" "-k" "modulelist") or, even just from the shell, what you see with diatheke -b system -k modulelist? Thanks...

Well, if you decide to use my parsing code, it will solve that problem.

If you decide not to, then I think all you need to do is not capture STDERR from Diatheke, because that line is output to STDERR.

If for some reason you need STDERR, then the line in the error is the first line of apparent output from Diatheke. I'm not at the system in question right now, so I can't reproduce the error at the moment. But I would hate for you to introduce code especially for that error message, because it's essentially an upstream bug that varies by system, and is probably solved in newer Ubuntu releases. I would rather make my own wrapper for Diatheke that removes that message and configure your package to use my wrapper instead of diatheke.

What do you think?

[dtk01]

Apologies for the delay in responding -- my free time is pretty minimal at present. Establishing shared parsing code for diatheke output is certainly the way to go. However, the sword-to-org parsing code for modulelist output does not organize all of the data in the output; instead, it discards information related to module categories as well as the longer module descriptions -- this is all information which is of value in developing a user-friendly interface for module selection. A (granted, more cumbersome) revision of SWORD-TO-ORG--DIATHEKE-GET-MODULES which organizes the diatheke output a bit more is described in d5814cdd29eda224fa89db51de6a535736f16e3e. I suspect this addresses the gnome-keyring issue. Perhaps it also takes us a bit closer to establishing what will be desirable in a shared parsing library?

[thomp]

With a1d4f9400d0276cb3ee0afe3c684c4bd0433f53f, this may no longer be an issue.