Improve mouse-based selection of text (double click to select word, triple click to select sentence)

[titaniumbones]

Not a bug, but an enhancement request, and I'm not sure how easy it would be to implement. Something I miss in pdf-tools is the ability to double click on a word to get the word highlighted for an annotation. In general, I find textselection in pdf-tools clumsierhtan some other PDF viewers, and of course it is much less smooth than when editing text buffers i nEmacs.

In my view,pdf-tools is superior in almost every way to other viewers, with small usability cases like this or #18 being the main remaining exceptions.

i should say I have no idea at all how one would implement this; I'm just identifying "better text selection (esp with mouse)" as a possible goal for the project.

[dalanicolai]

EDIT I've added a command for single word highlighting to that PR. END EDIT

It is not yet merged, but with #29 you could use the keyboard to select a single word by typing the word. or select a region by typing the beginning and end patterns of (don't have to be full words) of the region. Optionally, a command could get added for highlighting single words (so that typing begin and end letter(s), presents only a list of single word candidates).

Until this gets merged you could just load the code in this file to load the feature.

[titaniumbones]

ah, this looks cool, thank you for pointing to it. I've downloaded the code and will play with it over the next couple of days.

I am not sure, but would it maybe be better to have just a single input to the search regex in pdf-keyboard-highlight-single-word? If you're highlighting just one word, it seems likely to be easier to just type it a single time.

I still think a mouse-based workflow would be helpful -- though imperfect, the mouse is closer to the visually-driven annotation process that, in my muscle memory at least, derives from the sense of dragging a pen over a sequence of textual elements on a piece of paper. I'm surprised at how

[titaniumbones]

Hmm, I just found this code on stackexchange, that seems to do more or less exactly what I was asking for:

https://emacs.stackexchange.com/questions/52457/select-a-word-in-a-pdf-by-double-clicking-on-it-with-pdf-tools

I unfortunately don't have enough points on Emacs stackexchange to send a message to the author asking if he'd be willing to submit a PR!

If the code is usable, though, a next step would be to replicate behaviour I love in LibreOffice, where a triple-click selects a sentence. This would likely be harder, I guess, as you'd need to grab an arbitrary amount of text forward and backwards from the current line, looking for some kind of sentence regex, which is probably hard to put together in a pdf.

[dalanicolai]

To highlight a single word by typing it once can be achieved with the pdf-keyboard-highlight (the equivalent of pdf-annot-keyboard-annotate in the PR) by just typing the full word as 'From' pattern and keep the 'To' pattern empty (it is explained in the docstring of that function).

But that mouse double click function might even be handier. Do you mean not enough points to add a comment to that post? I will add a comment then in any case.

About the highlighting a sentence, the regexp search in pdf-tools should just work like ordinary regexp, so that matching a single sentence should be quite straightforward (I guess).

[dalanicolai]

I tried to create a function to highlight sentences, but unfortunately that is not so trivial. Moreover, it seems that Emacs does not recognize triple mouse clicks in pdf-view-mode. Also, I am unable to bind the function to for example [C-down-mouse-1].

However, the code works somewhat, for sentences on a single line. So I am just posting it here for documentation (and for the case you would like to try it out)

(defvar pdf-sel-mode-map nil
  "Keymap for `pdf-sel-mode'.")

(setq pdf-sel-mode-map
      (let ((map (make-sparse-keymap)))
        (define-key map [double-mouse-1] 'pdf-sel-mouse-sentence)
        ;; (define-key map [C-down-mouse-1] 'pdf-sel-mouse-sentence)
        map))

(define-minor-mode pdf-sel-mode
  "\\<pdf-sel-mode-map>Just binding \\[pdf-sel-mouse] to `pdf-sel-mouse'.
`pdf-sel-mouse' selects the text at point and copies it to `kill-ring'."
  :keymap pdf-sel-mode-map)

(defvar pdf-view-active-region) ;; defined in "pdf-view.el"

(defun pdf-sel-mouse-sentence (ev)
  "Select word at mouse event EV and copy it to `kill-ring'."
  (interactive "@e")
  (let* ((posn (event-start ev))
         (xy (posn-object-x-y posn))
         (size (pdf-view-image-size))
         (page (pdf-view-current-page))
         (x (/ (car xy) (float (car size))))
         (y (/ (cdr xy) (float (cdr size))))
         (pdf-view-get-word (pdf-info-gettext page (list x y x y) 'word))
         (pdf-view-get-line (pdf-info-gettext page (list x y x y) 'line)))
    (or (string-match (format "\\. \\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+[\\.?!]\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "^\\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+[\\.?!]\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "\\. \\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+$\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "^\\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+$\\)" pdf-view-get-word) pdf-view-get-line))
    (pdf-annot-add-highlight-markup-annotation
     (cadr
      (caddr
       (car (pdf-info-search-string (match-string 1 pdf-view-get-line) (pdf-view-current-page))))))))

[titaniumbones]

This is starting to get too hard for me to follow, but from what I can tell the issue is with pdf-info-gettext, which when called this way can only retrieve a single line of text. I can see this limitation is built into the way it calls pdf-info-query, but that function is somewhat opaque to me. Since isearch gives access to the text of the whole document, I suppose it ought to be possible to retrieve arbitrarily distant text while preserving the present location, but it is not yet clear to me how to do that.

[dalanicolai]

It is all not so trivial indeed, but I managed to come up with an improved command. What makes it hard is that a sentence can be divided over multiple pdf-info-textregions. Anyway, I polished the command quite a lot already, but obviously it can still get improved. However, for most sentences, without 'special regex characters', and that are not divided over too many text regions, the command works quite well (like in 'ordinary' written texts). It is a quite useful start, and the 'algorithm' can get improved gradually.

 (defvar pdf-sel-mode-map nil
  "Keymap for `pdf-sel-mode'.")

(setq pdf-sel-mode-map
      (let ((map (make-sparse-keymap)))
        (define-key map [double-mouse-1] 'pdf-sel-mouse-sentence)
        ;; (define-key map [C-down-mouse-1] 'pdf-sel-mouse-sentence)
        map))

(define-minor-mode pdf-sel-mode
  "\\<pdf-sel-mode-map>Just binding \\[pdf-sel-mouse] to `pdf-sel-mouse'.
`pdf-sel-mouse' selects the text at point and copies it to `kill-ring'."
  :keymap pdf-sel-mode-map)

(defun pdf-annot-match-sentence-containing-word (word)
  (format "\\([.?!] \\)*\\([^.?!]*%s[^.?!]*\\)\\n*" word))

(defun pdf-sel-mouse-sentence (ev)
  "Select word at mouse event EV and copy it to `kill-ring'."
  (interactive "@e")
  (let* ((posn (event-start ev))
         (xy (posn-object-x-y posn))
         (size (pdf-view-image-size))
         (page (pdf-view-current-page))
         (x (/ (car xy) (float (car size))))
         (y (/ (cdr xy) (float (cdr size))))
         (word (pdf-info-gettext page (list x y x y) 'word))
         (regions (pdf-info-textregions (pdf-view-current-page)))
         (region (seq-find (lambda (region)
                             (and (< (nth 0 region) x (nth 2 region))
                                  (< (nth 1 region) y (nth 3 region))))
                           regions))
         (text (pdf-info-gettext (pdf-view-current-page) region)))
    (if (string= (seq-subseq (pdf-info-gettext (pdf-view-current-page) region) -1) ".")
        (string-match (pdf-annot-match-sentence-containing-word word) text)
      (setq text2 (concat text
                          "\n"
                          (pdf-info-gettext (pdf-view-current-page)
                                            (elt regions (1+ (cl-position region regions))))))
      (if (string-match (pdf-annot-match-sentence-containing-word word) text2)
          (setq text text2)
        (string-match (pdf-annot-match-sentence-containing-word word) text)))
    (let ((edges-list (cdr (caddr (car (pdf-info-search-regexp
                                        (string-replace ")" "\\)"
                                                        (string-replace "(" "\\("
                                                                        (match-string 2 text)))
                                        (pdf-view-current-page)))))))
      (pdf-annot-add-highlight-markup-annotation
       (append (seq-subseq (car edges-list) 0 2) (seq-subseq (car (last edges-list)) 2 4))))))

Just would be great to find out how to bind it to some other mouse click.

[titaniumbones]

does the append line 5 from the bottom have 2 extra parentheses?

I am not completely sure, but I think it's not working for me. I was using that stackexchange code for word selection, tho, which may have messed something up, and I haven't restarted with a new emacs to test.

I would love to get the word selection with a double click, and the sentence with a triple click. But I think mouse events are really complicated to bind in emacs, does that seem right to you?

[dalanicolai]

Ah, the code after that line should not be there... I will remove it (and I have updated/improved a little the code also).

I am more or less sure that the code should work also there, so if it is not due to fixing the '2 parentheses' then probably you messed something up indeed (and of course don't forget to activate pdf-sel-mode). Anyway, this code seems to correctly highlight most sentences. Sometimes it fails to highlight, but then it works by clicking in another location of the sentence, and otherwise you can just highlight by dragging (or by using the keyboard highlight of #29).

Now we just have to find out how to bind this. I don't understand that the original binding works, but that it stops working when changing it slightly. I guess binding the triple click should be a matter of changing the double to triple, but just it seems that in pdf-view triple clicks do not get detected (which probably is a bug).

[vedang]

I have replied on the related change (#29) on how I think of implementing keyboard based annotation creation (on the back of keyboard based region selection and navigation).

Both keyboard navigation improvements as well as mouse selection improvements are on my road-map for pdf-tools, I'm thinking through how I'd like to implement them.

Submitting a cleaned up version of this code as a PR is most welcome @dalanicolai, it'll help anyone else looking for immediate solutions and it will give me a definite direction to think in, re: implementing these features.

[dalanicolai]

I would be happy to create a PR for this, but I would prefer to bind both the word and line highlighting functions. However, I can not find out how to bind triple click in pdf-view mode. Do you happen to know how to achieve that?

Also, I think this code, except for the docstrings has been mostly cleaned up already. So would you accept it after fixing the docstrings? Or do you already have some comments?

[tongjie-chen]

I have an extremely cumbersome implementation by reusing function pdf-view-mouse-set-region, this does not select the word, but it can pop up a quick menu.

(defun pdf-view-get-word (event &optional allow-extend-p
                      rectangle-p)
  "Select a region of text using the mouse with mouse event EVENT.

Allow for stacking of regions, if ALLOW-EXTEND-P is non-nil.

Create a rectangular region, if RECTANGLE-P is non-nil.

    Stores the region in \`pdf-view-active-region'."
  (interactive "@e")
  (setq pdf-view–have-rectangle-region rectangle-p)
  (unless (and (eventp event)
           (mouse-event-p event))
    (signal 'wrong-type-argument (list 'mouse-event-p event)))
  (unless (and allow-extend-p
           (or (null (get this-command 'pdf-view-region-window))
           (equal (get this-command 'pdf-view-region-window)
              (selected-window))))
    (pdf-view-deactivate-region))
  (put this-command 'pdf-view-region-window
       (selected-window))
  (let\* ((window (selected-window))
     (pos (event-start event))
     (begin-inside-image-p t)
     (begin (if (posn-image pos)
            (posn-object-x-y pos)
          (setq begin-inside-image-p nil)
          (posn-x-y pos)))
     (abs-begin (posn-x-y pos))
     pdf-view-continuous
     region)
    (let\* ((pos (event-start event))
       (end (posn-object-x-y pos))
       (end-inside-image-p
        (and (eq window (posn-window pos))
         (posn-image pos))))
      (when (or end-inside-image-p
        begin-inside-image-p)
    (cond
     ((and end-inside-image-p
           (not begin-inside-image-p))
      ;; Started selection outside the image, setup begin.
      (let\* ((xy (posn-x-y pos))
         (dxy (cons (- (car xy) (car begin))
                (- (cdr xy) (cdr begin))))
         (size (pdf-view-image-size t)))
        (setq begin (cons (max 0 (min (car size)
                      (- (car end) (car dxy))))
                  (max 0 (min (cdr size)
                      (- (cdr end) (cdr dxy)))))
          ;; Store absolute position for later.
          abs-begin (cons (- (car xy)
                     (- (car end)
                    (car begin)))
                  (- (cdr xy)
                     (- (cdr end)
                    (cdr begin))))
          begin-inside-image-p t)))
     ((and begin-inside-image-p
           (not end-inside-image-p))
      ;; Moved outside the image, setup end.
      (let\* ((xy (posn-x-y pos))
         (dxy (cons (- (car xy) (car abs-begin))
                (- (cdr xy) (cdr abs-begin))))
         (size (pdf-view-image-size t)))
        (setq end (cons (max 0 (min (car size)
                    (+ (car begin) (car dxy))))
                (max 0 (min (cdr size)
                    (+ (cdr begin) (cdr dxy))))))
        ))))
      (let ((iregion (if rectangle-p
             (list (min (car begin) (car end))
                   (min (cdr begin) (cdr end))
                   (max (car begin) (car end))
                   (max (cdr begin) (cdr end)))
               (list (car begin) (cdr begin)
                 (car end) (cdr end)))))
    (setq tmp-iregion iregion)
    (setq region
          (pdf-util-scale-pixel-to-relative iregion))))
    (setq pdf-view-word (replace-regexp-in-string "[,.!?，。\\"]" "" (pdf-info-gettext (pdf-view-current-page) region 'word)))
    (popup-menu pdf-word-quick-popup)
))

(define-key pdf-view-mode-map [double-mouse-1] 'pdf-view-get-word)

(easy-menu-define pdf-word-quick-popup nil "Quick search"
  \`("Word lone function"
    ["Occur" (lambda() (interactive) (pdf-occur pdf-view-word))]
    ["Google" (lambda() (interactive) (my-google-string pdf-view-word))]
    ["Google Images" (lambda() (interactive) (my-google-for-image-string pdf-view-word))]
    ["Wiki Summary" (lambda() (interactive) (wiki-summary pdf-view-word))]
    ["Thesaurus" (lambda() (interactive) (mw-thesaurus-lookup-string pdf-view-word))]
    ["Collegiate" (lambda() (interactive) (mw-collegiate-lookup-string pdf-view-word))]
    ["Leaners" (lambda() (interactive) (mw-learner-lookup-string pdf-view-word))]
    ["Wiki" (lambda() (interactive) (my-wiki pdf-view-word))]
  ))

You will need to unbind the double mouse event in some pdf-sync.el by commenting out or deleting (define-key kmap [double-mouse-1] 'pdf-sync-backward-search-mouse).

[titaniumbones]

I just spent way too much time on this but have a solution that works for me. Selection is limited to page borders by so far by binding the mouse function to C-double-mouse-1, I am now able to select sentences easily. Pretty great actually.

(defvar pdf-annot-sentence-end "\\([!.?]\\)[)]?\\(\\W+\\|\\n\\|$\\)+" )
(defvar pdf-annot-sentence-begin   "\\(^\\|\\([!.?]\\)\\(\\W+\\|\\n\\|$\\)\\)\\([[:upper:]]\\)")
(defvar pdf-annot-sentence-prefix "\\(^\\|\\([!.?]\\)\\(\\W+\\|\\n\\|$\\)\\)")
(defvar pdf-annot-sentence-contents "\\([[:upper:]]\\([^.!?]*\\)\\(\\n.?*\\)*?\\(\\n.*\\)?\\)")
(defvar pdf-annot-full-sentence (concat pdf-annot-sentence-prefix
                                        pdf-annot-sentence-contents
                                        pdf-annot-sentence-end))
;; (defun pdf-annot-match-sentence-containing-word (word)
;;   (format "\\([.?!] \\)*\\([^.?!]*%s[^.?!]*\\)\\n*" word))

(defun pdf-annot-text-contains-sentence (text)
  (let ((case-fold-search nil))
    (string-match pdf-annot-full-sentence text )
    ))
(defun pdf-annot-text-contains-sentence-start (text)
  (let ((case-fold-search nil))
    (string-match pdf-annot-sentence-begin text) ))
(defun pdf-annot-text-contains-sentence-end (text)
  (let ((case-fold-search nil))
    (string-match pdf-annot-sentence-end text) ))
(defun pdf-annot-find-sentence-start (text position regions page)
  "given a line with no begining, return either the sentence
beginning or the top of the page. "

  (while (and (not (pdf-annot-text-contains-sentence-start text))
              (<  0 (1- position) ))
    (setq position (1- position))
    (setq prev (pdf-info-gettext page (nth position regions)))
    (setq text (concat prev "\n" text)))
  (if  (match-string 4 text)
      (substring text (nth 8  (match-data 1)))
    text))

(defun pdf-annot-find-sentence-end (text position regions page)
  "given text with  no end, add successive lines until
the sentence end is found.  Trim from beginning of sentence before
sending. Check for infinite loop!"

  (while (and (not (pdf-annot-text-contains-sentence-end text))
              (<  (1+ position) (length regions)))
    (setq position (1+ position))
    (setq text (concat text "\n" (pdf-info-gettext page (nth position regions)))))
  (or  (substring text 0  (nth 1  (match-data 1)) ) text))

(defun pdf-sel-sentence-from-text (word text region-pos regions page)
  "allow recursion by abstraction of mouse event from text search.
return the search string for highlighting"
  ;; handle each situation differnetly
  ;;this is very awkward
  (let ((w-start (string-match word text)))
    (cond
     ;; best case: sentence is here! 
     ((pdf-annot-text-contains-sentence text)
      (let ((sentence (match-string 4 text))
            (s-start (nth 0 (match-data 1)))
            (s-end (nth 1 (match-data 1))))
        (cond
         ;; click is *inside* sentence
         ((and (<= s-end w-start) (>= s-start w-start))
          sentence)
         ;; click is *after* sentence. search forward for end. 
         ((> w-start s-end )
          ;; this is lazy. what if there are *two* sentences on line?? oh well
          (setq text (substring text  s-end))
          (pdf-sel-sentence-from-text word text region-pos regions page))
         ;; lcick is before sentence
         (t (pdf-sel-sentence-from-text
             word (substring text (1- s-start)) region-pos regions page)))))
     ;; next best: at start of sentence
     ((pdf-annot-text-contains-sentence-start text)
      (let ((s-start (nth 8 (match-data 1))))
        (cond
         ;; word is after sentence start! 
         ((> w-start s-start)
          ;; if there are more regions, check them out
          (if (elt regions (1+ region-pos)) 
              (pdf-annot-find-sentence-end (substring text s-start) region-pos regions page)
            ;; if not, give the sentence fragment
            (substring text s-start)))
         ;; word is before sentence. Go back and get earlier lines
         (t (pdf-sel-sentence-from-text word (substring text 0 s-start) region-pos regions page)))))
     ;; next try: found end of sentence. Since there's no start of sentence,
     ;; we can assume we're inside the sentence end  search back for
     ;; sentence start or begin.
     ((pdf-annot-text-contains-sentence-end text)
      (if (elt regions (1- region-pos))
        (pdf-annot-find-sentence-start
         (substring text 0  (nth 1  (match-data 1))) region-pos regions page)
        (substring text 0 (elt 1 (match-data 1)))))
     ;; (worst case: middle ofl ong sentence)
     (t
      (let ((start (pdf-annot-find-sentence-start text region-pos regions page )))
        (pdf-annot-find-sentence-end start region-pos regions page )))))
  )

(defun pdf-sel-mouse-sentence (ev)
  "Select sentence at mouse event EV and copy to kill-ring.
New version as of 2022-01-03"
  (interactive "@e")
  (let* ((posn (event-start ev))
         (xy (posn-object-x-y posn))
         (size (pdf-view-image-size))
         (page (pdf-view-current-page))
         (x (/ (car xy) (float (car size))))
         (y (/ (cdr xy) (float (cdr size))))
         (word (pdf-info-gettext page (list x y x y) 'word))
         (regions (pdf-info-textregions page))
         (region (seq-find (lambda (region)
                             (and (< (nth 0 region) x (nth 2 region))
                                  (< (nth 1 region) y (nth 3 region))))
                           regions))
         (region-pos (cl-position region regions))
         (text (pdf-info-gettext page region))
         (matched-sentence ;; store the sentence here after it gets returned 
          (pdf-sel-sentence-from-text word text region-pos regions page)) 
         (edges-list (cdr (caddr (car (pdf-info-search-regexp
                                       (string-replace ")" "\\)"
                                                       (string-replace "(" "\\(" matched-sentence))
                                       page)))))
         (region-edges (append  (seq-subseq (car edges-list) 0 2)
                                (seq-subseq (car (last edges-list)) 2 4))))
    (setq pdf-view-active-region (pdf-info-getselection page region-edges 'glyph))
    (pdf-view-display-region pdf-view-active-region)
    (kill-new (pdf-info-gettext page region-edges 'glyph))))

[mooseyboots]

(defvar pdf-sel-mode-map nil
  "Keymap for `pdf-sel-mode'.")

(setq pdf-sel-mode-map
      (let ((map (make-sparse-keymap)))
        (define-key map [double-mouse-1] 'pdf-sel-mouse-sentence)
        ;; (define-key map [C-down-mouse-1] 'pdf-sel-mouse-sentence)
        map))

(define-minor-mode pdf-sel-mode
  "\\<pdf-sel-mode-map>Just binding \\[pdf-sel-mouse] to `pdf-sel-mouse'.
`pdf-sel-mouse' selects the text at point and copies it to `kill-ring'."
  :keymap pdf-sel-mode-map)

(defvar pdf-view-active-region) ;; defined in "pdf-view.el"

(defun pdf-sel-mouse-sentence (ev)
  "Select word at mouse event EV and copy it to `kill-ring'."
  (interactive "@e")
  (let* ((posn (event-start ev))
         (xy (posn-object-x-y posn))
         (size (pdf-view-image-size))
         (page (pdf-view-current-page))
         (x (/ (car xy) (float (car size))))
         (y (/ (cdr xy) (float (cdr size))))
         (pdf-view-get-word (pdf-info-gettext page (list x y x y) 'word))
         (pdf-view-get-line (pdf-info-gettext page (list x y x y) 'line)))
    (or (string-match (format "\\. \\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+[\\.?!]\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "^\\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+[\\.?!]\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "\\. \\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+$\\)" pdf-view-get-word) pdf-view-get-line)
        (string-match (format "^\\([[:word:],;'‘’\" ]+%s[[:word:],;'‘’\" ]+$\\)" pdf-view-get-word) pdf-view-get-line))
    (pdf-annot-add-highlight-markup-annotation
     (cadr
      (caddr
       (car (pdf-info-search-string (match-string 1 pdf-view-get-line) (pdf-view-current-page))))))))

i tried this out, and also the precursor version on stackoverflow (https://emacs.stackexchange.com/questions/52457/select-a-word-in-a-pdf-by-double-clicking-on-it-with-pdf-tools) and i get this error: pdf-info-query: epdfinfo: Unable to create synctex scanner, did you run latex with `--synctex=1' ?. any idea what's up with that? i wouldn't think synctex would play a role here at all.

[dalanicolai]

@mooseyboots I guess your key-binding is not defined correctly. It is probably bound to some pdf-sync function. Maybe try C-h k to see if you can find out...

[mooseyboots]

@dalanicolai thanks for your response. i had only tried C-h c which doesn't handle mouse clicks.

turns out =pdf-sync-minor-mode= clashes sometimes.

[trace1729]

@dalanicolai thanks for your response. i had only tried C-h c which doesn't handle mouse clicks.

turns out =pdf-sync-minor-mode= clashes sometimes.

Have you solve this issue? I tried to remind [double-mouse-1], but with no luck.