Using Emacs 26 threads instead of processes?

[preetpalS]

I'm not sure if this a possibility (or out of scope) but could Emacs 26 threads (when available) be used instead of processes either by opting-in (possibly by customizing a variable (although this would be a breaking change)) or extending the API so that the caller of the library can choose whether to prefer threads or external processes (possibly with a fallback to external processes when threads are not available in an Emacs version) to avoid blocking.

By the way, I wrote a threaded version of async-start (that assumes that the given start-func argument can be invoked safely in multiple threads) that I'd be willing to contribute to any packages that are willing to use it (by posting it into the public domain; although it could be improved).

;; The following requires lexical binding to be activated in the file where it is defined

;; Possible replacement for existing function provided by library.
(defun async-start (start-func &optional finish-func)
  "`finish-func' is executed with the return values from `start-func'."
  (let* ((start-func-results nil)
         (start-func-thread (make-thread (lambda () (setq start-func-results (funcall start-func)))))
         (check-interval 3)
         (start-func-check-func
          (lambda (start-func-check-func)
            (if (thread-alive-p start-func-thread)
                (run-at-time check-interval nil start-func-check-func start-func-check-func)
              (progn
                (when (functionp finish-func)
                  (thread-join start-func-thread)
                  (funcall finish-func start-func-results)))))))
    (run-with-timer check-interval nil start-func-check-func start-func-check-func)))

;; Sample usage
(async-start (lambda () (sleep-for 14) "Hello") #'princ)

;; Emacs version information
(emacs-version) ;; "GNU Emacs 26.0.91 (build 1, x86_64-unknown-freebsd11.1, GTK+ Version 3.22.15) of 2018-01-24"

[preetpalS]

I think that this would be a good alternative for the future as it would avoid issues with spawning new Emacs processes which seems to present some problems including:

• Higher consumption of system resources (https://github.com/jwiegley/emacs-async/issues/96, etc.)
• Emacs initialization related issues (https://github.com/astahlman/ob-async/issues/19, etc.)

Although threading has its own set of issues, the trade-off could be worth it for some use cases of this library.

[Fuco1]

As far as I understand Emacs only runs one thread at a time so if the async function takes a lot of time it still blocks. This would be no different than simply running a task on idle-timer no?

[preetpalS]

@Fuco1 From what I understand based on the docs, is that Emacs Lisp could support running more than one thread at a time in the future (I didn't realize that this until you mentioned it now). Quoting the draft docs:

Emacs Lisp provides a limited form of concurrency, called threads. All the threads in a given instance of Emacs share the same memory. Concurrency in Emacs Lisp is “mostly cooperative”, meaning that Emacs will only switch execution between threads at well-defined times. However, the Emacs thread support has been designed in a way to later allow more fine-grained concurrency, and correct programs should not rely on cooperative threading.

[preetpalS]

Since the multithreading in Emacs 26 is cooperative, I'm closing this thread as I initially made a mistake in assuming that multiple threads could run at the same time.