Buffering issue with CCL

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:14 Created by kvutza on 2017-09-05 00:44:45 Original: https://sourceforge.net/p/maxima/bugs/3330

---

When using CCL-backed Maxima over sockets and without labels printed, outputs from Maxima are put out sort-of one push late. With short answers, it is one-answer late, with long answers, the last output part is delayed.

It can be tested e.g. via netcat as the UI; then the late part comes either after one next command or when typing a multi-line string. I tested it via: nc -l 12345 maxima --lisp=ccl64 --vey-quiet -s 12345

I am not sure whether this is issue of Maxima or of CCL. If it should be fixed on the CCL side, let me know please, so that I can file a ticket at the CCL project then.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:15 Created by kvutza on 2017-09-05 23:23:30 Original: https://sourceforge.net/p/maxima/bugs/3330/#f015

---

Adding flushing into mterpri partially cures it:

(defmfun mterpri ()
   (terpri)
   (force-output))

It is still not-output/delayed if a command ends by dollar, and alike for output on terminal-interrupt. If next commands end by semicolon, it works then again. If display2d is set to false, it is bad though, as if mterpri were without the flushing.

PS I forgot to notice that I use Maxima 5.40.0 under Ubuntu 16.04, and CCL64 is tested for 1.11 and 1.12-dev.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:17 Created by kvutza on 2017-09-05 23:48:55 Original: https://sourceforge.net/p/maxima/bugs/3330/#ce73

---

The case with display2d:false is obviously cured by putting flushing into linear display:

(defun linear-displa (form)
  (fresh-line *standard-output*)
  (mgrind form *standard-output*)
  (mterpri))

this via using the mterpri version with the added flushing, or explicitly flushed:

(defun linear-displa (form)
  (fresh-line *standard-output*)
  (mgrind form *standard-output*)
  (terpri)
  (force-output))

I guess that a global replacement from (terpri) into (mterpri) (with the exception of the mterpri definition itself) would be a safe way for this issue.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:19 Created by kvutza on 2017-09-06 04:12:09 Original: https://sourceforge.net/p/maxima/bugs/3330/#e09b

---

Changing (terpri) into (mterpri) at other places did not help at the cases that I've tested (though not sure whether it is relevant at some other situations).

Forcing outputs (under very-quiet CCL via sockets) on errors/interrupts seemed to be handled by putting (force-output) just before throw calls. Regarding part in init-cl.lisp:

(defun maxima-lisp-debugger-repl (condition me-or-my-encapsulation)
  ...
  (force-output)
  (throw 'to-maxima-repl t))

Regarding part in macsys.lisp:

(defun maxima-lisp-debugger (condition me-or-my-encapsulation)
  ...
  (force-output)
  (throw 'return-from-debugger t))

Then to get the evaluation-time info at calls ended by dollar (that is without other/proper output), the flushing has to be put by the end of the respective part (that is just above the "(unless $nolabels" line) of the continue method:

(defun continue (&optional (input-stream *standard-input*)
  ...
    (force-output)
  (unless $nolabels
    (putprop '$% (cons time-used 0) 'time)
    (putprop d-tag (cons time-used  0) 'time))
  ...

One area where I can still see a difference from the cases without console over sockets is calling the system function with an outer program with "&", that is with immediate return. While I get that immediate return when running Maxima at xterm/console, Maxima waits till the end of that called outer program (that is effectively ignoring the "&") when connected over sockets; and here it does not depend on the quietness state. This seems to be a different thing than the lack of flushing of the output though, since here the Maxima call is not ended (before the outer-program end); and that is a different thing than just non-flushed output.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:22 Created by kvutza on 2017-09-06 14:00:48 Original: https://sourceforge.net/p/maxima/bugs/3330/#6b39

---

Another case where it is not flushed is at merror, and a forcing of flushing there cures it for this case as usual.

(defun merror (sstring &rest l)
  ...
  (force-output)
  (throw 'macsyma-quit 'maxima-error))))

Notice that this time I got this issue at CMUCL that I've just tried. In other words, it seems that CMUCL is affected the same way as CCL.

Regarding CMUCL, there is a function open-network-stream that accepts option for buffering, thus may be using that (instead of connect-to-inet-socket) in the Maxima open-socket function would solve it safely at least for CMUCL.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:24 Created by kvutza on 2017-09-06 18:22:53 Original: https://sourceforge.net/p/maxima/bugs/3330/#adda

---

Trying to look at the functions and parameters that CMUCL nad CCL provide, the sys:make-fd-stream function at CMUCL accepts :buffering option that can be one of: :none, :line, :full; default value is :full. Maxima does not set that option, and that may be the source of buffering issues with respect to CMUCL. Notice that this does not need switching to the open-network-stream function. Am not sure whether :line value is safe here vs. whether :none would not lead to inefficiency. I've currently set it to :line for me; though I have all those flushing (see the previous comments) set in, thus am not sure whether this would be enough by itself.

(defun open-socket (host port &optional bin)
  ...
  #+cmu (sys:make-fd-stream (ext:connect-to-inet-socket host port)
               :input t :output t :element-type
               (if bin '(unsigned-byte 8) 'character)
               #+unicode :external-format #+unicode :utf-8
               :buffering :line)
  ...

I would argue that both setting a reasonble buffering where it is possible (like here for CMUCL), and putting flushing in for the cases where buffering can not be reasonably set, is advisable. Am currently looking at the make-fd-stream function at CCL, not being sure why it is not used (as an outer layer) for CCL at the open-socket function in Maxima.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:26 Created by kvutza on 2017-09-06 19:50:37 Original: https://sourceforge.net/p/maxima/bugs/3330/#5253

---

It seems that ccl::make-fd-stream does not have an option for buffering; I have filed a ticket at CCL trac.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:28 Created by kvutza on 2017-09-21 04:59:38 Original: https://sourceforge.net/p/maxima/bugs/3330/#ff69

---

Since overall use of mterpri instead of terpri is not safe (when tried that, got error on maxima --help), I'm only using mterpri in situations where it seems to be safe (at least according to my observations and tests). It means more cases with direct use of force-output that is ugly, though would need to push mterpri into a more prevalent state (to be available in all files/situations) otherwise.

[rtoy]

Imported from SourceForge on 2024-07-01 18:08:30 Created by kvutza on 2017-09-21 05:03:41 Original: https://sourceforge.net/p/maxima/bugs/3330/#e09b/6d88

---

The case with pushing evaluation time (needed e.g. when $nolabels=true and command without output) got more inside now: (force-output)) for being bound to the when $showtime condition, since it is there for that.