Issue with wbkgrnd when upgrading from ncurses 6.1 to 6.2

[McParen]

Since the upgrade to ncurses 6.2 the wbkgrnd function does not work properly any more.

More precisely, the wide-character background routine can not set anything but 7bit characters (0-127) as the window background.

A minimal example to reproduce the bug with SBCL/cffi:

(let ((scr (initscr)))
    (cffi:with-foreign-objects ((ptr '(:struct cchar_t))
                                (wch 'wchar_t 5))
      (dotimes (i 5)
        (setf (cffi:mem-aref wch 'wchar_t i) 0))

      ;; 126 works 7bit
      ;; 161 doesnt work 8bit
      (setf (cffi:mem-aref wch 'wchar_t 0) 126)
      ;;(setf (cffi:mem-aref wch 'wchar_t 0) #x2592)

      (setcchar ptr wch 0 0 (cffi:null-pointer))

      (wbkgrnd scr ptr)

      (wrefresh scr)
      (wgetch scr)
      (endwin))))

The corresponding C function works for me, so I can not reproduce the Lisp bug in C:

#define _XOPEN_SOURCE_EXTENDED
#include <locale.h>
#include "/home/build/ncurses-6.2-20200328/include/curses.h"

int main()
{
    cchar_t cch;
    const wchar_t wch[2] = {L'\u2591', L'\0'};

    setlocale(LC_ALL, "");

    setcchar(&cch,
             wch,
             0,
             0,
             NULL);             

    initscr();

    bkgrnd(&cch);

    refresh();
    getch();
    endwin();
    return 0;
}

The ncurses C function that does properly work with Lisp FFI: Link

The new version of the wkbgrnd function that does not work for 8bit and wide chars: Link

Is anybody knowlegeable enough about C or ncurses internals to identify the cause of this bug?

[cage2]

On Sun, Apr 05, 2020 at 10:32:51AM -0700, McParen wrote:

Hi!

Since the upgrade to ncurses 6.2 the wbkgrnd function does not work properly any more.

That's bad... :(

[...]

I would like to take a look at the bug (even i am not a cffi expert at all) but the package fails to load on my system with this error:

; caught ERROR: ; READ error during COMPILE-FILE: ; ; Symbol "NEWWIN" not found in the DE.ANVI.NCURSES package. ; ; Line: 571, Column: 36, File-Position: 22987

Moreover seems to me that there is at least a function with '%' prefix convention:

src/window.lisp line 5 `(%newwin height width begin-y begin-x))'

I am using sbcl 2.0.2 on debian x86_64.

Bye! C.

[cage2]

On Sun, Apr 05, 2020 at 10:32:51AM -0700, McParen wrote:

[...]

The corresponding C function works for me, so I can not reproduce the Lisp bug in C:

#define _XOPEN_SOURCE_EXTENDED
#include <locale.h>
#include "/home/build/ncurses-6.2-20200328/include/curses.h"

int main()
{
    cchar_t cch;
    const wchar_t wch[2] = {L'\u2591', L'\0'};

    setlocale(LC_ALL, "");

Just a random thought.

Is this last call needed? If yes i wonder which locale is set for the lisp version of the program.

Bye! C.

[McParen]

Hello, I have removed the % prefixes today, so that the ncurses system can be used separately with the package as a prefix.

The bug you have there should not happen, there are no function calls with % prefixes left. Can you please check whether this is an issue with your .cache/common-lisp?

Which file causes the newwin error?

The file src/window.lisp is not loaded by the asd file, it is an old file that I have not yet removed.

Is this last call needed? If yes i wonder which locale is set for the lisp version of the program.

Yes, the call to setlocale is absolutely needed in C. In lisp, it is inherited from the shell from which the sbcl executable is called, so sbcl calls setlocale when started.

[cage2]

On Sun, Apr 05, 2020 at 12:05:09PM -0700, McParen wrote:

Hi!

Hello, I have removed the % prefixes today, so that the ncurses system can be used separately with the package as a prefix.

That's a good idea!

The bug you have there should not happen, there are no function calls with % prefixes left. Can you please check whether this is an issue with your .cache/common-lisp?

I wiped all the cache than just, in a REPL typed (asdf:make "croatoan") and still get the error.

Which file causes the newwin error?

Sorry i forgot the file:

; Symbol "NEWWIN" not found in the DE.ANVI.NCURSES package. ; ; Line: 571, Column: 36, File-Position: 22987

of file croatoan/src/classes.lisp

The file src/window.lisp is not loaded by the asd file, it is an old file that I have not yet removed.

Oh, sorry again for the inconvenience i did not check the file properly. :(

Is this last call needed? If yes i wonder which locale is set for the lisp version of the program.

Yes, the call to setlocale is absolutely needed in C. In lisp, it is inherited from the shell from which the sbcl executable is called, so sbcl calls setlocale when started.

I see. I was able to compile the lisp program with an older revision of croatioan though. I get a blank screen. My version of ncurses is 6.2-1.

Hope this helps! C.

[McParen]

Sorry, I can not say why your asdf:make goes wrong. I usually ql:quickload the systems directly from the quicklisp/local-projects directory. Maybe asdf:make does something different. I cloned the repository to check and do not see the newwin error. You can see that newwin is exported in the ncurses/package.lisp file.

[McParen]

Yes, the blank screen is also what I get. Thanks for reproducing the bug. I have brought this up on the ncurses mailing list, it seems to be a known issue.

[cage2]

On Sun, Apr 05, 2020 at 01:02:04PM -0700, McParen wrote:

Hi!

Sorry, I can not say why your asdf:make goes wrong.

Sorry, my fault. I rebooted and everything works now! :)

Bye! C.

[cage2]

On Sun, Apr 05, 2020 at 01:18:26PM -0700, McParen wrote:

Hi!

Yes, the blank screen is also what I get. Thanks for reproducing the bug. I have brought this up on the ncurses mailing list, it seems to be a known issue.

I cloned the ncurses-snapshot and compiled it in my home directory:

/configure --prefix=/home/build/ --enable-widec

followed by the usual make and make install

then i compiled the C code

cd /home/build/lib

gcc -c test.c

gcc test.o libncursesw.a -o test

finally i run the program and, with big surprise, the screen is blank as the lisp code!

Am I missing something (very likely) again or the bug is entirely on the C side?

Bye! C.

[McParen]

My guess is that it is entirely on the C side, because the wbkgrnd routine has been substantially changed between ncurses 6.1 and 6.2. I have contacted the author of ncurses, and he is looking into it at the moment.

Thanks for reproducing the bugs, this helps very much. Until others confirm, you can never be sure that you haven't caused the bug yourself and are unaware of it.

[cage2]

On Mon, Apr 06, 2020 at 12:01:23PM -0700, McParen wrote:

Hi!

My guess is that it is entirely on the C side, because the wbkgrnd routine has been substantially changed between ncurses 6.1 and 6.2.

I am sorry that i can not help very much, my C skills are very basic. :(

I have contacted the author of ncurses, and he is looking into it at the moment.

Very well! :)

Bye! C.

[McParen]

As far as I can confirm, the wbkgrnd bug has been fixed with the recent patch ncurses-6.2-20210206:

https://github.com/ThomasDickey/ncurses-snapshots/releases/tag/v6_2_20210206

Arbitrary unicode characters can be set as window backgrounds again.

Unicode test ut05 should work now as it did previously with ncurses 6.1:

https://github.com/McParen/croatoan/blob/89ecd147cf1548f569f23353b3ab656cfb74de1f/test/unicode.lisp#L131

The patch will likely be part of the upcoming ncurses 6.3 release.

[cage2]

On Fri, Feb 12, 2021 at 01:34:13PM -0800, McParen wrote:

As far as I can confirm, the wbkgrnd bug has been fixed with the recent patch ncurses-6.2-20210206:

🎉🎉 😃

[McParen]

Ncurses 6.3 has been released two days ago:

https://www.invisible-island.net/ncurses/announce-6.3.html

The wide background issue is mentioned in the release notes:

These are improvements to existing features:

• provide for wide-characters as background character in wbkgrnd

I've built and (cursorily) tested the new release, everything works as expected, so I'm closing the issue. Enjoy!