Compiler warning when applying ligatures-scrollback patch

[ghost]

When I apply the st-ligatures-scrollback-20200405-28ad288.diff patch and compile st, I get the following warning:

In function ‘hbtransform’,
    inlined from ‘xmakeglyphfontspecs’ at x.c:1363:2:
hb.c:55:31: warning: argument 1 range [18446744071562067968, 18446744073709551615] exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
   55 |  hb_codepoint_t *codepoints = calloc(len, sizeof(hb_codepoint_t));
      |                               ^
hb.c: In function ‘xmakeglyphfontspecs’:
/usr/include/stdlib.h:542:14: note: in a call to allocation function ‘calloc’ declared here
  542 | extern void *calloc (size_t __nmemb, size_t __size)
      |

[cog1to]

can you dump the whole output of the make command? I don't get that warning on my system. I guess your default CC uses some additional compile flags.

[ghost]

Yeah, here:

st build options:
CFLAGS  = -I/usr/X11R6/include  -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include   -I/usr/include/freetype2 -I/usr/include/libpng16 -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include   -I/usr/include/harfbuzz -I/usr/include/glib-2.0 -I/usr/lib/glib-2.0/include  -DVERSION="0.8.2" -D_XOPEN_SOURCE=600  -O3 -flto -march=native
LDFLAGS = -flto -L/usr/X11R6/lib -lm -lrt -lX11 -lutil -lXft -lXrender -lfontconfig -lfreetype   -lfreetype   -lharfbuzz  -flto
CC      = c99
c99 -I/usr/X11R6/include  `pkg-config --cflags fontconfig`  `pkg-config --cflags freetype2`  `pkg-config --cflags harfbuzz` -DVERSION=\"0.8.2\" -D_XOPEN_SOURCE=600  -O3 -flto -march=native -c st.c
c99 -I/usr/X11R6/include  `pkg-config --cflags fontconfig`  `pkg-config --cflags freetype2`  `pkg-config --cflags harfbuzz` -DVERSION=\"0.8.2\" -D_XOPEN_SOURCE=600  -O3 -flto -march=native -c x.c
c99 -I/usr/X11R6/include  `pkg-config --cflags fontconfig`  `pkg-config --cflags freetype2`  `pkg-config --cflags harfbuzz` -DVERSION=\"0.8.2\" -D_XOPEN_SOURCE=600  -O3 -flto -march=native -c hb.c
c99 -o st st.o x.o hb.o -flto -L/usr/X11R6/lib -lm -lrt -lX11 -lutil -lXft -lXrender `pkg-config --libs fontconfig`  `pkg-config --libs freetype2`  `pkg-config --libs harfbuzz` -flto
In function ‘hbtransform’,
    inlined from ‘xmakeglyphfontspecs’ at x.c:1363:2:
hb.c:55:31: warning: argument 1 range [18446744071562067968, 18446744073709551615] exceeds maximum object size 9223372036854775807 [-Walloc-size-larger-than=]
   55 |  hb_codepoint_t *codepoints = calloc(len, sizeof(hb_codepoint_t));
      |                               ^
hb.c: In function ‘xmakeglyphfontspecs’:
/usr/include/stdlib.h:542:14: note: in a call to allocation function ‘calloc’ declared here
  542 | extern void *calloc (size_t __nmemb, size_t __size)
      |              ^

[ghost]

Ok, after a little debugging, it seems to be caused by -O3 together with -flto. But it should work with -O3 -flto, right?

[cog1to]

yes, it should. the code it's referring is allocating memory for a single terminal line, so I doubt it will get above 9223372036854775807 symbols :)

I can suppress this the same way st suppresses this all over the code: by wrapping the call to calloc with another function that does implicit type conversion. I'm a little surprised the compiler on your side doesn't warn about that.

[cog1to]

Should be fixed now, I think.

[ghost]

still happens on st-ligatures master for me

[ghost]

I think changing len from size_t to unsigned int in hbtransform should fix this problem.

[cog1to]

Fixed in e3197593