spkac command generates 0 size result with HP C/aC++ compiler

[kinichiro]

spkac command generates 0 size result with HP C/aC++ compiler on HP-UX. with gcc on HP-UX, it works fine.

command line is like this.

openssl spkac -key  -challenge hello -out 

checking with gdb, I found that at apps/spkac.c line 243,

BIO_printf(out, "SPKAC=%s\n", spkstr);

generates 0 size output.

spkstr contains encoded string like this.

(gdb) p spkstr
$3 = 0x600000000004c640 "MIICQzCCASswggEgMA0GCSqGSIb3DQEBAQUAA4IBDQAwggEIAoIBAQCtrLZGvgIPU1/dZQQsNt1zl+GKNHubxQbc0T6rGN5zNAC9niY2qXWyF//E0YjJHcsETBAGJBIhCYedEvSDqOOXEk0xclA6LK6HmsSDPydJieEq/7ozJYwneC8R7iGSt4nzILuoNoeSmcxanhKC"...

[kinichiro]

1st, I added fprintf() before BIO_printf(),

242          } else {
243                  fprintf(stderr, "(stderr) SPKAC=%s\n", spkstr);
244                  BIO_printf(out, "SPKAC=%s\n", spkstr);
245                  ret = 0;
246          }

fprintf outputs full of spkstr, but BIO_printf nothing.

2nd, put '\0' in some place of spkstr, and found spkstr[116] = '\0' enable BIO_printf to output 116 bytes string.

                     spkstr[116] = '\0';
                     BIO_printf(out, "SPKAC=%s\n", spkstr);

3rd, check spkstr[116] with gdb, but it was just character code '8'.

# gdb ../apps/.libs/openssl
..
(gdb) dir ../crypto/.libs
Source directories searched: /tmp/work/x/libressl-2.2.0/test/../crypto/.libs:$cdir:$cwd
(gdb) break spkac_main
Breakpoint 1 at 0x40000000000e5c00:0: file spkac.c, line 187 from /tmp/work/x/libressl-2.2.0/test/../apps/.libs/openssl.
(gdb) run spkac -key test/key/genpkey_rsa.pem -challenge hello
Starting program: /tmp/work/x/libressl-2.2.0/test/../apps/.libs/openssl spkac -key test/key/genpkey_rsa.pem -challenge hello
Breakpoint 1, spkac_main (argc=5, argv=0x9ffffffffffff748) at spkac.c:187
187             ENGINE *e = NULL;
(gdb) b 243
Breakpoint 2 at 0x40000000000e6340:0: file spkac.c, line 243 from /tmp/work/x/libressl-2.2.0/test/../apps/.libs/openssl.
(gdb) c
Continuing.
Breakpoint 2, spkac_main (argc=5, argv=0x9ffffffffffff748) at spkac.c:243
243                             fprintf(stderr, "(stderr) SPKAC=%s\nlength=%d\n", spkstr, strlen(spkstr));
(gdb) p spkstr
$1 = 0x600000000004c640 "MIICQzCCASswggEgMA0GCSqGSIb3DQEBAQUAA4IBDQAwggEIAoIBAQC64GUK9W0z5o+nG79mWJNn6hE8uD4C104F2elHsPdbPkOD+Q1wBkKtg65bOi4y8drGLhfu3F8UbtVwsynfogpt1JozXsWeKEIIAHOKCw+nf+qePkxMAXj9P9Dz5N3j+/Drlvb0qcqti6kyDOvK"...
(gdb) p spkstr[116]
$2 = 56 '8'
(gdb)

4th, I thought BIO_printf with aCC compiler can not output large size string. and tested with small program using BIO_printf though, it works fine.

/* biosample.c */
#include 
#include 
#include 
main(int argc, char *argv[]) {
    BIO *out = NULL;
    char msg[1024];
    out = BIO_new_fp(stdout, BIO_NOCLOSE);
    strcpy(msg, "12345678901234567890123456789012345678901234567890");
    BIO_printf(out, "msg=%s\n", msg);
    strcat(msg, "12345678901234567890123456789012345678901234567890");
    BIO_printf(out, "msg=%s\n", msg);
    strcat(msg, "12345678901234567890123456789012345678901234567890");
    BIO_printf(out, "msg=%s\n", msg);
    strcat(msg, "12345678901234567890123456789012345678901234567890");
    BIO_printf(out, "msg=%s\n", msg);
    BIO_free_all(out);
}

result was this, it outputs over 100 bytes string.

# ./biosample
msg=12345678901234567890123456789012345678901234567890
msg=1234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
msg=123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890
msg=12345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789012345678901234567890

5th, comment out NETSCAPE_SPKI_b64_encode() and allocate spkstr by malloc(), BIO_printf outputs string.

   227          /*spkstr = NETSCAPE_SPKI_b64_encode(spki);*/
   228          char dbg[201] = "123456789012345678901234567890...(200 bytes string)...";
   229          spkstr = malloc(201);
   230          strcpy(spkstr, dbg);

[kinichiro]

using BIO_puts() or BIO_write() instead of BIO_printf() solves this problem.

/*BIO_printf(out, "SPKAC=%s\n", spkstr);*/
BIO_puts(out, "SPKAC=");
BIO_puts(out, spkstr);

/*BIO_printf(out, "SPKAC=%s\n", spkstr);*/
BIO_write(out, "SPKAC=", 6);
BIO_write(out, spkstr, strlen(spkstr));

[kinichiro]

BIO_printf() calls vasprintf() and it uses vsnprintf() to calcurate string length. But with C/aC++, vsnprintf() returns just -1 if maxsize is smaller than the number of characters formatted. with GCC, vsnprintf() returns the number of bytes that would have been written to.

testing with small program like this.

#include 
main() {
    int ret;
    char buf[5];
    char str[] = "123456789";
    ret = snprintf(buf, 5, "%s", str);
    printf("ret = %d\n", ret);
}

C/aC++ : ret = -1 GCC : ret = 9

in crypto/compat/bsd-asprintf.c, INIT_SZ is 128, and this case needs 778 bytes, then vsnprintf() returns always -1 and outputs 0 size result.

And I found that UNIX 2003 standard settings are needed when compiling time. -D_XOPEN_SOURCE=600 and export UNIX_STD=2003

[kinichiro]

After all, export UNIX_STD=2003 before ./configure and make check solves this issue.