riscv64: calloc() fails to handle overflow correctly

[mateusz-bloch]

When calling calloc() with SIZE_MAX as both the number of elements and the size of each element, the function does not behave as expected. Instead of returning NULL and setting errno to ENOMEM, it behaves unexpectedly.

Reproduction:

#include <stdio.h>
#include <stdlib.h>
#include <stdint.h>
#include <errno.h>

int main() {
    errno = 0;
    int *ptr = calloc(SIZE_MAX, SIZE_MAX);
    printf("%d \n", ptr);
    printf("%d \n", errno);
    free(ptr);

    return 0;
}

Output RISCV64:

Output IA32:

[lukileczo]

Fixed in https://github.com/phoenix-rtos/libphoenix/commit/4346c835dd6e7b6fd2baa71700983fa5eb255ac3