Support user defined memory management

[skdeep2001]

Problem:

s2n today uses its own memory management routines for allocating memory. This is not ideal and/or breaks for uses cases that have very specific memory management requirements especially in a multi-threaded and/or multi-socket and/or some embedded environments.

Proposed Solution:

1) s2n API exposes a new structure of callbacks:

struct s2n_allocator {
    void *(*mem_alloc0)(struct s2n_allocator *allocator, size_t size);
    int (*mem_alloc_aligned)(struct s2n_allocator *allocator, void **memptr, size_t alignment, size_t size);
    int (*mem_lock)(struct s2n_allocator *allocator, const void *mem, size_t size);
    int (*mem_unlock)(struct s2n_allocator *allocator, const void *mem, size_t size);
    int (*mem_advise)(struct s2n_allocator *allocator, void *mem, size_t size, int flags);
    void (*mem_free)(struct s2n_allocator *allocator, void *mem);
    void *impl;
};

2) Provide a default allocator that implements the current functionality:

s2n_mem.h

struct s2n_allocator *s2n_default_allocator(void);

s2n_mem.c

static void *s2n_mem_alloc0(struct s2n_allocator *alloc, size_t size)
{
    (void)alloc;
    return calloc(1, size);
}

static int s2n_mem_alloc_aligned(struct s2n_allocator *alloc, void **memptr, size_t alignment, size_t size)
{
    (void)alloc;
    return posix_memalign(memptr, alignment, size);
}

static int s2n_mem_lock(struct s2n_allocator *alloc, const void *mem, size_t size)
{
    (void)alloc;
    return mlock(mem, size);
}

static int s2n_mem_unlock(struct s2n_allocator *alloc, const void *mem, size_t size)
{
    (void)alloc;
    return munlock(mem, size);
}

static int s2n_mem_advise(struct s2n_allocator *alloc, void *mem, size_t size, int flags)
{
    (void)alloc;
#ifdef MADV_DONTDUMP
    return madvise(mem, size, flags);
#else
    return -1;
#endif
}

static void s2n_mem_free(struct s2n_allocator *alloc, void *mem)
{
    (void)alloc;
    free(mem);
}

static struct s2n_allocator s2n_default_alloc = {
    .mem_alloc0 = s2n_mem_alloc0,
    .mem_alloc_aligned = s2n_mem_alloc_aligned,
    .mem_lock = s2n_mem_lock,
    .mem_unlock = s2n_mem_unlock,
    .mem_advise = s2n_mem_advise,
    .mem_free = s2n_mem_free
};

struct s2n_allocator *s2n_default_allocator(void)
{
    return &s2n_default_alloc;
}

3) Existing s2n_mem APIs need to be modified to accept an s2n_allocator*.

int s2n_alloc(struct s2n_allocator *alloc, struct s2n_blob *b, uint32_t size);
int s2n_realloc(struct s2n_allocator *alloc, struct s2n_blob *b, uint32_t size);
int s2n_free(struct s2n_allocator *alloc, struct s2n_blob *b);
int s2n_free_object(struct s2n_allocator *alloc, uint8_t **p_data, uint32_t size);
int s2n_dup(struct s2n_allocator *alloc, struct s2n_blob *from, struct s2n_blob *to);

4) New APIs for applications that need user defined memory management. These will pass in the allocator to all the underlying calls such as s2n_alloc, s2n_realloc, s2n_free etc. The s2n_config, s2n_connection, s2n_cert_chain_and_key structures will also have a pointer to the allocator so that it can be used by its associated functions and the corresponding *_free() functions.

s2n.h

struct s2n_config *s2n_config_new_pmr(struct s2n_allocator *alloc);
struct s2n_cert_chain_and_key *s2n_cert_chain_and_key_new_pmr(struct s2n_allocator *alloc);
struct s2n_connection *s2n_connection_new_pmr(struct s2n_allocator *alloc, s2n_mode mode);

5) Existing APIs modified to use the default allocator:

For example

struct s2n_config *s2n_config_new(void)
{
    return s2n_config_new_pmr(s2n_default_allocator());
}

Risks: 1) One of the risks is that this requires modifying lots of internal interfaces and propagating the allocators through the codebase. 2) Doesn't address any allocations done by the underlying OpenSSL calls.

Benefits: 1) Allocators can be instrumented to tally allocs vs frees. 2) User can customize the allocation strategy fo multi-thread, multi-socket or even finer granularity.

[alexw91]

Related to: https://github.com/awslabs/s2n/issues/772

[colmmacc]

See my PR for a go at this. I ended up implementing it a little differently in practice, because I have to work around the mlock / advise but I think I came up with a clean way to do it all.

[skdeep2001]

See my PR for a go at this. I ended up implementing it a little differently in practice, because I have to work around the mlock / advise but I think I came up with a clean way to do it all.

This will be very useful ! Tested with mock server client for my application that requires this.

[zaherd]

Addressed in PR#1763