How to use other alloc, free and realloc functions with duk_create_heap()?

[edoardomazzella]

Hi, i'm trying to use this function as alloc: static inline void chHeapAlloc(memory_heap_t heapp, size_t size) { return chHeapAllocAligned(heapp, size, CH_HEAP_ALIGNMENT); }

I've tried in the main.c to wrap it in this way, before calling duk_create_heap()

void scriptly_alloc(void ctx, duk_size_t size) { void *res;

duk_context thr = (duk_context ) ctx;

res = chHeapAlloc(thr->heap,size);

return res; }

But an error occurs: "dereferencing pointer to incomplete type duk_context". Can you tell me the right way to do it? Thank you!

[svaarala]

You shouldn't be deferencing thr->heap at all because the duk_context type should be treated as an opaque pointer in the calling application.

The first argument to a wrapped alloc/realloc/free function is a void pointer userdata, which is filled based on the heap_udata argument of duk_create_heap(), see http://duktape.org/api.html#duk_create_heap.

typedef void *(*duk_alloc_function) (void *udata, duk_size_t size);
typedef void *(*duk_realloc_function) (void *udata, void *ptr, duk_size_t size);
typedef void (*duk_free_function) (void *udata, void *ptr);

In your case you could initialize as:

/* Assume 'heapp' is a `memory_heap_t *` value that you need. */
duk_context *ctx = duk_create_heap(scriptly_alloc,
                                   scriptly_realloc,
                                   scriptly_free,
                                   (void *) heapp,
                                   my_fatal);

Then:

void *scripty_alloc(void *udata, duk_size_t size) {
    memory_heap_t *heapp = (memory_heap_t *) udata;
    return chHeapAlloc(heapp, size);
}

[edoardomazzella]

i don't know if the problem is linked with alloc free and realloc, but my program crashes when i do duk_eval_string(), whatever is the argument. In particular it crashes inside duk__parse_func_body(comp_ctx, 1, / expect_eof / 1, / implicit_return_value / -1); / expect_token / that is a function called some levels lower than duk_eval_string(). Inside duk__parse_func_body expect_token is evaluated <0 and duk_advance(comp_ctx) is executed and crashes. What's wrong?

[svaarala]

Realloc is needed yes.

You could start diagnosing the problem first by making the scriptly_alloc() etc calls use normal ANSI C memory functions: alloc(), realloc(), free(). If that works, then the wrapping itself and other aspects of the setup are correct. If not, the custom memory calls are not necessarily the root cause and you may need to look elsewhere.

Re: the eval, when you say "crash" do you mean that Duktape calls abort() due to a fatal error (uncaught error typically)?

[edoardomazzella]

The operating system calls this expection handler /*

• @brief Unhandled exceptions handler.
• @details Any undefined exception vector points to this function by default.
• This function simply stops the system into an infinite loop.
• @notapi / /lint -save -e9075 [8.4] All symbols are invoked from asm context.*/ void _unhandled_exception(void) { ...

[svaarala]

Do you have a fatal error handling in your duk_create_heap() call?

[edoardomazzella]

no i used the dafault one

[svaarala]

The default one calls abort() without any I/O. You could add a custom fatal error handler and see if it is called. If it is called, there's no actual crash, but most likely just an uncaught error, e.g. a syntax error in the input.

[edoardomazzella]

ok, it's not a fatal error, the problem is this one: many levels below the eval_string, inside duk_hbuffer_resize(thr, bw_ctx->buf, new_sz); it throws this error: if (new_size > DUK_HBUFFER_MAX_BYTELEN) { DUK_ERROR_RANGE(thr, "buffer too long"); } but i still don't understand how to solve this problem

[svaarala]

Could you provide the C code that causes this problem?

[edoardomazzella]

it's a ChibiOS project that has to run on STM32, do you want just the main.c or the entire project? I'm testing eval_string with a simple print. (It should work also with default malloc free and realloc for ChibiOS creator's opinion)

This is the main

include "hal.h"

include "ch_test.h"

include "chprintf.h"

include "scriptly.h"

BaseSequentialStream chp = (BaseSequentialStream ) &SD2; / Duktape contexts and strings / static duk_context *ctx;

/ Green LED blinker thread, times are in milliseconds. / static THD_WORKING_AREA(waThread1, 128); static THD_FUNCTION(Thread1, arg) {

(void)arg; chRegSetThreadName("blinker"); while (true) { palClearPad(GPIOA, GPIOA_LED_GREEN); chThdSleepMilliseconds(500); palSetPad(GPIOA, GPIOA_LED_GREEN); chThdSleepMilliseconds(500); } }

static duk_ret_t native_print(duk_context ctx) { chprintf(chp,"ciao mondo!\n"); return 0; / no return value (= undefined) */ }

static void my_fatal(void udata, const char msg) { (void) udata;

chprintf(chp, "*** FATAL ERROR: %s\n", (msg ? msg : "no message"));
abort();

}

/ Application entry point. / int main(void) {

/* System initializations.

• HAL initialization, this also initializes the configured device drivers and performs the board-specific initializations.

• Kernel initialization, the main() function becomes a thread and the RTOS is active. */ halInit(); chSysInit();

  / Duktape initialization. / ctx = duk_create_heap(NULL,NULL,NULL,NULL,my_fatal);

  duk_push_c_function(ctx, native_print, 0); duk_put_global_string(ctx, "print"); / Creates the blinker thread. / chThdCreateStatic(waThread1, sizeof(waThread1), NORMALPRIO, Thread1, NULL);

  while (true) { duk_eval_string(ctx,"print();"); chThdSleepMilliseconds(500); } }

[svaarala]

Hmm, what if you just reduce the main to doing the Duktape stuff with no other threads etc? I.e.

• duk_create_heap()
• duk_push_c_function() + duk_put_global_string() for the print() binding
• duk_eval_string(ctx, "print();");

If that also fails, it would probably be a good idea to enable DUK_USE_ASSERTIONS (configure.py -DDUK_USE_ASSERTIONS) to see if some assert gets triggered. It may be that e.g. type detection or some built-in bindings are broken. For example, if floor() or fmod() works incorrectly, very interesting outcomes may happen.

[svaarala]

@edoardomazzella Any progress on this?

[edoardomazzella]

Sorry for the lateness, it does not work also removing the other thread, and using duk_use_assertions i get the following errors and the compilation fails:

c:/chibistudio/tools/gnu tools arm embedded/6.3 2017q1/bin/../lib/gcc/arm-none-eabi/6.3.1/../../../../arm-none-eabi/bin/ld.exe: address 0x808c0d4 of build/ch.elf section .text' is not within regionflash0'

c:/chibistudio/tools/gnu tools arm embedded/6.3 2017q1/bin/../lib/gcc/arm-none-eabi/6.3.1/../../../../arm-none-eabi/bin/ld.exe: build/ch.elf section .rodata' will not fit in regionflash0'

c:/chibistudio/tools/gnu tools arm embedded/6.3 2017q1/bin/../lib/gcc/arm-none-eabi/6.3.1/../../../../arm-none-eabi/bin/ld.exe: address 0x808c0d4 of build/ch.elf section .text' is not within regionflash0'

c:/chibistudio/tools/gnu tools arm embedded/6.3 2017q1/bin/../lib/gcc/arm-none-eabi/6.3.1/../../../../arm-none-eabi/bin/ld.exe: region `flash0' overflowed by 761592 bytes

it seems like enabling the assertions the program is too big. May disable some DUK_USE to make it fit in the memory?

p.s. i just substituted the #undef DUK_USE_ASSERTIONS with #define DUK_USE_ASSERTIONS in duk_config.h. Is this the same of configuring with -DDUK_USE_ASSERTIONS?

[svaarala]

Hmm, I don't think you can reduce footprint enough with other options to make assertions fit unfortunately. Editing duk_config.h manually with #define DUK_USE_ASSERTIONS should be OK.

You could try with DUK_USE_SELF_TESTS instead of assertions, to see if anything catches.

I've seen issues similar to yours with some platforms where the math library provided by the platform is broken. In particular, any math library functions that Duktape itself needs (beyond implementing the Math built-in) may cause very odd behavior, at least fmod() and floor() are like that. So you could also double check that the math library is implemented correctly.

[edoardomazzella]

apparentely floor and fmod works fine, i don't know how to test the entire library.

[svaarala]

What's particularly important for fmod() is that for exact integers up to 2^53 fmod() with modulus 2^32 works correctly. For example, if the input is 123 * 2^32 + 234:

• fmod(input, 2^32) must be exactly 123.
• floor(input / 2^32) must be exactly 234.

In addition to the math library sometimes there are problems with 64-bit integers and/or integer-float casts. Though I've encountered these mostly with older embedded targets (with an out of date uclibc for example).

In any case it seems sometimes is fundamentally broken in your build. What would be important is to try to find a case which works so you could then try to pinpoint what fails. Towards that end one interesting test would be to create a heap and use the C API without compiling any Ecmascript code. Another would be the smallest possible eval test.

C API only:

duk_create_heap(...);

duk_push_object(ctx);
duk_push_uint(ctx, 123);
duk_put_prop_string(ctx, -2, "testProperty");

duk_get_prop_string(ctx, -1, "testProperty");
printf("property=%ld\n", (long) duk_get_int(ctx, -1));

duk_destroy_heap(ctx);

Minimal eval:

duk_create_heap(...);
duk_eval_string(ctx, "123 + 234");
printf("result=%s\n", duk_safe_to_string(ctx, -1));
duk_destroy_heap(ctx);

[svaarala]

DUK_USE_SELF_TESTS covers fmod() in a very basic manner, but I'll merge in a test for a fmod-splitting a 52-bit integer in #1657.

[edoardomazzella]

int ciao = 123 * 2^32 + 234; chprintf(chp, "fmod: %f\n\r", fmod(ciao,2^32)); //i obtain 32 chprintf(chp, "floor: %f\n\r", floor(ciao / 2^32)); //i obtain 222 so they don't work. The test that uses only the C api without compiling works fine. With the second test the program crashes as usual with duk_eval_string.

[svaarala]

In C ^ is not exponentation, I just used it as a general math notation. The correct check would be:

#include <stdio.h>
#include <math.h>
void main(void) {
    double ciao = 528280977642.0;
    printf("fmod: %f\n", fmod(ciao, 4294967296.0));
    printf("floor: %f\n", floor(ciao / 4294967296.0));
}

This should print:

fmod: 234.000000
floor: 123.000000

[edoardomazzella]

i'm sorry for that stupid error. Anyway both fmod and floor work correctly. Thank you for the reply.

[svaarala]

@edoardomazzella Ok, good to hear fmod() and floor() work correctly. Unfortunately that means the mystery remains :-/

[edoardomazzella]

how many bytes should duk_create_heap alloc?

[svaarala]

I don't know the exact figure - it allocates a heap, creates a default thread which involves creating built-in objects, interning built-in strings, etc. On x86 that comes to around maybe 70-80kB with ES5.1 and all post-ES5 features enabled, lowmem builds come to around 30kB, and ROM built-in enabled builds come close to 1.5kB.

[edoardomazzella]

Using a function of the RTOS, i can see that the size, in bytes, of the free core memory after using duk_create_heap is 16KB. Is it enough? Anyway this error is thrown inside duk_eval_string(ctx,"123 + 234") : h = duk_hbuffer_alloc(thr->heap, size, flags, &buf_data); if (DUK_UNLIKELY(h == NULL)) { DUK_ERROR_ALLOC_FAILED(thr); }

[svaarala]

16kB is not enough unless you're using ROM built-ins - and even then, after 1.5kB for init you'll only have 14.5kB for running code which is very little.

[svaarala]

Oops, I was commenting on the RAM required to create a heap, now I noticed you said 16kB was available after heap creation. It's quite possible you'll run into an out-of-memory with 16kB free after heap creation if you're not using low memory options. Are you using config/examples/low_memory.yaml?

[edoardomazzella]

ok i tried with an stm32f7 and the available ram memory is now over 100K just before "duk_eval_string(ctx, "123 + 234");" but i still have problems. In particular debugging i notice that this happens:

duk_eval_string(ctx, "123 + 234"); -------> duk_compile_raw(ctx, src_buffer, src_length, flags | DUK_COMPILE_EVAL); -------> dukdo_compile(ctx, (void ) comp_args); -------> duk_js_compile(thr, comp_args->src_buffer, comp_args->src_length, flags); -------> h = duk_hbuffer_alloc(thr->heap, size, flags, &buf_data); -------> DUK_ALLOC(heap, alloc_size); -------> DUK__VOLUNTARY_PERIODIC_GC(heap); -------> duk_heap_mark_and_sweep(heap, DUK_MS_FLAG_VOLUNTARY /flags*/); -------> dukmark_roots_heap(heap); -------> duk__mark_heaphdr(heap, (duk_heaphdr ) heap->heap_thread); -------> duk__mark_hobject(heap, (duk_hobject ) h); -------> duk__mark_hobject(heap, (duk_hobject ) h); -------> duk_hstring key = DUK_HOBJECT_E_GET_KEY(heap, h, i); ------->

void _unhandled_exception(void) { /lint -restore/

while (true) { } }

Any idea? Thank you!

[edoardomazzella]

Solved, it was just a stack overflow. I'm sorry for making you waste you time, thank you for the support!

[svaarala]

No worries, you solved it before I had time to react :) Stack overflows are quite difficult because they cause unpredictable behavior and Duktape doesn't automatically notice them.