The max size of exectuble file!

[FFtust]

I config the stack and heap of mp_task like this:

define MP_TASK_STACK_SIZE (8 * 1024)

 #define MP_TASK_STACK_LEN       (MP_TASK_STACK_SIZE / sizeof(StackType_t))
 #define MP_TASK_HEAP_SIZE       (60 * 1024) 

Then I tried to execute a script of whitch size is about 10k, but failed, the log out is: MemoryError: memory allocation failed, allocating 5400 bytes

So, my question is, base on this configer, what is the max size of script that micropython can execute? And what about I using the official configer?

[MrSurly]

Hi @FFtust

While there is a fixed amount of RAM available, it's used for both compile bytecode, and for program objects. It's possible to have very small source code that consumes all of the RAM by allocating large objects, and conversely you can have a very large source code that doesn't create a lot of objects.

It's a trade-off between both. Without seeing your source, or knowing where the error occured (loading the program? After it's running), it's impossible to say why you're having a problem.

[MrSurly]

Here's an example:

MicroPython v1.8.6-1683-ga9517c04 on 2017-10-21; ESP32 module with ESP32
Type "help()" for more information.
>>> import gc
>>> gc.mem_free()
90800
>>> x = []
>>> for i in range(100): x.append(bytearray(1000))
... 
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
MemoryError: memory allocation failed, allocating 1000 bytes

[FFtust]

Hi @MrSurly

I tried two source codes, one only has repeats of " a = 1 ", the other one has codes that import my own library(as you say , created many objects). The former could have the size more than 16K, and no error output, but the latter just could have the size less than 10k, the error output just contained one line, MemoryError: memory allocation failed, allocating 1844 bytes and nothing executed. I think the error occured where loading the program.

Now the question is, If I have a source code of 10KB, just has repeats of "a = 1", how much RAM will it consume?

By the way, all I have said rely on the configer:

define MP_TASK_STACK_SIZE (8 * 1024)

 #define MP_TASK_STACK_LEN (MP_TASK_STACK_SIZE / sizeof(StackType_t))
 #define MP_TASK_HEAP_SIZE (60 * 1024)

[dpgeorge]

If I have a source code of 10KB, just has repeats of "a = 1", how much RAM will it consume?

There are 2 parts to the RAM usage: the first is in the compilation stage where the script is loaded up and compiled into bytecode. The second is the actual storage in RAM required for the bytecode. For a 10kb script with repeated "a = 1", so 1667 lines, it takes about 6670 bytes of RAM to store the corresponding bytecode.

To see if it's the compilation or the execution of the script that runs out of memory, you can try putting a "print(1)" at the start of the script. If it prints that out then the MemoryError is within the execution. You can also try splitting the script into two pieces and importing them separately.

[FFtust]

Hi @dpgeorge

I forgot to tell you that I execute the script (prior written to the file system )by calling "pyexec_file("main.py")" , I have put a "print("1")" at the start of the script, and nothing printed. I also tried to put the same script to the esp32/module folder, and call the function "pyexec_frozen_module()", it worked well.

Then I gradually add the size of script, and I use the "gc.mem_free()" to get the free memory, here is my script: import gc print("memory free at start", gc.mem_free()) a = 1 a = 1 ... # n lines "a = 1" print("memory free at end", gc.mem_free()) When the size of my script is 9353 bytes, the output is memory free at start 39088 memory free at end 39088 when the size is 10563 bytes, it did not print anything but the MemoryError MemoryError: memory allocation failed, allocating 7533 bytes

I tend to believe that the memory erros occured before execution. If so, is there a limit to the script size, whatever the file content is?

[YanMinge]

I have done a similar test. and before the main.py be executed, the heap memory(mp_task_heap) will used about 25*2 = 50k byte. so if mp_task_heap is only 60k byte space. 12k byte size of main.py will cause “MemoryError: memory allocation failed”

but if main.py is 8-10k byte size. After main.py be executed, if we use gc.mem_free() to get its free memeory, seems that the free memroy will become bigger.

Is the memory recovery of GC only work when main.py be executed ? or why does this phenomenon occur？

[dpgeorge]

I tend to believe that the memory erros occured before execution. If so, is there a limit to the script size, whatever the file content is?

Yes, in your case described above it looks like the error occurs during compilation (ie before execution). This is because it needs memory to compile the script to bytecode. If you don't have enough memory for this then it will give the error you are seeing.

It will depend on the actual contents of the script as to how big it can be before running out of memory. One script at 10000 bytes may fail while another at 10000 bytes may succeed.

To get around this issue you can precompile the script using mpy-cross, then copy the resulting .mpy file to the filesystem on the board. Then import as usual. That process doesn't require any compilation to happen on the board.

but if main.py is 8-10k byte size. After main.py be executed, if we use gc.mem_free() to get its free memeory, seems that the free memroy will become bigger.

Is the memory recovery of GC only work when main.py be executed ? or why does this phenomenon occur？

As said above, the compilation process takes some memory. After compilation is finished the memory is freed, so you get more available.

[FFtust]

Hi @dpgeorge @MrSurly, Thank you very much for your help, I precompile the script using mpy-cross, and I did ececute a larger script!

[MrSurly]

@FFtust Can this be closed?