[BUG] ALLOCATION_FAILED when drawing Textures to display

[AlexanderPilhar]

Describe the bug Device is a M5Stack Core2. Sometimes, when drawing a new scene, this happens (no matter what scene should be drawn):

EXCEPTION error. 
ALLOCATION_FAILED
  0: ByteArray                 <sdk>/core/collections.toit:1030:5
  1: Canvas                    <pkg:toit-pixel-display>/true_color.toit:43:13
  2: Canvas.create_similar     <pkg:toit-pixel-display>/true_color.toit:67:12
  3: WindowTexture_.write2_    <pkg:toit-pixel-display>/texture.toit:904:31
  4: SizedTexture.write_       <pkg:toit-pixel-display>/texture.toit:285:5
  5: Texture.write             <pkg:toit-pixel-display>/texture.toit:158:5
  6: TrueColorPixelDisplay.redraw_rect_.<block> <pkg:toit-pixel-display>/pixel_display.toit:961:22
  7: HashedInsertionOrderedCollection_.hash_do_ <sdk>/core/collections.toit:2125:3
  8: Set.do                    <sdk>/core/collections.toit:2275:10
  9: TrueColorPixelDisplay.redraw_rect_ <pkg:toit-pixel-display>/pixel_display.toit:961:15
 10: PixelDisplay.update_frame_buffer <pkg:toit-pixel-display>/pixel_display.toit:368:13
 11: PixelDisplay.draw.<block> <pkg:toit-pixel-display>/pixel_display.toit:304:7
 12: PixelDisplay.draw         <pkg:toit-pixel-display>/pixel_display.toit:289:3
 13: show_scene_change_status  /C:/path/to/my-project/main.toit:354:11
 14: on_touch_up               /C:/path/to/my-project/main.toit:502:5
 15: main                      /C:/path/to/my-project/main.toit:641:9
 16: __entry__.<lambda>        <sdk>/core/entry.toit:48:20

To reproduce I don't have any instructions on how to reproduce this issue. I have different scenes in my application - each one first removing all textures from the scene before, then adding some SizedTextures and finally drawing them to the display. Most of the time everything is working as expected. Only sometimes the application crashes when drawing a new scene, producing the output above.

Expected behavior Scene gets drawn to display without crashing the application.

Screenshots None

SDK and console version (please complete the following information): Model: esp32-4mb Firmware: v1.6.9

Additional context None

EDIT Firmware: v1.6.11 (updated)

[AlexanderPilhar]

I modified my code a bit to see if the application manages re-drawing the scene:

show_scene --retry/int=0 -> none:

  // clean
  display.remove_all

  // adding multiple SizedTextures
  // ...

  draw_exception := catch:

    // draw
    display.draw
    current_scene = SCENES[1]

  if draw_exception:

    retry++
    print "Retry drawing scene #$retry.stringify"
    show_scene --retry=retry

This leads to following output:

Retry drawing scene #1
Software reset via esp_restart. crashes=1 out-of-memory=1

Also, this time the application restarts immediately. Before, the application just stopped to do anything without restarting.

[erikcorry]

It looks like you are running out of memory. It's hard to say what exactly could cause this.

Perhaps your scenes are just too complicated, or perhaps you are somehow leaking memory by holding onto references.

You may find it helpful to use process_stats https://libs.toit.io/core/utils/library-summary#process_stats(0%2C0%2C0%2C) to get information about memory usage. It gives the most accurate answer right after a garbage collection. You can't directly trigger a gc, but you can see the gc count in the results from process_stats. Just before throwing this exception it will do three GCs in a row in an attempt to free up enough memory.

The stack trace shows the texture code trying to create a canvas to composit transparent and semitransparent graphics. At this point it needs to create three byte arrays of up to 2000 bytes each, which maybe pushes it over the edge. You could try to reduce this limit (at the expense of slowing down screen update), which comes from around line 943 in pixel_display.toit. (To do this you will have to work from a local copy of the released package - see https://docs.toit.io/language/package/pkgconcepts/#local-package )

We are working on a new garbage collector, which will be more economical with memory.

[erikcorry]

Instead of using a local copy of the package you could also make a subclass. It's a bit dodgy because you are overriding a private method, but currently it works:

class FrugalTrueColorDisplay extends pixel_display.TrueColorPixelDisplay:
  constructor driver:
    super driver

  max_canvas_height_ width/int -> int:
    height := round_down (1200 / width) 8
    // We can't work well with canvases that are less than 4 pixels tall.
    return max 4 height

See https://github.com/toitware/toit-pixel-display/blob/daa51c6a443e4a38fcee7eefedac41ec19bb9322/src/pixel_display.toit#L943

[AlexanderPilhar]

Thanks for all the information! I modified my application to print process_stats right before drawing a new scene and this is the output:

[main.toit] 2022-04-28T07:04:51.216113Z: <process initiated>
[main.toit] process_stats:
[main.toit]   GC count                       = 0
[main.toit]   Allocated memory               = 0
[main.toit]   Reserved memory                = 0
[main.toit]   Process message count          = 0
[main.toit]   Bytes allocated in object heap = 14236
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 19
[main.toit]   Allocated memory               = 9732
[main.toit]   Reserved memory                = 12288
[main.toit]   Process message count          = 2
[main.toit]   Bytes allocated in object heap = 293866
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 34
[main.toit]   Allocated memory               = 10512
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 0
[main.toit]   Bytes allocated in object heap = 579803
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 53
[main.toit]   Allocated memory               = 10260
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 1
[main.toit]   Bytes allocated in object heap = 966887
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 65
[main.toit]   Allocated memory               = 10560
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 0
[main.toit]   Bytes allocated in object heap = 1243237
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 93
[main.toit]   Allocated memory               = 11948
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 1
[main.toit]   Bytes allocated in object heap = 1666678
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 123
[main.toit]   Allocated memory               = 12672
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 1
[main.toit]   Bytes allocated in object heap = 2092397
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
[main.toit] process_stats:
[main.toit]   GC count                       = 170
[main.toit]   Allocated memory               = 13976
[main.toit]   Reserved memory                = 16384
[main.toit]   Process message count          = 0
[main.toit]   Bytes allocated in object heap = 2526559
[main.toit]   Group ID                       = 2
[main.toit]   Process ID                     = 2
[main.toit] 
Software reset via esp_restart. crashes=1 out-of-memory=1

I do have some objects in global space which serve as buttons (a RoundedCornerWindow for the background and a SizedTexture for name) - they are assigned each time right before a new scene gets drawn. Also, is display.remove_all the correct way to clean up a scene?

[AlexanderPilhar]

Tried another round switching between two very simple scenes with less Textures and this is how far I got:

process_stats:
  GC count                       = 2248
  Allocated memory               = 14584
  Reserved memory                = 16384
  Process message count          = 1
  Bytes allocated in object heap = 30608776
  Group ID                       = 1
  Process ID                     = 1
Software reset via esp_restart. crashes=1 out-of-memory=1

[erikcorry]

You heap is growing suspiciously, but it's still quite small at < 16k. It probably peaks at more than that during display draw.

If you can connect to the USB/serial port of the M5Stack you can see if the system process (marked with "*") is the one that is growing (it prints info when it does GC). Use toit serial monitor on the command line. If it's the system process you might consider using the open source Toit (see github.com/toitlang/jaguar ) which is more lightweight. But you won't get the console support in that case - Jaguar is currenly just wifi-based.

You could use print_histogram to get a list of the objects that are taking space in your process. This also prints its output on the USB/serial port.

[erikcorry]

display.remove_all should clear the scene just fine.

[AlexanderPilhar]

Okay, using toit serial monitor i get another line of information:

Out of memory due to heap fragmentation; restarting to attempt to recover.

I can't find print_histogram. Did you mean print_objects?

[AlexanderPilhar]

This is the output of serial_print_heap_report (first and last outputs):

Heap report:
  ┌───────────┬─────────┬───────────────────────┐
  │   Bytes   │  Count  │  Type                 │
  ├───────────┼─────────┼───────────────────────┤
  |    1624   |      2  |  external byte array  |
  |   18776   |     57  |  bignum               |
  |   86016   |     21  |  toit                 |
  |    4824   |     23  |  lwip                 |
  |    7072   |    578  |  heap overhead        |
  |    2656   |     44  |  event source         |
  |    9736   |    107  |  other threads        |
  |   27776   |     31  |  thread spawn         |
  |   24032   |    171  |  null tag             |
  |   31584   |     78  |  wifi                 |
  └───────────┴─────────┴───────────────────────┘
  Total: 214096 bytes in 534 allocations (85%)

...

Heap report @ out of memory:
  ┌───────────┬─────────┬───────────────────────┐
  │   Bytes   │  Count  │  Type                 │
  ├───────────┼─────────┼───────────────────────┤
  |    1600   |      2  |  external byte array  |
  |   18776   |     57  |  bignum               |
  |   81920   |     20  |  toit                 |
  |    8120   |     27  |  lwip                 |
  |    8224   |    713  |  heap overhead        |
  |    2656   |     44  |  event source         |
  |   37864   |    213  |  other threads        |
  |   27776   |     31  |  thread spawn         |
  |   24032   |    171  |  null tag             |
  |   31776   |     79  |  wifi                 |
  └───────────┴─────────┴───────────────────────┘
  Total: 242744 bytes in 644 allocations (97%)
encode_error_ primitive failed: EXCEPTION, ALLOCATION_FAILED

[AlexanderPilhar]

It seems I have solved this issue for now. Everytime I use display.remove_all I set all global references to null as well.

Thank you for your help @erikcorry !