search

espressif / esp-idf

Espressif IoT Development Framework. Official development framework for Espressif SoCs.
Apache License 2.0
13.71k stars 7.3k forks source link

[mesh] v4.4 MTXON task corrupting heap (IDFGH-9664) #11006

Closed KonssnoK closed 1 year ago

KonssnoK commented 1 year ago

Answers checklist.

IDF version.

v4.4.4-116-g00bb43ff24

Operating System used.

Windows

How did you build your project?

VS Code IDE

If you are using Windows, please specify command line type.

PowerShell

Development Kit.

ESP32S3-WROOM N8R2

Power Supply used.

USB

What is the expected behavior?

As reported in https://github.com/espressif/esp-idf/issues/10992

we are investigating crashes that happen to our devices.

We managed to reduce the triggering conditions to the following:

Please note that we don't know what other tasks MTXON works with, so there might be FREEs that are allocated by other tasks (maybe TCPIP)!

Here is the result:


W (16:07:12.957) mesh_main: <MESH_EVENT_TODS_REACHABLE>reachable:1
I (16:07:12.964) mesh_main: <MESH_EVENT_ROOT_ADDRESS>root address:7c:df:a1:e0:8b:7d
I (16:07:12.973) offline: object saved to PSRAM, free 490247
I (16:07:13.933) esp_netif_handlers: sta ip: 192.168.253.227, mask: 255.255.255.0, gw: 192.168.253.43
W (16:07:13.934) network: GOT IP from sta
I (16:07:13.938) mesh_netif: Interface AP
I (16:07:13.946) network: Network connected
I (16:07:13.948) mqtt_app: MQTT_EVENT_BEFORE_CONNECT
I (16:07:13.953) mqtt_app: Try [1] connection to mqtts://mqtt-stg.tiko.energy:8883
I (40932) wifi:<ba-add>idx:0 (ifx:0, d6:92:7d:c3:02:1e), tid:0, ssn:3, winSize:64
CORRUPT HEAP: Invalid data at 0x3dea2214. Expected 0xfefefefe got 0x0048fefe
CORRUPT HEAP: Invalid data at 0x3dea2218. Expected 0xfefefefe got 0xfefe0000
(456) heap_history[0] action HEAP_ENTER_MALLOC, task MTXON
(455) heap_history[1] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96E08-0x3DE96E38
(454) heap_history[2] action HEAP_ENTER_MALLOC, task MTXON
(453) heap_history[3] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96E48-0x3DE96E72
(452) heap_history[4] action HEAP_ENTER_FREE, task MTXON, address 0x3DE11BF4-0x00000000
(451) heap_history[5] action HEAP_EXIT_FREE, task MTXON
(450) heap_history[6] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(449) heap_history[7] action HEAP_EXIT_FREE, task MTXON
(448) heap_history[8] action HEAP_ENTER_FREE, task MTXON, address 0x3DE11BF4-0x00000000
(447) heap_history[9] action HEAP_EXIT_FREE, task MTXON
(446) heap_history[10] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(445) heap_history[11] action HEAP_EXIT_FREE, task MTXON
(444) heap_history[12] action HEAP_ENTER_MALLOC, task MTXON
(443) heap_history[13] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96CF4-0x3DE96D24
(442) heap_history[14] action HEAP_ENTER_MALLOC, task MTXON
(441) heap_history[15] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96D34-0x3DE96D5E
(440) heap_history[16] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(439) heap_history[17] action HEAP_EXIT_FREE, task MTXON
(438) heap_history[18] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(437) heap_history[19] action HEAP_EXIT_FREE, task MTXON
(436) heap_history[20] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(435) heap_history[21] action HEAP_EXIT_FREE, task MTXON
(434) heap_history[22] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(433) heap_history[23] action HEAP_EXIT_FREE, task MTXON
(432) heap_history[24] action HEAP_ENTER_MALLOC, task MTXON
(431) heap_history[25] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96CF4-0x3DE96D24
(430) heap_history[26] action HEAP_ENTER_MALLOC, task MTXON
(429) heap_history[27] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96D34-0x3DE96D5E
(428) heap_history[28] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(427) heap_history[29] action HEAP_EXIT_FREE, task MTXON
(426) heap_history[30] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(425) heap_history[31] action HEAP_EXIT_FREE, task MTXON
(424) heap_history[32] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(423) heap_history[33] action HEAP_EXIT_FREE, task MTXON
(422) heap_history[34] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(421) heap_history[35] action HEAP_EXIT_FREE, task MTXON
(420) heap_history[36] action HEAP_ENTER_MALLOC, task MTXON
(419) heap_history[37] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96CF4-0x3DE96D24
(418) heap_history[38] action HEAP_ENTER_MALLOC, task MTXON
(417) heap_history[39] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96D34-0x3DE96D5E
(416) heap_history[40] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(415) heap_history[41] action HEAP_EXIT_FREE, task MTXON
(414) heap_history[42] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(413) heap_history[43] action HEAP_EXIT_FREE, task MTXON
(412) heap_history[44] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96CF4-0x00000000
(411) heap_history[45] action HEAP_EXIT_FREE, task MTXON
(410) heap_history[46] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(409) heap_history[47] action HEAP_EXIT_FREE, task MTXON
(408) heap_history[48] action HEAP_ENTER_MALLOC, task MTXON
(407) heap_history[49] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96CF4-0x3DE96D24
(406) heap_history[50] action HEAP_ENTER_MALLOC, task MTXON
(405) heap_history[51] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE96D34-0x3DE96D5E
(404) heap_history[52] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(403) heap_history[53] action HEAP_EXIT_FREE, task MTXON
(402) heap_history[54] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(401) heap_history[55] action HEAP_EXIT_FREE, task MTXON
(400) heap_history[56] action HEAP_ENTER_FREE, task MTXON, address 0x3DE96E08-0x00000000
(399) heap_history[57] action HEAP_EXIT_FREE, task MTXON
(398) heap_history[58] action HEAP_ENTER_FREE, task MTXON, address 0x3DE12180-0x00000000
(397) heap_history[59] action HEAP_EXIT_FREE, task MTXON
(396) heap_history[60] action HEAP_ENTER_MALLOC, task MTXON
(395) heap_history[61] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE979E0-0x3DE97A10
(394) heap_history[62] action HEAP_ENTER_MALLOC, task MTXON
(393) heap_history[63] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE97A20-0x3DE97A4A
(392) heap_history[64] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(391) heap_history[65] action HEAP_EXIT_FREE, task MTXON
(390) heap_history[66] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(389) heap_history[67] action HEAP_EXIT_FREE, task MTXON
(388) heap_history[68] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(387) heap_history[69] action HEAP_EXIT_FREE, task MTXON
(386) heap_history[70] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(385) heap_history[71] action HEAP_EXIT_FREE, task MTXON
(384) heap_history[72] action HEAP_ENTER_MALLOC, task MTXON
(383) heap_history[73] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE978E4-0x3DE97914
(382) heap_history[74] action HEAP_ENTER_MALLOC, task MTXON
(381) heap_history[75] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE97924-0x3DE9794E
(380) heap_history[76] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(379) heap_history[77] action HEAP_EXIT_FREE, task MTXON
(378) heap_history[78] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(377) heap_history[79] action HEAP_EXIT_FREE, task MTXON
(376) heap_history[80] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(375) heap_history[81] action HEAP_EXIT_FREE, task MTXON
(374) heap_history[82] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(373) heap_history[83] action HEAP_EXIT_FREE, task MTXON
(372) heap_history[84] action HEAP_ENTER_MALLOC, task MTXON
(371) heap_history[85] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE978E4-0x3DE97914
(370) heap_history[86] action HEAP_ENTER_MALLOC, task MTXON
(369) heap_history[87] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE97924-0x3DE9794E
(368) heap_history[88] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(367) heap_history[89] action HEAP_EXIT_FREE, task MTXON
(366) heap_history[90] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(365) heap_history[91] action HEAP_EXIT_FREE, task MTXON
(364) heap_history[92] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97874-0x00000000
(363) heap_history[93] action HEAP_EXIT_FREE, task MTXON
(362) heap_history[94] action HEAP_ENTER_FREE, task MTXON, address 0x3DE97834-0x00000000
(361) heap_history[95] action HEAP_EXIT_FREE, task MTXON
(360) heap_history[96] action HEAP_ENTER_MALLOC, task MTXON
(359) heap_history[97] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1214-0x3DEA1244
(358) heap_history[98] action HEAP_ENTER_MALLOC, task MTXON
(357) heap_history[99] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1254-0x3DEA127E
(356) heap_history[100] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA11A4-0x00000000
(355) heap_history[101] action HEAP_EXIT_FREE, task MTXON
(354) heap_history[102] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA23BC-0x00000000
(353) heap_history[103] action HEAP_EXIT_FREE, task MTXON
(352) heap_history[104] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA22EC-0x00000000
(351) heap_history[105] action HEAP_EXIT_FREE, task MTXON
(350) heap_history[106] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA22AC-0x00000000
(349) heap_history[107] action HEAP_EXIT_FREE, task MTXON
(348) heap_history[108] action HEAP_ENTER_MALLOC, task MTXON
(347) heap_history[109] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA235C-0x3DEA238C
(346) heap_history[110] action HEAP_ENTER_MALLOC, task MTXON
(345) heap_history[111] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA239C-0x3DEA23C6
(344) heap_history[112] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA22EC-0x00000000
(343) heap_history[113] action HEAP_EXIT_FREE, task MTXON
(342) heap_history[114] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA22AC-0x00000000
(341) heap_history[115] action HEAP_EXIT_FREE, task MTXON
(340) heap_history[116] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1800-0x00000000
(339) heap_history[117] action HEAP_EXIT_FREE, task MTXON
(338) heap_history[118] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA22AC-0x00000000
(337) heap_history[119] action HEAP_EXIT_FREE, task MTXON
(336) heap_history[120] action HEAP_ENTER_MALLOC, task MTXON
(335) heap_history[121] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1F90-0x3DEA1FC0
(334) heap_history[122] action HEAP_ENTER_MALLOC, task MTXON
(333) heap_history[123] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1DA0-0x3DEA1DCA
(332) heap_history[124] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1E88-0x00000000
(331) heap_history[125] action HEAP_EXIT_FREE, task MTXON
(330) heap_history[126] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA23BC-0x00000000
(329) heap_history[127] action HEAP_EXIT_FREE, task MTXON
(328) heap_history[128] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1358-0x00000000
(327) heap_history[129] action HEAP_EXIT_FREE, task MTXON
(326) heap_history[130] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA274C-0x00000000
(325) heap_history[131] action HEAP_EXIT_FREE, task MTXON
(324) heap_history[132] action HEAP_ENTER_MALLOC, task MTXON
(323) heap_history[133] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0EBC-0x3DEA0EEC
(322) heap_history[134] action HEAP_ENTER_MALLOC, task MTXON
(321) heap_history[135] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA23BC-0x3DEA23E6
(320) heap_history[136] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA108C-0x00000000
(319) heap_history[137] action HEAP_EXIT_FREE, task MTXON
(318) heap_history[138] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FA4-0x00000000
(317) heap_history[139] action HEAP_EXIT_FREE, task MTXON
(316) heap_history[140] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1D00-0x00000000
(315) heap_history[141] action HEAP_EXIT_FREE, task MTXON
(314) heap_history[142] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0D80-0x00000000
(313) heap_history[143] action HEAP_EXIT_FREE, task MTXON
(312) heap_history[144] action HEAP_ENTER_MALLOC, task MTXON
(311) heap_history[145] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1F08-0x3DEA1F38
(310) heap_history[146] action HEAP_ENTER_MALLOC, task MTXON
(309) heap_history[147] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1F48-0x3DEA1F72
(308) heap_history[148] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0D00-0x00000000
(307) heap_history[149] action HEAP_EXIT_FREE, task MTXON
(306) heap_history[150] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA23BC-0x00000000
(305) heap_history[151] action HEAP_EXIT_FREE, task MTXON
(304) heap_history[152] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(303) heap_history[153] action HEAP_EXIT_FREE, task MTXON
(302) heap_history[154] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(301) heap_history[155] action HEAP_EXIT_FREE, task MTXON
(300) heap_history[156] action HEAP_ENTER_MALLOC, task MTXON
(299) heap_history[157] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0F3C-0x3DEA0F6C
(298) heap_history[158] action HEAP_ENTER_MALLOC, task MTXON
(297) heap_history[159] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0FE4-0x3DEA100E
(296) heap_history[160] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(295) heap_history[161] action HEAP_EXIT_FREE, task MTXON
(294) heap_history[162] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(293) heap_history[163] action HEAP_EXIT_FREE, task MTXON
(292) heap_history[164] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0F3C-0x00000000
(291) heap_history[165] action HEAP_EXIT_FREE, task MTXON
(290) heap_history[166] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(289) heap_history[167] action HEAP_EXIT_FREE, task MTXON
(288) heap_history[168] action HEAP_ENTER_MALLOC, task MTXON
(287) heap_history[169] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1024-0x3DEA1054
(286) heap_history[170] action HEAP_ENTER_MALLOC, task MTXON
(285) heap_history[171] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0EFC-0x3DEA0F26
(284) heap_history[172] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(283) heap_history[173] action HEAP_EXIT_FREE, task MTXON
(282) heap_history[174] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(281) heap_history[175] action HEAP_EXIT_FREE, task MTXON
(280) heap_history[176] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(279) heap_history[177] action HEAP_EXIT_FREE, task MTXON
(278) heap_history[178] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(277) heap_history[179] action HEAP_EXIT_FREE, task MTXON
(276) heap_history[180] action HEAP_ENTER_MALLOC, task MTXON
(275) heap_history[181] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1024-0x3DEA1054
(274) heap_history[182] action HEAP_ENTER_MALLOC, task MTXON
(273) heap_history[183] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0EFC-0x3DEA0F26
(272) heap_history[184] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(271) heap_history[185] action HEAP_EXIT_FREE, task MTXON
(270) heap_history[186] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(269) heap_history[187] action HEAP_EXIT_FREE, task MTXON
(268) heap_history[188] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(267) heap_history[189] action HEAP_EXIT_FREE, task MTXON
(266) heap_history[190] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(265) heap_history[191] action HEAP_EXIT_FREE, task MTXON
(264) heap_history[192] action HEAP_ENTER_MALLOC, task MTXON
(263) heap_history[193] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1024-0x3DEA1054
(262) heap_history[194] action HEAP_ENTER_MALLOC, task MTXON
(261) heap_history[195] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA16B0-0x3DEA16DA
(260) heap_history[196] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(259) heap_history[197] action HEAP_EXIT_FREE, task MTXON
(258) heap_history[198] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(257) heap_history[199] action HEAP_EXIT_FREE, task MTXON
(256) heap_history[200] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(255) heap_history[201] action HEAP_EXIT_FREE, task MTXON
(254) heap_history[202] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(253) heap_history[203] action HEAP_EXIT_FREE, task MTXON
(252) heap_history[204] action HEAP_ENTER_MALLOC, task MTXON
(251) heap_history[205] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1024-0x3DEA1054
(250) heap_history[206] action HEAP_ENTER_MALLOC, task MTXON
(249) heap_history[207] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0EFC-0x3DEA0F26
(248) heap_history[208] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(247) heap_history[209] action HEAP_EXIT_FREE, task MTXON
(246) heap_history[210] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(245) heap_history[211] action HEAP_EXIT_FREE, task MTXON
(244) heap_history[212] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0EFC-0x00000000
(243) heap_history[213] action HEAP_EXIT_FREE, task MTXON
(242) heap_history[214] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(241) heap_history[215] action HEAP_EXIT_FREE, task MTXON
(240) heap_history[216] action HEAP_ENTER_MALLOC, task MTXON
(239) heap_history[217] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1024-0x3DEA1054
(238) heap_history[218] action HEAP_ENTER_MALLOC, task MTXON
(237) heap_history[219] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA16B0-0x3DEA16DA
(236) heap_history[220] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(235) heap_history[221] action HEAP_EXIT_FREE, task MTXON
(234) heap_history[222] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(233) heap_history[223] action HEAP_EXIT_FREE, task MTXON
(232) heap_history[224] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(231) heap_history[225] action HEAP_EXIT_FREE, task MTXON
(230) heap_history[226] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(229) heap_history[227] action HEAP_EXIT_FREE, task MTXON
(228) heap_history[228] action HEAP_ENTER_MALLOC, task MTXON
(227) heap_history[229] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(226) heap_history[230] action HEAP_ENTER_MALLOC, task MTXON
(225) heap_history[231] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(224) heap_history[232] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(223) heap_history[233] action HEAP_EXIT_FREE, task MTXON
(222) heap_history[234] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(221) heap_history[235] action HEAP_EXIT_FREE, task MTXON
(220) heap_history[236] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(219) heap_history[237] action HEAP_EXIT_FREE, task MTXON
(218) heap_history[238] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(217) heap_history[239] action HEAP_EXIT_FREE, task MTXON
(216) heap_history[240] action HEAP_ENTER_MALLOC, task MTXON
(215) heap_history[241] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A4-0x3DEA22D4
(214) heap_history[242] action HEAP_ENTER_MALLOC, task MTXON
(213) heap_history[243] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22E4-0x3DEA230E
(212) heap_history[244] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(211) heap_history[245] action HEAP_EXIT_FREE, task MTXON
(210) heap_history[246] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(209) heap_history[247] action HEAP_EXIT_FREE, task MTXON
(208) heap_history[248] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(207) heap_history[249] action HEAP_EXIT_FREE, task MTXON
(206) heap_history[250] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(205) heap_history[251] action HEAP_EXIT_FREE, task MTXON
(204) heap_history[252] action HEAP_ENTER_MALLOC, task MTXON
(203) heap_history[253] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22CC-0x3DEA22FC
(202) heap_history[254] action HEAP_ENTER_MALLOC, task MTXON
(201) heap_history[255] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA230C-0x3DEA2336
(200) heap_history[256] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(199) heap_history[257] action HEAP_EXIT_FREE, task MTXON
(198) heap_history[258] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(197) heap_history[259] action HEAP_EXIT_FREE, task MTXON
(196) heap_history[260] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(195) heap_history[261] action HEAP_EXIT_FREE, task MTXON
(194) heap_history[262] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(193) heap_history[263] action HEAP_EXIT_FREE, task MTXON
(192) heap_history[264] action HEAP_ENTER_MALLOC, task MTXON
(191) heap_history[265] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE9E914-0x3DE9E944
(190) heap_history[266] action HEAP_ENTER_MALLOC, task MTXON
(189) heap_history[267] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0FE4-0x3DEA100E
(188) heap_history[268] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E8A4-0x00000000
(187) heap_history[269] action HEAP_EXIT_FREE, task MTXON
(186) heap_history[270] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(185) heap_history[271] action HEAP_EXIT_FREE, task MTXON
(184) heap_history[272] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(183) heap_history[273] action HEAP_EXIT_FREE, task MTXON
(182) heap_history[274] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA0FE4-0x00000000
(181) heap_history[275] action HEAP_EXIT_FREE, task MTXON
(180) heap_history[276] action HEAP_ENTER_MALLOC, task MTXON
(179) heap_history[277] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA21F0-0x3DEA2220
(178) heap_history[278] action HEAP_ENTER_MALLOC, task MTXON
(177) heap_history[279] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA16B0-0x3DEA16DA
(176) heap_history[280] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(175) heap_history[281] action HEAP_EXIT_FREE, task MTXON
(174) heap_history[282] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(173) heap_history[283] action HEAP_EXIT_FREE, task MTXON
(172) heap_history[284] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(171) heap_history[285] action HEAP_EXIT_FREE, task MTXON
(170) heap_history[286] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(169) heap_history[287] action HEAP_EXIT_FREE, task MTXON
(168) heap_history[288] action HEAP_ENTER_MALLOC, task MTXON
(167) heap_history[289] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DE9E92C-0x3DE9E95C
(166) heap_history[290] action HEAP_ENTER_MALLOC, task MTXON
(165) heap_history[291] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA1030-0x3DEA105A
(164) heap_history[292] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E8A4-0x00000000
(163) heap_history[293] action HEAP_EXIT_FREE, task MTXON
(162) heap_history[294] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(161) heap_history[295] action HEAP_EXIT_FREE, task MTXON
(160) heap_history[296] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(159) heap_history[297] action HEAP_EXIT_FREE, task MTXON
(158) heap_history[298] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(157) heap_history[299] action HEAP_EXIT_FREE, task MTXON
(156) heap_history[300] action HEAP_ENTER_MALLOC, task MTXON
(155) heap_history[301] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(154) heap_history[302] action HEAP_ENTER_MALLOC, task MTXON
(153) heap_history[303] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(152) heap_history[304] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(151) heap_history[305] action HEAP_EXIT_FREE, task MTXON
(150) heap_history[306] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(149) heap_history[307] action HEAP_EXIT_FREE, task MTXON
(148) heap_history[308] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(147) heap_history[309] action HEAP_EXIT_FREE, task MTXON
(146) heap_history[310] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(145) heap_history[311] action HEAP_EXIT_FREE, task MTXON
(144) heap_history[312] action HEAP_ENTER_MALLOC, task MTXON
(143) heap_history[313] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A4-0x3DEA22D4
(142) heap_history[314] action HEAP_ENTER_MALLOC, task MTXON
(141) heap_history[315] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22E4-0x3DEA230E
(140) heap_history[316] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(139) heap_history[317] action HEAP_EXIT_FREE, task MTXON
(138) heap_history[318] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(137) heap_history[319] action HEAP_EXIT_FREE, task MTXON
(136) heap_history[320] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(135) heap_history[321] action HEAP_EXIT_FREE, task MTXON
(134) heap_history[322] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(133) heap_history[323] action HEAP_EXIT_FREE, task MTXON
(132) heap_history[324] action HEAP_ENTER_MALLOC, task MTXON
(131) heap_history[325] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA21F0-0x3DEA2220
(130) heap_history[326] action HEAP_ENTER_MALLOC, task MTXON
(129) heap_history[327] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA16B0-0x3DEA16DA
(128) heap_history[328] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(127) heap_history[329] action HEAP_EXIT_FREE, task MTXON
(126) heap_history[330] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(125) heap_history[331] action HEAP_EXIT_FREE, task MTXON
(124) heap_history[332] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(123) heap_history[333] action HEAP_EXIT_FREE, task MTXON
(122) heap_history[334] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(121) heap_history[335] action HEAP_EXIT_FREE, task MTXON
(120) heap_history[336] action HEAP_ENTER_MALLOC, task MTXON
(119) heap_history[337] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(118) heap_history[338] action HEAP_ENTER_MALLOC, task MTXON
(117) heap_history[339] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(116) heap_history[340] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(115) heap_history[341] action HEAP_EXIT_FREE, task MTXON
(114) heap_history[342] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(113) heap_history[343] action HEAP_EXIT_FREE, task MTXON
(112) heap_history[344] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(111) heap_history[345] action HEAP_EXIT_FREE, task MTXON
(110) heap_history[346] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(109) heap_history[347] action HEAP_EXIT_FREE, task MTXON
(108) heap_history[348] action HEAP_ENTER_MALLOC, task MTXON
(107) heap_history[349] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA21F0-0x3DEA2220
(106) heap_history[350] action HEAP_ENTER_MALLOC, task MTXON
(105) heap_history[351] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2230-0x3DEA225A
(104) heap_history[352] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(103) heap_history[353] action HEAP_EXIT_FREE, task MTXON
(102) heap_history[354] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(101) heap_history[355] action HEAP_EXIT_FREE, task MTXON
(100) heap_history[356] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(99) heap_history[357] action HEAP_EXIT_FREE, task MTXON
(98) heap_history[358] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(97) heap_history[359] action HEAP_EXIT_FREE, task MTXON
(96) heap_history[360] action HEAP_ENTER_MALLOC, task MTXON
(95) heap_history[361] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A4-0x3DEA22D4
(94) heap_history[362] action HEAP_ENTER_MALLOC, task MTXON
(93) heap_history[363] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22E4-0x3DEA230E
(92) heap_history[364] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(91) heap_history[365] action HEAP_EXIT_FREE, task MTXON
(90) heap_history[366] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(89) heap_history[367] action HEAP_EXIT_FREE, task MTXON
(88) heap_history[368] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(87) heap_history[369] action HEAP_EXIT_FREE, task MTXON
(86) heap_history[370] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(85) heap_history[371] action HEAP_EXIT_FREE, task MTXON
(84) heap_history[372] action HEAP_ENTER_MALLOC, task MTXON
(83) heap_history[373] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(82) heap_history[374] action HEAP_ENTER_MALLOC, task MTXON
(81) heap_history[375] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(80) heap_history[376] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(79) heap_history[377] action HEAP_EXIT_FREE, task MTXON
(78) heap_history[378] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(77) heap_history[379] action HEAP_EXIT_FREE, task MTXON
(76) heap_history[380] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(75) heap_history[381] action HEAP_EXIT_FREE, task MTXON
(74) heap_history[382] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(73) heap_history[383] action HEAP_EXIT_FREE, task MTXON
(72) heap_history[384] action HEAP_ENTER_MALLOC, task MTXON
(71) heap_history[385] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA21F0-0x3DEA2220
(70) heap_history[386] action HEAP_ENTER_MALLOC, task MTXON
(69) heap_history[387] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA16B0-0x3DEA16DA
(68) heap_history[388] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(67) heap_history[389] action HEAP_EXIT_FREE, task MTXON
(66) heap_history[390] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(65) heap_history[391] action HEAP_EXIT_FREE, task MTXON
(64) heap_history[392] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(63) heap_history[393] action HEAP_EXIT_FREE, task MTXON
(62) heap_history[394] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(61) heap_history[395] action HEAP_EXIT_FREE, task MTXON
(60) heap_history[396] action HEAP_ENTER_MALLOC, task MTXON
(59) heap_history[397] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(58) heap_history[398] action HEAP_ENTER_MALLOC, task MTXON
(57) heap_history[399] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(56) heap_history[400] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(55) heap_history[401] action HEAP_EXIT_FREE, task MTXON
(54) heap_history[402] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(53) heap_history[403] action HEAP_EXIT_FREE, task MTXON
(52) heap_history[404] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(51) heap_history[405] action HEAP_EXIT_FREE, task MTXON
(50) heap_history[406] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(49) heap_history[407] action HEAP_EXIT_FREE, task MTXON
(48) heap_history[408] action HEAP_ENTER_MALLOC, task MTXON
(47) heap_history[409] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA21F0-0x3DEA2220
(46) heap_history[410] action HEAP_ENTER_MALLOC, task MTXON
(45) heap_history[411] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2230-0x3DEA225A
(44) heap_history[412] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(43) heap_history[413] action HEAP_EXIT_FREE, task MTXON
(42) heap_history[414] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(41) heap_history[415] action HEAP_EXIT_FREE, task MTXON
(40) heap_history[416] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1224-0x00000000
(39) heap_history[417] action HEAP_EXIT_FREE, task MTXON
(38) heap_history[418] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(37) heap_history[419] action HEAP_EXIT_FREE, task MTXON
(36) heap_history[420] action HEAP_ENTER_MALLOC, task MTXON
(35) heap_history[421] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A4-0x3DEA22D4
(34) heap_history[422] action HEAP_ENTER_MALLOC, task MTXON
(33) heap_history[423] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22E4-0x3DEA230E
(32) heap_history[424] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(31) heap_history[425] action HEAP_EXIT_FREE, task MTXON
(30) heap_history[426] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(29) heap_history[427] action HEAP_EXIT_FREE, task MTXON
(28) heap_history[428] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E11C-0x00000000
(27) heap_history[429] action HEAP_EXIT_FREE, task MTXON
(26) heap_history[430] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E864-0x00000000
(25) heap_history[431] action HEAP_EXIT_FREE, task MTXON
(24) heap_history[432] action HEAP_ENTER_MALLOC, task MTXON
(23) heap_history[433] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(22) heap_history[434] action HEAP_ENTER_MALLOC, task MTXON
(21) heap_history[435] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(20) heap_history[436] action HEAP_ENTER_MALLOC, task MTXON
(19) heap_history[437] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(18) heap_history[438] action HEAP_ENTER_MALLOC, task MTXON
(17) heap_history[439] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(16) heap_history[440] action HEAP_ENTER_MALLOC, task MTXON
(15) heap_history[441] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(14) heap_history[442] action HEAP_ENTER_MALLOC, task MTXON
(13) heap_history[443] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(12) heap_history[444] action HEAP_ENTER_MALLOC, task MTXON
(11) heap_history[445] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA2290
(10) heap_history[446] action HEAP_ENTER_MALLOC, task MTXON
(9) heap_history[447] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA22A0-0x3DEA22CA
(8) heap_history[448] action HEAP_ENTER_MALLOC, task MTXON
(7) heap_history[449] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA0FE4-0x3DEA1014
(6) heap_history[450] action HEAP_ENTER_MALLOC, task MTXON
(5) heap_history[451] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2260-0x3DEA228A
(4) heap_history[452] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA21F0-0x00000000
(3) heap_history[453] action HEAP_EXIT_FREE, task MTXON
(2) heap_history[454] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA1030-0x00000000
(1) heap_history[455] action HEAP_EXIT_FREE, task MTXON

assert failed: multi_heap_malloc multi_heap_poisoning.c:237 (ret)

Backtrace: 0x403760e2:0x3fcd6fb0 0x40382323:0x3fcd6fd0 0x4038aa91:0x3fcd6ff0 0x40389bde:0x3fcd7110 0x403764ec:0x3fcd7130 0x4037654b:0x3fcd7150 0x403769ed:0x3fcd7170 0x4037dc88:0x3fcd7190 0x420e7535:0x3fcd71b0 0x420db341:0x3fcd71d0 0x420db3bd:0x3fcd71f0 0x420d7573:0x3fcd7210 0x420d61da:0x3fcd7250 0x420d666f:0x3fcd7270 0x420d6a33:0x3fcd72e0 0x420d7c93:0x3fcd7310 0x4216409d:0x3fcd7330 0x42095924:0x3fcd7350 0x40385d8a:0x3fcd7390
KonssnoK commented 1 year ago

We confirm it's the TCPIP the other component in this crashes.

    Line    2: CORRUPT HEAP: Invalid data at 0x3de9e130. Expected 0xfefefefe got 0x008cfefe
    Line    3: CORRUPT HEAP: Invalid data at 0x3de9e134. Expected 0xfefefefe got 0xfefe0000
    Line   59: (969) heap_history[1003] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E609
    Line   60: (968) heap_history[1004] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  213: (815) heap_history[133] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E609
    Line  214: (814) heap_history[134] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  367: (661) heap_history[287] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E609
    Line  368: (660) heap_history[288] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  541: (487) heap_history[461] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E5C8
    Line  546: (482) heap_history[466] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  697: (331) heap_history[617] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E5C8
    Line  702: (326) heap_history[622] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  763: (265) heap_history[683] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E5C8
    Line  768: (260) heap_history[688] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  789: (239) heap_history[709] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E5C8
    Line  790: (238) heap_history[710] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  807: (221) heap_history[727] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E5C8
    Line  812: (216) heap_history[732] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  895: (133) heap_history[815] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E6F4
    Line  896: (132) heap_history[816] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000
    Line  987: (41) heap_history[907] action HEAP_EXIT_MALLOC, task tiT, address 0x3DE9E10C-0x3DE9E6F4
    Line  996: (32) heap_history[916] action HEAP_ENTER_FREE, task MTXON, address 0x3DE9E10C-0x00000000

heap_corrupt.txt

Considering the size of the mallocs, i suppose this is a pbuf allocation from lwip

10C is the first address invalid data at 130-134 with data 0000008C

I suspect a packet size or some flags

chipweinberger commented 1 year ago

That's an interest approach to debugging heap corruption.

I usually:

Perhaps you can try this approach to narrow things down further.

KonssnoK commented 1 year ago

That's an interest approach to debugging heap corruption.

I usually:

* enable corruption detection

* add lots of calls to heap_caps_check_integrity()

* start commenting out code until I know which code is causing the corruption

Perhaps you can try this approach to narrow things down further.

we'll refine our investigation on this once we have more resources available. Problem is, the mesh code is closed sourced and all our done steps lead to those modules.

On top, there are pending changes to the mesh library following our highest priority ticket, https://github.com/espressif/esp-idf/issues/9955

zhangyanjiaoesp commented 1 year ago

@KonssnoK Can you provide a demo to reproduce this issue?

KonssnoK commented 1 year ago

i think we can try to modify our mesh sample to make it crash! will update when we have sometihng

KonssnoK commented 1 year ago

@zhangyanjiaoesp please find the code at

https://github.com/KonssnoK/esp-idf/tree/bug/mtxon-heap-corruption/examples/mesh/ip_internal_network

To have an easy crash just put it on 3 devices and let it run a bit. Please notice that you have to disable watchdogs, i don't know how to put a "disable" in the sdkconfig.defaults, so i put

# CONFIG_TASK_WDT_CHECK_IDLE_TASK_CPU0 is not set
# CONFIG_TASK_WDT_CHECK_IDLE_TASK_CPU1 is not set
KonssnoK commented 1 year ago

the more the devices, the easier the crash example: device_7.txt device_13.txt device_5.txt

I don't know why but in these logs the names of the tasks are wrong, in the terminal they are displayed correctly :)

Please note that this issue is impacting our firmware update capabilities...

KonssnoK commented 1 year ago

just to complete the reporting, apparently it's possible to trigger more heap corruption crashes by simply enabling comprehensive heap poisoning in the example

https://github.com/KonssnoK/esp-idf/tree/bug/mesh_wifi/examples/mesh/ip_internal_network

used for https://github.com/espressif/esp-idf/issues/9955

One way to make the child device crash is to keep the hotspot off apparently


I (07:15:59.609) mesh_main: <MESH_EVENT_PARENT_CONNECTED>layer:0-->2, parent:7c:df:a1:e0:8b:7d<layer2>, ID:77:77:77:77:77:76
I (07:15:59.621) mesh_netif: It was a wifi station removing stuff
I (11487) wifi:<ba-add>idx:0 (ifx:0, 7c:df:a1:e0:8b:7d), tid:5, ssn:0, winSize:64
I (11497) wifi:AP's beacon interval = 102400 us, DTIM period = 1
E (07:16:00.675) mesh_netif: Send with err code 16394 ESP_ERR_MESH_TIMEOUT
W (07:16:00.676) mesh_main: <MESH_EVENT_TODS_REACHABLE>state:1
CORRUPT HEAP: Invalid data at 0x3fcf3edc. Expected 0xfefefefe got 0x0000fefe
CORRUPT HEAP: Invalid data at 0x3fcf3ee0. Expected 0xfefefefe got 0xfefe0000

assert failed: multi_heap_malloc multi_heap_poisoning.c:241 (ret)

Backtrace: 0x40375ace:0x3fcafaa0 0x4037c0c9:0x3fcafac0 0x40382d61:0x3fcafae0 0x40381e94:0x3fcafc00 0x40375dc9:0x3fcafc20 0x40375e29:0x3fcafc40 0x40375e66:0x3fcafc60 0x40382d71:0x3fcafc80 0x40378bb1:0x3fcafca0 0x4206fcf3:0x3fcafcc0 0x4206fd50:0x3fcafce0 0x4204e054:0x3fcafd00 0x420675d9:0x3fcafd50 0x42055aae:0x3fcafe30 0x4206faaf:0x3fcafe60 0x4206fb79:0x3fcafe80 0x4205ee2d:0x3fcafeb0 0x4037f3f6:0x3fcb01e0
xtensa-esp32s3-elf-addr2line -pfiaC -e c:\src\esp-idf\examples\mesh\ip_internal_network\build\ip_internal_network.elf 0x40375ace:
KonssnoK commented 1 year ago

@zhangyanjiaoesp can you give us an update? are you able to reproduce the issue? thanks :)

zhangyanjiaoesp commented 1 year ago

@KonssnoK Yes, I can reproduce this issue according to your branch. And I'm debugging on it. (By the way, the changes in mesh lib has been merged into v4.4, 90d6e45d9ffc6c060d664d50063fe05c712d3042)

KonssnoK commented 1 year ago

oh, ok! thanks, i'll check back in a couple of days as usual :)

zRedShift commented 1 year ago

We hit this (I assume) exact issue in v5.0.1, when continuously uploading a sizable (about 200 KiB/s) amount of data to the server over TCP for a period of hours to days until we get a panic_abort and a reboot.

assert failed: tcp_free_acked_segments /IDF/components/lwip/lwip/src/core/tcp_in.c:1138 (tcp_receive: valid queue length)

Backtrace: 0x4037b406:0x3fcd3fc0 0x40382935:0x3fcd3fe0 0x40389095:0x3fcd4000 0x420dcc99:0x3fcd4120 0x420dcf18:0x3fcd4140 0x420de0b5:0x3fcd4170 0x420e2c1a:0x3fcd41e0 0x420e675e:0x3fcd4210 0x420d8765:0x3fcd4230
0x4037b406 - panic_abort
    at /home/ronen/.espressif/esp-idf/master/components/esp_system/panic.c:452
0x3fcd3fc0 - _nimble_bss_end
    at ??:??
0x40382935 - esp_system_abort
    at /home/ronen/.espressif/esp-idf/master/components/esp_system/port/esp_system_chip.c:77
0x3fcd3fe0 - _nimble_bss_end
    at ??:??
0x40389095 - __assert_func
    at /home/ronen/.espressif/esp-idf/master/components/newlib/assert.c:81
0x3fcd4000 - _nimble_bss_end
    at ??:??
0x420dcc99 - tcp_free_acked_segments
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/core/tcp_in.c:1138
0x3fcd4120 - _nimble_bss_end
    at ??:??
0x420dcf18 - tcp_receive
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/core/tcp_in.c:1301
0x3fcd4140 - _nimble_bss_end
    at ??:??
0x420de0b5 - tcp_process
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/core/tcp_in.c:996
0x3fcd4170 - _nimble_bss_end
    at ??:??
0x420e2c1a - ip4_input
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/core/ipv4/ip4.c:749
0x3fcd41e0 - _nimble_bss_end
    at ??:??
0x420e675e - ethernet_input
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/netif/ethernet.c:186
0x3fcd4210 - _nimble_bss_end
    at ??:??
0x420d8765 - tcpip_thread_handle_msg
    at /home/ronen/.espressif/esp-idf/master/components/lwip/lwip/src/api/tcpip.c:174
0x3fcd4230 - _nimble_bss_end
    at ??:??

That's with poisoning disabled. With it enabled it trigger more often in multi_heap_malloc as posted earlier, and various other place like TLSF code, etc.

We thought that it was a PSRAM corruption issue for a while, so we enabled ECC there but it didn't help.

Some configurations which might be related that we use are

CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
...
CONFIG_LWIP_TCP_SND_BUF_DEFAULT=64800
CONFIG_LWIP_TCP_WND_DEFAULT=5744
CONFIG_LWIP_TCP_RECVMBOX_SIZE=64
CONFIG_LWIP_TCPIP_RECVMBOX_SIZE=64

and in another case:

CONFIG_SPIRAM_TRY_ALLOCATE_WIFI_LWIP=y
...
CONFIG_LWIP_TCP_SND_BUF_DEFAULT=118000
CONFIG_LWIP_TCP_WND_DEFAULT=5744
CONFIG_LWIP_TCP_RECVMBOX_SIZE=64
CONFIG_LWIP_TCPIP_RECVMBOX_SIZE=64
CONFIG_LWIP_WND_SCALE=y

Is there a lead on the horizon? I'm willing to patch our esp-idf with an experimental fix, including a different revision of esp-wifi if it's localized there.

KonssnoK commented 1 year ago

@zRedShift are you using a mesh network or simple wifi? In our case it's likely an interaction between tcpip task and the MTXON task, which is part of the mesh library.

More generally speaking, it seems that in general this is happening when someone writes in a already released TCP packet.

zRedShift commented 1 year ago

@KonssnoK No, we're using it as a STA connected to a configurable AP, standard usage. It was the assumption that it's a UAF of TCP/IP buffer, that would be the most likely culprit. We also use BLE Coex and APSTA during provisioning but it still occurs without them.

Specifically, we're using esp_http_client with mbedtls to upload massive files generated in real-time.

KonssnoK commented 1 year ago

@zRedShift we also use mbedtls, but our setup is using the wifi mesh environment. BLE is off for us after provisioning.

When i have some time i'll try to understand if i can do some tests to see if the issue is in an open part of the code.

The possible involved modules i see are:

zhangyanjiaoesp commented 1 year ago

@KonssnoK I'm still debugging on this issue, will feedback to you ASAP when I find the root cause.

KonssnoK commented 1 year ago

@zhangyanjiaoesp did you manage to understand which component is triggering it? if it's an open sourced one we can focus on it too.

zhangyanjiaoesp commented 1 year ago

@KonssnoK Please use the new wifi lib to test, the corrupt issue has been solved on my side. wifi_lib_0404.zip wifi firmware version: 7fec53d If your test is passed, then we will merged the fix into release/v4.4 ASAP.

KonssnoK commented 1 year ago

thanks @zhangyanjiaoesp , we'll test asap

KonssnoK commented 1 year ago

just a question, what is it built on top of? 15b1309726f600122b5d4539e73758ae1852168d ?

zhangyanjiaoesp commented 1 year ago

@KonssnoK The wifi lib is base on your branch https://github.com/KonssnoK/esp-idf/tree/bug/mtxon-heap-corruption/examples/mesh/ip_internal_network, and it contains all the fixes from https://github.com/espressif/esp-idf/issues/9955. If you want to update your release/v4.4 version, you can base on https://github.com/espressif/esp-idf/commit/90d6e45d9ffc6c060d664d50063fe05c712d3042.

KonssnoK commented 1 year ago

@zhangyanjiaoesp we saw there were additional fixes to the wifi library, so we are currently aiming at 4c7d97e2bdbd26b1ad6adc6de8051888e1feec10

First though, let's check if everything is ok with the current branch

KonssnoK commented 1 year ago

@zhangyanjiaoesp we continue to test but currently you have the green light for merging, we are not able to reproduce the issue with the same examples. Thanks :)

zRedShift commented 1 year ago

Would this fix affect v5.x versions of esp-idf? And is there a possibility that it solves a non-mesh related esp-wifi bug I mentioned here: https://github.com/espressif/esp-idf/issues/11006#issuecomment-1487529101?

KonssnoK commented 1 year ago

Would this fix affect v5.x versions of esp-idf? And is there a possibility that it solves a non-mesh related esp-wifi bug I mentioned here: #11006 (comment)?

these fixes are usually backported everywhere (still maintained), so it will be pushed also to 5.0 for sure. The 4.4 push is focused just because our product line works on v4.4 :)

KonssnoK commented 1 year ago

@zhangyanjiaoesp we were able to reproduce again one crash, we'll try now to isolate and reproduce consistently 

W (10:10:57.634) stats: iteration_time_us=1000004 Name=last% IDLE=40 IDLE=63 task_mesh_rx=1 tiT=4 ipc1=24 wifi=5 updater=56 MTXON=1
Guru Meditation Error: Core  1 panic'ed (StoreProhibited). Exception was unhandled.

Core  1 register dump:
PC      : 0x40389eb2  PS      : 0x00060e33  A0      : 0x8038a678  A1      : 0x3fcb7370
0x40389eb2: remove_free_block at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:207
 (inlined by) block_locate_free at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:442
 (inlined by) tlsf_malloc at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:849

A2      : 0x3d800014  A3      : 0x0000010c  A4      : 0x3d896e6c  A5      : 0x3d8a8dd4
A6      : 0x3d8112c4  A7      : 0x3d8a8d44  A8      : 0x3d8a8e24  A9      : 0x00000013
A10     : 0x964c06f3  A11     : 0x00000004  A12     : 0xfffffffc  A13     : 0x00007001
A14     : 0x00060e20  A15     : 0x00000001  SAR     : 0x0000001d  EXCCAUSE: 0x0000001d
EXCVADDR: 0x0000700d  LBEG    : 0x400570e8  LEND    : 0x400570f3  LCOUNT  : 0x00000000

Backtrace: 0x40389eaf:0x3fcb7370 0x4038a675:0x3fcb7390 0x40376709:0x3fcb73b0 0x4037685b:0x3fcb73d0 0x403763f9:0x3fcb7420 0x421156f7:0x3fcb7440 0x42115754:0x3fcb7460 0x42116215:0x3fcb7480 0x4038b149:0x3fcb7640 0x4038b235:0x3fcb7690 0x4003f4bd:0x3fcb76b0 |<-CORRUPTED
0x40389eaf: remove_free_block at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:206
 (inlined by) block_locate_free at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:442
 (inlined by) tlsf_malloc at C:/repos/v3/esp-idf/components/heap/heap_tlsf.c:849

0x4038a675: multi_heap_malloc_impl at C:/repos/v3/esp-idf/components/heap/multi_heap.c:187

0x40376709: heap_caps_malloc_base at C:/repos/v3/esp-idf/components/heap/heap_caps.c:175

0x4037685b: heap_caps_malloc_prefer at C:/repos/v3/esp-idf/components/heap/heap_caps.c:290

0x403763f9: wifi_malloc at C:/repos/v3/esp-idf/components/esp_wifi/esp32s3/esp_adapter.c:71

0x421156f7: mesh_malloc at ??:?

0x42115754: esp_mesh_create_context at ??:?

0x42116215: esp_mesh_wifi_recv_cb at ??:?

0x4038b149: sta_input at ??:?

0x4038b235: sta_rx_cb at ??:?
KonssnoK commented 1 year ago

Generically speaking the stability of the system has improved thousands of folds. We still get some crashes, which seem very similar to what was happening before. @zhangyanjiaoesp maybe you can check for similar code (to the one you fixed) in the libraries?

To trigger them, instead of doing high traffic, we do parallel firmware updates.

Here an example :

    Line 1609: CORRUPT HEAP: Invalid data at 0x3dea2dbc. Expected 0xfefefefe got 0x3de00018
    Line 2290: (344) heap_history[738] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2CBC-0x3DEA2D4F
    Line 2334: (300) heap_history[782] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2D3C-0x3DEA2D6C
    Line 2336: (298) heap_history[784] action HEAP_EXIT_MALLOC, task MTXON, address 0x3DEA2D80-0x3DEA2DAA
    Line 2337: (297) heap_history[785] action HEAP_ENTER_FREE, task MTX, address 0x3DEA2D80-0x00000000
    Line 2339: (295) heap_history[787] action HEAP_ENTER_FREE, task MTX, address 0x3DEA2D3C-0x00000000
    Line 2341: (293) heap_history[789] action HEAP_ENTER_FREE, task MRX, address 0x3DEA2D80-0x00000000
    Line 2343: (291) heap_history[791] action HEAP_ENTER_FREE, task MRX, address 0x3DEA2D3C-0x00000000
    Line 2353: (281) heap_history[801] action HEAP_ENTER_FREE, task task_mesh_rx, address 0x3DEA2D3C-0x00000000
    Line 2384: (250) heap_history[832] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2401: (233) heap_history[849] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2414: (220) heap_history[862] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2435: (199) heap_history[883] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2448: (186) heap_history[896] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2449: (185) heap_history[897] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2464: (170) heap_history[912] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2477: (157) heap_history[925] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2500: (134) heap_history[948] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2509: (125) heap_history[957] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2530: (104) heap_history[978] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2543: (91) heap_history[991] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2566: (68) heap_history[1014] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2567: (67) heap_history[1015] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2596: (38) heap_history[20] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2609: (25) heap_history[33] action HEAP_ENTER_FREE, task MTXON, address 0x3DEA2D3C-0x00000000
    Line 2632: (2) heap_history[56] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEA2D3C-0x3DEA2D9C
    Line 2835: CORRUPT HEAP: Bad head at 0x3deaf2cc. Expected 0xabba1234 got 0x3de00018
    Line 3508: (352) heap_history[17] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEAEDD8-0x3DEAF2EB
    Line 3732: (128) heap_history[241] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEAEDD8-0x3DEAF230
    Line 3738: (122) heap_history[247] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEAF2EC-0x3DEAF31C
    Line 3763: (97) heap_history[272] action HEAP_ENTER_FREE, task MTX, address 0x3DEAF2EC-0x00000000
    Line 3812: (48) heap_history[321] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEAEDD8-0x3DEAF230
    Line 3818: (42) heap_history[327] action HEAP_EXIT_MALLOC, task tiT, address 0x3DEAF2D8-0x3DEAF308
    Line 3851: (9) heap_history[360] action HEAP_ENTER_FREE, task MTX, address 0x3DEAF2D8-0x00000000
0x4038aeb9: free at C:/src/esp-idf/components/newlib/heap.c:39

0x42115543: mesh_free at ??:?

0x421155f1: esp_mesh_free_context at ??:?

0x42112368: esp_mesh_discard_context at ??:?

0x42113225: mesh_tx_task_main at ??:?

0x4038668a: vPortTaskWrapper at C:/src/esp-idf/components/freertos/port/xtensa/port.c:142

device1.txt

zhangyanjiaoesp commented 1 year ago

@KonssnoK I will check the code

KonssnoK commented 1 year ago

@zhangyanjiaoesp would it be possible to get the fix on top of 4c7d97e2bdbd26b1ad6adc6de8051888e1feec10 ? This way we can check also latest wifi fixes on the field, since we have devices isolating themselves (still investigating on why)

zhangyanjiaoesp commented 1 year ago

@KonssnoK the wifi lib based on https://github.com/espressif/esp-idf/commit/4c7d97e2bdbd26b1ad6adc6de8051888e1feec10 is here: wifi_lib_0406.zip wifi firmware version: 85cf6e5

To trigger them, instead of doing high traffic, we do parallel firmware updates. 3 devices trigger a firmware update on all 3 devices at the same time be lucky ( 100 ) ... triggering crashes is now quite difficult. We are investigating in order to replicate these crashes.

This issue is still debugging.

zhangyanjiaoesp commented 1 year ago

@zRedShift Since your issue is not related to wifi mesh, please create a new ticket to report your issue

KonssnoK commented 1 year ago

@KonssnoK the wifi lib based on 4c7d97e is here: wifi_lib_0406.zip wifi firmware version: 85cf6e5

To trigger them, instead of doing high traffic, we do parallel firmware updates. 3 devices trigger a firmware update on all 3 devices at the same time be lucky ( 100 ) ... triggering crashes is now quite difficult. We are investigating in order to replicate these crashes.

This issue is still debugging.

meaning you were able to replicate some other crashes? We are still trying to understand how to trigger them. The new library much more stable!

zhangyanjiaoesp commented 1 year ago

@KonssnoK No, I didn't reproduce any crash issue, just look at your logs for some ideas. If you find a good way to reproduce the problem, it will speed up the debug process.

zhangyanjiaoesp commented 1 year ago

@KonssnoK Since the fix has solved the original crash issue, and the new crash issue is hard to reproduce, we will merge the fix into release/v4.4 firstly. Is it OK?

KonssnoK commented 1 year ago

@zhangyanjiaoesp that would be great, so we can move inside esp-idf without having to change wifi-lib with a custom one

KonssnoK commented 1 year ago

i will close this ticket and open a new one when we manage to replicate other crashes