"Verbose" mode for memap.py

javier-moreno-tridonic-com commented 8 years ago

Note: This is just a template, so feel free to use/remove the unnecessary things

Description

Type: Feature request Priority: Minor

Feature Request

To analyse memory usage and identify possible optimisation points, I've been modifying memap.py in the following manner:

original code

    @staticmethod
    def path_object_to_module_name(txt):
        """ Parse a path to object file to extract it's module and object data
        Positional arguments:
        txt - the path to parse the object and module name from
        """

        txt = txt.replace('\\', '/')
        rex_mbed_os_name = r'^.+mbed-os\/(.+)\/(.+\.o)$'
        test_rex_mbed_os_name = re.match(rex_mbed_os_name, txt)

        if test_rex_mbed_os_name:

            object_name = test_rex_mbed_os_name.group(2)
            data = test_rex_mbed_os_name.group(1).split('/')
            ndata = len(data)

            if ndata == 1:
                module_name = data[0]
            else:
                module_name = data[0] + '/' + data[1]

            return [module_name, object_name]
        else:
            return ['Misc', ""]

My code

    @staticmethod
    def path_object_to_module_name(txt):
        """ Parse a path to object file to extract it's module and object data

        Positional arguments:
        txt - the path to parse the object and module name from
        """

        txt = txt.replace('\\', '/')

        # print('txt: ' + txt)

        rex_mbed_os_name = r'^.+mbed-os\/(.+)\/(.+\.o)$'
        test_rex_mbed_os_name = re.match(rex_mbed_os_name, txt)

        if test_rex_mbed_os_name:
            object_name = test_rex_mbed_os_name.group(2)
            data = test_rex_mbed_os_name.group(1).split('/')
            ndata = len(data)

            if ndata == 1:
                module_name = data[0]
            else:
                module_name = data[0] + '/' + data[1]

            return [module_name, object_name]
        else:
            rex_obj_name = r'^.+\/(.+\.o\)*)$'
            test_rex_obj_name = re.match(rex_obj_name, txt)
            if test_rex_obj_name:
                object_name = test_rex_obj_name.group(1)
                return [object_name, ""]       

            return ['Misc', ""]

With this I get and output similar to this:

Module	.text	.data	.bss
BufferedSerial.o	12	0	0
Fill	713	34	67
at24mac.o	134	0	20
crt0.o	116	0	0
crtbegin.o	92	12	28
crtend.o	4	4	0
crti.o	8	0	0
crtn.o	16	0	0
driverAtmelRFInterface.o	2541	1	8
driverRFPhy.o	2334	15	196
features/FEATURE_CLIENT	39077	11	53
features/FEATURE_COMMON_PAL	19454	93	8412
features/frameworks	3827	92	848
features/mbedtls	75976	51	8755
features/net	140911	302	13841
hal/common	5689	20	421
hal/targets	10767	4	1472
libc.a(lib_a-__atexit.o)	164	0	0
libc.a(lib_a-__call_atexit.o)	232	4	0
libc.a(lib_a-abort.o)	16	0	0
libc.a(lib_a-abs.o)	8	0	0
libc.a(lib_a-assert.o)	123	0	0
libc.a(lib_a-atexit.o)	12	0	0
libc.a(lib_a-atof.o)	8	0	0
libc.a(lib_a-atoi.o)	8	0	0
libc.a(lib_a-atol.o)	8	0	0
libc.a(lib_a-calloc.o)	16	0	0
libc.a(lib_a-callocr.o)	96	0	0
libc.a(lib_a-closer.o)	36	0	0
libc.a(liba-ctype.o)	257	4	0
libc.a(lib_a-cxa_atexit.o)	20	0	0
libc.a(lib_a-div.o)	64	0	0
libc.a(lib_a-dtoa.o)	4044	0	0
libc.a(lib_a-environ.o)	0	4	4
libc.a(lib_a-errno.o)	12	0	0
libc.a(lib_a-exit.o)	32	0	0
libc.a(lib_a-fclose.o)	148	0	0
libc.a(lib_a-fflush.o)	420	0	0
libc.a(lib_a-findfp.o)	456	0	0
libc.a(lib_a-fini.o)	44	0	0
libc.a(lib_a-fiprintf.o)	40	0	0
libc.a(lib_a-flags.o)	116	0	0
libc.a(lib_a-fopen.o)	152	0	0
libc.a(lib_a-fputwc.o)	212	0	0
libc.a(lib_a-freer.o)	588	0	0
libc.a(lib_a-fseek.o)	4	0	0
libc.a(lib_a-fseeko.o)	696	0	0
libc.a(lib_a-fstatr.o)	40	0	0
libc.a(lib_a-fvwrite.o)	792	0	0
libc.a(lib_a-fwalk.o)	160	0	0
libc.a(lib_a-gdtoa-gethex.o)	1716	0	0
libc.a(lib_a-gdtoa-hexnan.o)	388	0	0
libc.a(lib_a-getc.o)	52	0	0
libc.a(lib_a-getchar.o)	16	0	0
libc.a(lib_a-getenv_r.o)	152	0	0
libc.a(lib_a-gettzinfo.o)	8	64	0
libc.a(lib_a-gmtime_r.o)	364	0	0
libc.a(lib_a-impure.o)	6	1068	0
libc.a(lib_a-init.o)	80	0	0
libc.a(lib_a-isalpha.o)	20	0	0
libc.a(lib_a-isattyr.o)	36	0	0
libc.a(lib_a-isdigit.o)	20	0	0
libc.a(lib_a-isspace.o)	20	0	0
libc.a(lib_a-iswspace.o)	28	0	0
libc.a(lib_a-lcltime.o)	16	0	0
libc.a(lib_a-lcltime_r.o)	600	0	0
libc.a(lib_a-locale.o)	38	92	0
libc.a(lib_a-lseekr.o)	40	0	0
libc.a(lib_a-makebuf.o)	224	0	0
libc.a(lib_a-malloc.o)	32	0	0
libc.a(lib_a-mallocr.o)	1316	1040	52
libc.a(lib_a-mbrtowc.o)	93	0	0
libc.a(lib_a-mbtowc_r.o)	44	4	0
libc.a(lib_a-memchr.o)	148	0	0
libc.a(lib_a-memcmp.o)	100	0	0
libc.a(lib_a-memcpy.o)	308	0	0
libc.a(lib_a-memmove.o)	200	0	0
libc.a(lib_a-memset.o)	156	0	0
libc.a(lib_a-mktime.o)	1512	0	0
libc.a(lib_a-month_lengths.o)	96	0	0
libc.a(lib_a-mprec.o)	2432	0	0
libc.a(lib_a-openr.o)	40	0	0
libc.a(lib_a-printf.o)	40	0	0
libc.a(lib_a-rand.o)	84	0	0
libc.a(lib_a-readr.o)	40	0	0
libc.a(lib_a-realloc.o)	16	0	0
libc.a(lib_a-reallocr.o)	1004	0	0
libc.a(lib_a-reent.o)	0	0	4
libc.a(lib_a-refill.o)	292	0	0
libc.a(lib_a-rget.o)	52	0	0
libc.a(lib_a-sbrkr.o)	36	0	0
libc.a(lib_a-sccl.o)	120	0	0
libc.a(lib_a-setbuf.o)	16	0	0
libc.a(lib_a-setvbuf.o)	248	0	0
libc.a(lib_a-sf_nan.o)	8	0	0
libc.a(lib_a-signal.o)	104	0	0
libc.a(lib_a-signalr.o)	44	0	0
libc.a(lib_a-siscanf.o)	88	0	0
libc.a(lib_a-snprintf.o)	156	0	0
libc.a(lib_a-sprintf.o)	76	0	0
libc.a(lib_a-sscanf.o)	88	0	0
libc.a(lib_a-stdio.o)	136	0	0
libc.a(lib_a-strchr.o)	204	0	0
libc.a(lib_a-strcmp.o)	844	0	0
libc.a(lib_a-strcpy.o)	188	0	0
libc.a(lib_a-strerror.o)	2781	0	0
libc.a(lib_a-strlen.o)	92	0	0
libc.a(lib_a-strncmp.o)	152	0	0
libc.a(lib_a-strrchr.o)	36	0	0
libc.a(lib_a-strstr.o)	1232	0	0
libc.a(lib_a-strtod.o)	4095	0	0
libc.a(lib_a-strtol.o)	308	0	0
libc.a(lib_a-strtoll_r.o)	460	0	0
libc.a(lib_a-strtoul.o)	320	0	0
libc.a(lib_a-strtoull_r.o)	472	0	0
libc.a(lib_a-svfiprintf.o)	260	0	0
libc.a(lib_a-svfiscanf.o)	2846	0	0
libc.a(lib_a-svfprintf.o)	5406	0	0
libc.a(lib_a-svfscanf.o)	3699	0	0
libc.a(lib_a-tzcalc_limits.o)	396	0	0
libc.a(lib_a-tzlock.o)	8	0	0
libc.a(lib_a-tzset.o)	12	0	0
libc.a(lib_a-tzset_r.o)	887	0	26
libc.a(lib_a-tzvars.o)	4	8	8
libc.a(lib_a-u_strerr.o)	4	0	0
libc.a(lib_a-ungetc.o)	120	0	0
libc.a(lib_a-vfiprintf.o)	3431	0	0
libc.a(lib_a-vfprintf.o)	5402	0	0
libc.a(lib_a-vsnprintf.o)	148	0	0
libc.a(lib_a-vsprintf.o)	76	0	0
libc.a(lib_a-wbuf.o)	168	0	0
libc.a(lib_a-wcrtomb.o)	84	0	0
libc.a(lib_a-wctomb_r.o)	28	4	0
libc.a(lib_a-writer.o)	40	0	0
libc.a(lib_a-wsetup.o)	196	0	0
libgcc.a(_aeabi_ldivmod.o)	56	0	0
libgcc.a(_aeabi_uldivmod.o)	48	0	0
libgcc.a(_arm_addsubdf3.o)	880	0	0
libgcc.a(_arm_cmpdf2.o)	272	0	0
libgcc.a(_arm_fixdfsi.o)	80	0	0
libgcc.a(_arm_fixunsdfsi.o)	64	0	0
libgcc.a(_arm_muldivdf3.o)	1060	0	0
libgcc.a(_arm_truncdfsf2.o)	160	0	0
libgcc.a(_ashldi3.o)	24	0	0
libgcc.a(_ashrdi3.o)	28	0	0
libgcc.a(_divdi3.o)	668	0	0
libgcc.a(_dvmd_tls.o)	4	0	0
libgcc.a(_fixunssfdi.o)	76	0	0
libgcc.a(_lshrdi3.o)	24	0	0
libgcc.a(_udivdi3.o)	620	0	0
libgcc.a(bpabi.o)	96	0	0
libgcc.a(libunwind.o)	428	0	0
libgcc.a(pr-support.o)	892	0	0
libgcc.a(unwind-arm.o)	2640	0	0
libm.a(lib_a-s_fpclassify.o)	88	0	0
libnosys.a(getpid.o)	16	0	0
libnosys.a(kill.o)	16	0	0
libstdc++.a(atexit_arm.o)	12	0	0
libstdc++.a(bad_alloc.o)	116	0	0
libstdc++.a(class_type_info.o)	338	0	0
libstdc++.a(del_op.o)	4	0	0
libstdc++.a(del_opv.o)	4	0	0
libstdc++.a(eh_alloc.o)	136	0	2052
libstdc++.a(eh_arm.o)	400	0	0
libstdc++.a(eh_call.o)	304	0	0
libstdc++.a(eh_catch.o)	280	0	0
libstdc++.a(eh_exception.o)	226	0	0
libstdc++.a(eh_globals.o)	16	0	12
libstdc++.a(eh_personality.o)	1508	0	0
libstdc++.a(eh_term_handler.o)	0	4	0
libstdc++.a(eh_terminate.o)	120	0	0
libstdc++.a(eh_throw.o)	308	0	0
libstdc++.a(eh_unex_handler.o)	0	4	0
libstdc++.a(functexcept.o)	376	0	0
libstdc++.a(guard.o)	64	0	0
libstdc++.a(guard_error.o)	115	0	0
libstdc++.a(locale-inst.o)	356	4	48
libstdc++.a(locale_facets.o)	136	4	0
libstdc++.a(new_handler.o)	16	0	4
libstdc++.a(new_op.o)	68	0	0
libstdc++.a(new_opv.o)	20	0	0
libstdc++.a(si_class_type_info.o)	381	0	0
libstdc++.a(snprintf_lite.o)	396	0	0
libstdc++.a(stdexcept.o)	402	0	0
libstdc++.a(string-inst.o)	1337	0	16
libstdc++.a(system_error.o)	473	12	0
libstdc++.a(tinfo.o)	77	0	0
libstdc++.a(wlocale-inst.o)	196	4	48
main.o	4285	21	410
rtos/rtos	575	4	0
rtos/rtx	7037	20	2683
Subtotals	386365	3008	39488

Allocated Heap: unknown Allocated Stack: unknown Total Static RAM memory (data + bss): 42496 bytes Total RAM memory (data + bss + heap + stack): 42496 bytes Total Flash memory (text + data + misc): 389373 bytes

I think it would be nice to have such print as optional for the mbed-cli (maybe even for mbed-os features so we could help you identify and fix issues).

MarceloSalazar commented 8 years ago

Hi Javier,

This feature looks very useful :) A few suggestions:

Add a verbose option to generate more information when invoking memap from the command line (by default should be disabled): https://github.com/ARMmbed/mbed-os/blob/5c82db820357508a27f7c6a18d90dc1d081851af/tools/memap.py#L597
Display Misc to make the output more consistent with the default behavior: libc.a(lib_a-__atexit.o) --> Misc\libc.a(lib_a-__atexit.o)

It would be great if you could send us a PR and we'll be happy to review it.

Thanks, Marcelo

javier-moreno-tridonic-com commented 8 years ago

Well I'd be happy to prepare it, just I would need some pointers for this:

Should be this a memap only feature? I mean, should this feature be also accessible by using a "mbed compile" flag? If so, any flag name suggestion that can be shared across both scripts?

MarceloSalazar commented 8 years ago

I'd suggest this to be a memap only feature for now (e.g. invoke the tool using python memap ...). This will let us prepare the interfaces and do further testing. We will consider calling it from the mbed compile script.

nuket commented 8 years ago

This is great, I'm also looking for a way to have a deeper view into flash / stack / heap memory allocations.

sg- commented 8 years ago

@meriac Anything you want to share here?

meriac commented 8 years ago

@nuket : I had good results parsing arm-non-eabi-nm output - feel free to have a look on an early prototype for d3.js based linker visualization.

You can find the command line settings I chose here. Please note that I had to enable debug information to get line number resolution in the mbed release build. You can find a quick patch for GCC in my tool repo. The official pull request is #2139 for adding debug symbol support to all mbed supported release builds.

javier-moreno-tridonic-com commented 7 years ago

Since this functionality was merged already (see https://github.com/ARMmbed/mbed-os/pull/2845) this can be closed

ARMmbed / mbed-os