search

BiomedSciAI / fuse-med-ml

A python framework accelerating ML based discovery in the medical field by encouraging code reuse. Batteries included :)
Apache License 2.0
137 stars 34 forks source link

`NDict` optimization #271

Closed SagiPolaczek closed 1 year ago

SagiPolaczek commented 1 year ago

✅ Ready for review

Profiling

EHR Transformer (5 epochs)

Not Optimized

ncalls  tottime  percall  cumtime  percall filename:lineno(function)

2565    0.003    0.000    0.004    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:150(keys)
2565    0.004    0.000    0.005    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:159(items)
15390    0.043    0.000    0.117    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:223(get_closest_key)
15390    0.021    0.000    0.138    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:294(__contains__)
61560/2565    0.066    0.000    0.185    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:118(_flatten_static)
2565    0.008    0.000    0.193    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:100(flatten)
53905/2575    0.088    0.000    0.203    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:136(_keypaths_static)
2575    0.018    0.000    0.221    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:130(keypaths)
412420    0.497    0.000    1.916    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:172(__getitem__)
2565    4.561    0.002    4.561    0.002 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:264(<listcomp>)
2565    2.039    0.001    7.170    0.003 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:249(indices)
6275340/6272775    4.483    0.000   16.573    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:191(__setitem__)
330290    2.766    0.000   20.023    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:63(__init__)
Optimized
ncalls  tottime  percall  cumtime  percall filename:lineno(function)

       10    0.000    0.000    0.000    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:106(keys)
     2565    0.004    0.000    0.005    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:115(items)
    12835    0.024    0.000    0.027    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:100(keypaths)
     5130    0.026    0.000    0.049    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:149(is_prefix)
     5130    0.019    0.000    0.051    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:224(get_closest_key)
     5130    0.005    0.000    0.056    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:303(__contains__)
     5140    0.022    0.000    0.075    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:47(__init__)
     2565    0.026    0.000    0.076    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:172(get_sub_dict)
153960/143700    0.080    0.000    0.174    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:129(__getitem__)
    56450    0.047    0.000    0.275    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:193(__setitem__)
     5130    8.242    0.002    8.242    0.002 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:267(<listcomp>)
     5130    3.152    0.001   11.579    0.002 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:251(indices)

ISIC (2 epochs)

Not Optimized
ncalls  tottime  percall  cumtime  percall filename:lineno(function)

75/5    0.000    0.000    0.000    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:136(_keypaths_static)
5    0.000    0.000    0.000    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:130(keypaths)
4184    0.006    0.000    0.007    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:159(items)
25104    0.053    0.000    0.173    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:223(get_closest_key)
25104    0.024    0.000    0.197    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:294(__contains__)
65560    0.148    0.000    0.617    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:172(__getitem__)
4189    0.035    0.000    0.709    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:63(__init__)
179337/29303    0.350    0.000    0.960    0.000 /dccstor/mm_hcls/usr/sagi/fuse_2/fuse/utils/ndict.py:191(__setitem__)
Optimized
ncalls  tottime  percall  cumtime  percall filename:lineno(function)

5    0.000    0.000    0.000    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:106(keys)
4184    0.006    0.000    0.007    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:115(items)
20925    0.031    0.000    0.035    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:100(keypaths)
4184    0.042    0.000    0.110    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:172(get_sub_dict)
90664/82296    0.048    0.000    0.191    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:129(__getitem__)
16736    0.105    0.000    0.198    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:149(is_prefix)
25104    0.056    0.000    0.228    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:224(get_closest_key)
25104    0.017    0.000    0.245    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:303(__contains__)
8373    0.070    0.000    0.368    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:47(__init__)
133313/133263    0.125    0.000    0.500    0.000 /dccstor/mm_hcls/usr/sagi/fuse_3/fuse/utils/ndict.py:193(__setitem__)
SagiPolaczek commented 1 year ago

Added a self-CR.