search

davidpng / FCS_Database

Program to scrape an FCS directory of metadata
GNU General Public License v3.0
3 stars 2 forks source link

Paralellize stat generation #53

Closed hermands closed 9 years ago

hermands commented 9 years ago

Notes:

David -- try to parallelize (only push unflagged) Dan -- Get db to handle the flag (and add to TubeCases table)

davidpng commented 9 years ago

As expected, the load and processing step is most expensive. output% stats 30 Fri Jan 23 12:59:27 2015 output

     10993399 function calls (10924800 primitive calls) in 47.876 seconds

Ordered by: cumulative time List reduced from 5675 to 30 due to restriction <30>

ncalls tottime percall cumtime percall filename:lineno(function) 1 0.000 0.000 47.892 47.892 flowanal.py:3() 1 0.175 0.175 47.882 47.882 main.py:96(main) 1 0.245 0.245 44.548 44.548 par_add_stats_db.py:48(action) 10 0.001 0.000 19.852 1.985 FCS.py:94(comp_scale_FCS_data) 10 0.054 0.005 19.851 1.985 Process_FCS_Data.py:37(init) 10 0.000 0.000 12.896 1.290 FCS.py:149(extract_FCS_histostats) 10 0.000 0.000 12.896 1.290 Extract_HistoStats.py:23(init) 10 0.433 0.043 11.110 1.111 Process_FCS_Data.py:239(_LogicleRescale) 10 0.001 0.000 8.362 0.836 Process_FCS_Data.py:257(LogicleTransform) 10 0.000 0.000 8.292 0.829 polyint.py:62(call) 10 0.157 0.016 8.292 0.829 interpolate.py:472(_evaluate) 10 3.661 0.366 8.068 0.807 interpolate.py:423(_call_linear) 10 0.006 0.001 7.413 0.741 Extract_HistoStats.py:73(generate_comp_corr_mtx) 900 0.020 0.000 7.396 0.008 Extract_HistoStats.py:96(comp_correlation) 10 0.000 0.000 5.210 0.521 FCS.py:50(init) 10 0.000 0.000 5.210 0.521 FCS.py:74(load_from_file) 10 0.001 0.000 5.210 0.521 loadFCS.py:44(init) 900 0.013 0.000 4.989 0.006 Extract_HistoStats.py:108(UL_gating) 10 0.426 0.043 4.830 0.483 loadFCS.py:194(parse_data) 3 0.000 0.000 4.500 1.500 query_database.py:36(init) 1 0.000 0.000 4.442 4.442 FCS_database.py:77(query) 1 0.341 0.341 4.440 4.440 query_database.py:119(**getfiles) 10 0.001 0.000 4.403 0.440 loadFCS.py:212(__float_parsing) 530/330 0.003 0.000 4.348 0.013 frame.py:2097(setitem) 52397/51291 3.933 0.000 3.977 0.000 {numpy.core.multiarray.array} 920 0.002 0.000 3.823 0.004 path.py:479(contains_points) 920 3.819 0.004 3.821 0.004 {built-in method points_in_path} 5315 0.035 0.000 3.738 0.001 frame.py:1757(getitem**) 1180 3.701 0.003 3.701 0.003 {method 'searchsorted' of 'numpy.ndarray' objects} 10 0.000 0.000 3.648 0.365 fromnumeric.py:955(searchsorted)