Performance issue around sorting ?

[mitra42]

In my most recent test using 70 clients, 70 steps, the interesting part is in the first comment ...

What I notice is that 219 of 822 seconds is spent in _sort_and_truncate (which has no file reading). The only reason we do that is to reduce load, and it makes me wonder if we should change the constants and raise the max data points per test - maybe by a factor of 10, or even more since its cheaper on the server to send a large data packet than to sort the lists and pick from them ?

Thoughts @danaronson

[mitra42]

    10152    1.043    0.000  822.598    0.081 server.py:168(render)
    10152    0.115    0.000  810.103    0.080 /Users/mitra/git/bct-server2/contacts.py:597(execute_route)
     4531    1.517    0.000  648.850    0.143 /Users/mitra/git/bct-server2/contacts.py:691(scan_status)
     4531    8.994    0.002  645.664    0.142 /Users/mitra/git/bct-server2/contacts.py:816(_scan_or_sync)
     4531    0.022    0.000  378.698    0.084 /Users/mitra/git/bct-server2/contacts.py:471(list_over_bounding_boxes)
     4531   41.745    0.009  378.677    0.084 /Users/mitra/git/bct-server2/contacts.py:460(list_over_bounding_boxes_A)
 44698337   30.073    0.000  261.980    0.000 /Users/mitra/git/bct-server2/contacts.py:441(_flatten)
     4531    0.747    0.000  219.617    0.048 /Users/mitra/git/bct-server2/contacts.py:787(_sort_and_truncate)
   588195    0.897    0.000  217.747    0.000 /Users/mitra/git/bct-server2/contacts.py:447(_intersections)
   588195    6.075    0.000  207.906    0.000 /usr/local/lib/python3.7/site-packages/rtree/index.py:752(_get_objects)
   583664   19.177    0.000  201.497    0.000 /usr/local/lib/python3.7/site-packages/rtree/index.py:1154(__init__)
      801    1.784    0.002  120.438    0.150 /Users/mitra/git/bct-server2/contacts.py:663(_update_or_result)
   583664   65.563    0.000  119.220    0.000 /usr/local/lib/python3.7/site-packages/rtree/index.py:35(_get_bounds)
      657    0.046    0.000  114.109    0.174 /Users/mitra/git/bct-server2/contacts.py:717(status_result)
     4406   91.004    0.021  110.880    0.025 {method 'sort' of 'list' objects}
  1201504   27.556    0.000  109.814    0.000 {method 'extend' of 'list' objects}
   170111    1.018    0.000   97.069    0.001 /Users/mitra/git/bct-server2/contacts.py:646(_update)
   171059    0.385    0.000   95.991    0.001 {built-in method builtins.any}
   504844    0.731    0.000   95.669    0.000 /Users/mitra/git/bct-server2/contacts.py:652(<genexpr>)
   334733    1.328    0.000   94.938    0.000 /Users/mitra/git/bct-server2/contacts.py:306(update)
    40897   21.204    0.001   94.301    0.002 /Users/mitra/git/bct-server2/contacts.py:181(insert)
 37000597   80.801    0.000   80.801    0.000 /Users/mitra/git/bct-server2/contacts.py:798(<lambda>)
 44693806   49.069    0.000   74.952    0.000 /Users/mitra/git/bct-server2/contacts.py:462(<lambda>)
   583664    3.221    0.000   58.325    0.000 /usr/local/lib/python3.7/site-packages/rtree/index.py:1180(get_object)
   583664   29.746    0.000   48.844    0.000 /usr/local/lib/python3.7/site-packages/rtree/index.py:67(_get_data)
    31808    4.724    0.000   44.629    0.001 /Users/mitra/git/bct-server2/contacts.py:221(retrieve_json_from_file_path)
    72708   37.529    0.001   39.814    0.001 {built-in method io.open}

[mitra42]

Done in config