Perf: Profile backfill script

[bmtcril]

We know that data backfill is slow using the event-routing-backends management command. We're not sure where and should profile it to see where improvements can be made. To get the most out of the testing we should:

• Run on multiple large files, at least 3, to see which parts add up when running the whole process. Should be run in k8s as we would in a large instance with different performance characteristics, but first pass in tutor local would be next best
• Update the Aspects command to wrap the command in cProfile like changing this line: https://github.com/openedx/tutor-contrib-aspects/blob/5e0f970d4fef3c33739426fb8e7ee22a84fb8d78/tutoraspects/commands_v1.py#L113 to
• return [("lms", f"python -m cProfile -s time ./manage.py lms transform_tracking_logs {options}")]
• Capturing the output (there will be a lot), if the place your running has a writeable mount that can be accessed you can just redirect the output of the command above to that
• Once the log is available you can chop out the output from the script and run tools like SnakeViz on the output to examine where it's spending time

[Ian2012]

@bmtcril can this be closed?

[bmtcril]

I'm not sure where it got left, honestly. I think we had some plan to bring the times down by like 1/3, but you were the one doing the testing. :)

[bmtcril]

This work is complete, below are the cumulative profile times. Based on this I think we can really speed things up by reducing the queries for actors and course ids by doing some aggressive caching. This is the wrong repo for that, I've added a ticket here for the work: https://github.com/openedx/event-routing-backends/issues/354

Ordered by: cumulative time

   ncalls  tottime  percall  cumtime  percall filename:lineno(function)
   6423/1    0.035    0.000 2311.210 2311.210 {built-in method builtins.exec}
        1    0.000    0.000 2311.210 2311.210 manage.py:2(<module>)
        1    0.000    0.000 2308.027 2308.027 __init__.py:416(execute_from_command_line)
        1    0.000    0.000 2308.027 2308.027 __init__.py:334(execute)
        1    0.000    0.000 2308.005 2308.005 base.py:337(run_from_argv)
        1    0.000    0.000 2308.004 2308.004 base.py:373(execute)
        1    0.000    0.000 2306.344 2306.344 transform_tracking_logs.py:267(handle)
        1   60.439   60.439 2303.691 2303.691 transform_tracking_logs.py:55(transform_tracking_logs)
   882323    0.522    0.000 1976.575    0.002 queued_sender.py:67(transform_and_queue)
   254744    0.111    0.000 1963.438    0.008 queued_sender.py:84(queue)
       25    0.717    0.029 1712.769   68.511 queued_sender.py:103(send)
       25    0.078    0.003 1712.052   68.482 events_router.py:119(bulk_send)
       25    3.281    0.131 1656.746   66.270 events_router.py:56(prepare_to_send)
   250000    0.845    0.000 1648.634    0.007 events_router.py:175(process_event)
   250000    1.494    0.000 1647.584    0.007 base_transformer_processor.py:21(__call__)
   250000    3.547    0.000 1646.091    0.007 transformer_processor.py:29(transform_event)
   250000    0.477    0.000 1526.797    0.006 base_transformer_processor.py:42(transform_event)
   250000    1.040    0.000 1526.210    0.006 base_transformer_processor.py:77(get_transformed_event)
   250000    1.240    0.000 1523.985    0.006 transformer.py:34(transform)
   250000    2.343    0.000 1469.088    0.006 base_transformer.py:90(transform)
   250000    2.493    0.000 1390.372    0.006 transformer.py:49(base_transform)
  1246054    6.513    0.000  916.025    0.001 query.py:1322(_fetch_all)
   748006   11.055    0.000  908.824    0.001 query.py:45(__iter__)
   748005    7.997    0.000  770.238    0.001 compiler.py:1147(execute_sql)
   250000    1.684    0.000  746.536    0.003 transformer.py:66(get_actor)
   250000    1.437    0.000  726.945    0.003 helpers.py:45(get_anonymous_user_id)
   749765    0.985    0.000  701.127    0.001 query.py:265(__iter__)
   250000    5.532    0.000  687.299    0.003 helpers.py:86(get_user)
   250000    1.155    0.000  600.471    0.002 transformer.py:123(get_context)
   249314    4.662    0.000  582.576    0.002 transformer.py:97(get_context_activities)
   249969    1.141    0.000  547.563    0.002 helpers.py:138(get_course_from_id)
   249969    2.298    0.000  528.564    0.002 api.py:79(get_course_overviews)
   249969    1.339    0.000  439.397    0.002 serializers.py:743(data)
   249969    1.036    0.000  437.292    0.002 serializers.py:232(data)
   249969    0.899    0.000  435.981    0.002 serializers.py:655(to_representation)
748006/748005   13.506    0.000  364.822    0.000 compiler.py:503(as_sql)
   748007    1.158    0.000  344.726    0.000 utils.py:65(execute)
   748007    1.552    0.000  343.568    0.000 utils.py:71(_execute_with_wrappers)
   748007    2.792    0.000  342.016    0.000 utils.py:77(_execute)
   748007    0.997    0.000  338.618    0.000 base.py:70(execute)
   748007    5.579    0.000  337.621    0.000 cursors.py:171(execute)
   748007    2.814    0.000  325.155    0.000 cursors.py:316(_query)
   748007    1.531    0.000  298.696    0.000 connections.py:250(query)
   748007  296.940    0.000  296.940    0.000 {function Connection.query at 0x7fc9e2af8820}
   249916    1.924    0.000  286.177    0.001 query.py:672(first)
   248119    2.333    0.000  280.204    0.001 user.py:317(get_potentially_retired_user_by_username)
   748006    4.241    0.000  273.167    0.000 compiler.py:49(pre_sql_setup)
       25  250.207   10.008  250.207   10.008 {built-in method time.sleep}
    14760    0.049    0.000  242.560    0.016 socket.py:655(readinto)
    14760    0.047    0.000  242.478    0.016 ssl.py:1230(recv_into)
    14760    0.033    0.000  242.421    0.016 ssl.py:1090(read)
    14760  242.385    0.016  242.385    0.016 {method 'read' of '_ssl._SSLSocket' objects}
   748006    3.961    0.000  227.164    0.000 compiler.py:43(setup_query)
   748006   28.208    0.000  222.063    0.000 compiler.py:199(get_select)
      652    0.015    0.000  219.861    0.337 base.py:519(request)
      652    0.012    0.000  219.665    0.337 base.py:650(_retryable_request)
19787196/18030524    2.878    0.000  218.935    0.000 {built-in method builtins.len}
      652    0.028    0.000  218.132    0.335 sessions.py:671(send)
      652    0.017    0.000  216.391    0.332 adapters.py:436(send)
   248144    0.351    0.000  216.350    0.001 query.py:261(__len__)
      652    0.032    0.000  215.965    0.331 connectionpool.py:522(urlopen)
      648    0.007    0.000  215.956    0.333 transform_tracking_logs.py:21(_get_chunks)
      648    0.015    0.000  215.947    0.333 s3.py:557(download_object_range_as_stream)
      648    0.040    0.000  215.614    0.333 http.py:244(prepared_request)
      651    0.022    0.000  215.572    0.331 connectionpool.py:361(_make_request)
      651    0.006    0.000  197.480    0.303 client.py:1304(getresponse)
      651    0.031    0.000  197.459    0.303 client.py:309(begin)
     9218    0.026    0.000  196.943    0.021 {method 'readline' of '_io.BufferedReader' objects}
      651    0.042    0.000  196.810    0.302 client.py:276(_read_status)
22218618/20472681   19.171    0.000  174.807    0.000 compiler.py:440(compile)
   748047    2.656    0.000  170.391    0.000 manager.py:84(manager_method)
   871515    2.546    0.000  160.476    0.000 __init__.py:1565(_log)
   748007    1.110    0.000  154.094    0.000 query.py:935(filter)
   748008    2.171    0.000  152.093    0.000 query.py:951(_filter_or_exclude)
   748000    2.084    0.000  138.642    0.000 __init__.py:1436(info)
   748008    2.758    0.000  129.730    0.000 query.py:964(_filter_or_exclude_inplace)
   871515    1.151    0.000  125.001    0.000 __init__.py:1591(handle)
   871515    3.513    0.000  123.160    0.000 __init__.py:1645(callHandlers)
   748008    3.345    0.000  122.790    0.000 query.py:1404(add_q)
  1743022    3.634    0.000  119.647    0.000 __init__.py:941(handle)
26281376/24386463   19.880    0.000  119.612    0.000 {method 'join' of 'str' objects}
   748008    6.809    0.000  117.457    0.000 query.py:1424(_add_q)
3136759/2886759   10.099    0.000  104.060    0.000 serializable_base.py:54(__init__)
38221061/26819327   49.146    0.000  103.819    0.000 base.py:48(__setattr__)
   748009   12.798    0.000   94.690    0.000 query.py:1254(build_filter)
   748005    2.570    0.000   93.944    0.000 compiler.py:1126(results_iter)
3172276/2886759    8.573    0.000   93.391    0.000 base.py:27(__init__)
   748005   20.231    0.000   90.087    0.000 compiler.py:1105(get_converters)
 19724679   18.475    0.000   86.377    0.000 expressions.py:832(as_sql)
11333501/10691423    4.574    0.000   83.964    0.000 {built-in method builtins.setattr}
   749886    2.253    0.000   73.433    0.000 serializable_base.py:92(to_json)
   748005   27.224    0.000   66.386    0.000 compiler.py:662(get_default_columns)
  1743022    4.416    0.000   65.106    0.000 __init__.py:1073(emit)
129048219/125023345   30.877    0.000   63.590    0.000 {built-in method builtins.getattr}
748006/748005    3.162    0.000   63.238    0.000 where.py:65(as_sql)
4867100/749886   29.159    0.000   57.018    0.000 serializable_base.py:108(as_version)
   748006    2.631    0.000   55.258    0.000 lookups.py:192(as_sql)
       25    0.000    0.000   55.215    2.209 task.py:414(delay)
       25    0.003    0.000   55.215    2.209 task.py:427(apply_async)
       25    0.001    0.000   55.211    2.208 base.py:704(send_task)
   249886    1.201    0.000   53.657    0.000 statement.py:59(__init__)
 40197364   31.170    0.000   52.323    0.000 compiler.py:423(quote_name_unless_alias)
     1402    0.006    0.000   50.606    0.036 models.py:812(generate)
     1402    0.005    0.000   50.600    0.036 response.py:607(stream)
   249886    0.962    0.000   50.477    0.000 statement_base.py:50(__init__)
       25    0.000    0.000   49.335    1.973 amqp.py:366(as_task_v1)
1894780/50    0.674    0.000   49.334    0.987 saferepr.py:67(saferepr)
22632/21338    0.011    0.000   49.254    0.002 {built-in method builtins.next}
     1294    0.013    0.000   49.228    0.038 files.py:26(read_in_chunks)
      649    0.019    0.000   49.189    0.076 response.py:535(read)
11868415/8078955    7.774    0.000   48.445    0.000 saferepr.py:166(_saferepr)
      649    0.006    0.000   45.055    0.069 response.py:487(_fp_read)
      649    0.529    0.001   45.049    0.069 client.py:448(read)
      647    0.021    0.000   44.491    0.069 client.py:480(readinto)
      647    0.034    0.000   44.451    0.069 {method 'readinto' of '_io.BufferedReader' objects}
   498089    1.604    0.000   43.842    0.000 lookups.py:408(as_sql)
   871511    2.001    0.000   42.536    0.000 handlers.py:472(emit)
   748009    4.109    0.000   42.175    0.000 query.py:1189(build_lookup)
 18726753   13.121    0.000   38.841    0.000 expressions.py:846(get_db_converters)
 38453373   24.926    0.000   36.931    0.000 __init__.py:515(__eq__)
  4265490    5.258    0.000   36.877    0.000 base_transformer.py:145(get_data)
258919319/258418789   29.463    0.000   35.512    0.000 {built-in method builtins.isinstance}
  2614537    2.942    0.000   35.130    0.000 __init__.py:796(filter)
   249886    0.537    0.000   33.698    0.000 statement_base.py:137(timestamp)
   748007    4.711    0.000   33.679    0.000 lookups.py:183(process_lhs)
 11868415   23.194    0.000   33.500    0.000 saferepr.py:205(reprstream)
 19724681   14.468    0.000   32.989    0.000 __init__.py:394(get_col)
  1743022    1.543    0.000   32.782    0.000 __init__.py:918(format)
   748006    5.660    0.000   32.688    0.000 compiler.py:271(get_order_by)
   249886    0.354    0.000   32.590    0.000 iso8601.py:128(make_datetime)
   249886    0.722    0.000   32.236    0.000 iso8601.py:162(_make_datetime)
   249886    1.350    0.000   31.489    0.000 time.py:182(parse_datetime)
  1743022    4.708    0.000   31.239    0.000 __init__.py:655(format)
 18726754   24.700    0.000   31.014    0.000 operations.py:294(get_db_converters)
  1247936    1.369    0.000   30.311    0.000 query.py:1296(_chain)
2244025/748006    5.053    0.000   29.826    0.000 asyncio.py:21(inner)
   871511    0.812    0.000   29.631    0.000 __init__.py:1178(emit)
   871511    1.457    0.000   29.542    0.000 filters.py:27(filter)
  1247936    4.949    0.000   28.445    0.000 query.py:1308(_clone)
   871511    5.613    0.000   28.084    0.000 __init__.py:45(get_current_user)
  1497270    0.942    0.000   27.225    0.000 connection.py:56(__getitem__)
   743769    3.009    0.000   26.979    0.000 transformer.py:155(verb)
   871515    1.422    0.000   26.879    0.000 __init__.py:1550(makeRecord)
1247843/997926    3.797    0.000   26.760    0.000 expressions.py:230(resolve_expression)
750103/250111    1.842    0.000   26.039    0.000 serializers.py:117(__new__)
   871516   12.707    0.000   25.457    0.000 __init__.py:288(__init__)
   249996    2.047    0.000   25.385    0.000 serializers.py:128(many_init)
   123282    0.390    0.000   25.077    0.000 __init__.py:1448(warning)
71375987/71127569   15.678    0.000   24.982    0.000 {built-in method builtins.hasattr}
  1506721    3.513    0.000   23.761    0.000 local.py:99(__getattr__)
748007/748005    0.808    0.000   22.132    0.000 base.py:256(cursor)
   871511    5.312    0.000   21.765    0.000 dispatcher.py:185(send_robust)
  1247846    1.555    0.000   21.585    0.000 expressions.py:345(copy)
  1247936    1.891    0.000   21.505    0.000 query.py:356(chain)
748007/748005    3.197    0.000   21.324    0.000 base.py:234(_cursor)
   748006    2.842    0.000   21.121    0.000 compiler.py:1640(cursor_iter)
  1510835    2.164    0.000   20.265    0.000 local.py:80(_get_storage)
   748007    1.512    0.000   20.161    0.000 lookups.py:83(process_lhs)
   249886    0.708    0.000   20.102    0.000 python.py:298(build_datetime)
  1253067    5.162    0.000   20.059    0.000 copy.py:66(copy)
  1247937   12.027    0.000   19.296    0.000 query.py:305(clone)
749658/499772    1.768    0.000   19.192    0.000 __init__.py:453(_build_object)
  1743034    3.183    0.000   18.878    0.000 __init__.py:1062(flush)
   499169    2.810    0.000   18.560    0.000 activity_definition.py:57(__init__)
  2014538    1.864    0.000   18.358    0.000 __init__.py:299(loads)
      651    0.002    0.000   17.882    0.027 connectionpool.py:1034(_validate_conn)
       26    0.004    0.000   17.880    0.688 connection.py:361(connect)
   249970    1.762    0.000   17.858    0.000 __init__.py:170(from_string)
   749988    1.116    0.000   17.740    0.000 __init__.py:183(dumps)
   748007    3.428    0.000   17.655    0.000 cursors.py:144(_do_get_result)
  1510835    7.794    0.000   17.480    0.000 local.py:44(_get_context_id)
 18726756   11.327    0.000   17.196    0.000 expressions.py:382(select_format)
    64867    0.438    0.000   17.088    0.000 video_events.py:194(get_result)
   743782    1.294    0.000   16.924    0.000 verb.py:36(__init__)
   250000    1.337    0.000   16.862    0.000 agent.py:40(__init__)
   749988    1.940    0.000   16.624    0.000 encoder.py:182(encode)
  1912046    2.830    0.000   15.985    0.000 decoder.py:332(decode)
   498089    3.094    0.000   15.912    0.000 lookups.py:375(process_rhs)
   499169    1.108    0.000   15.691    0.000 activity.py:39(__init__)
   748009    1.798    0.000   15.435    0.000 lookups.py:23(__init__)
   500107    1.720    0.000   15.381    0.000 serializers.py:108(__init__)
   748006    2.425    0.000   15.008    0.000 query.py:290(get_compiler)
 71394311   14.625    0.000   14.625    0.000 {method 'startswith' of 'str' objects}
 19474763   10.222    0.000   14.571    0.000 operations.py:177(quote_name)
   749989   14.223    0.000   14.223    0.000 encoder.py:204(iterencode)
   249918    0.552    0.000   14.121    0.000 lookups.py:303(as_sql)
   501004   11.563    0.000   13.534    0.000 fields.py:323(__init__)
   748083    2.500    0.000   13.444    0.000 manager.py:142(get_queryset)
  1743022    4.458    0.000   13.360    0.000 __init__.py:588(formatTime)
  2741778    0.880    0.000   13.185    0.000 base.py:37(<lambda>)
   748006    3.187    0.000   12.895    0.000 compiler.py:1646(<lambda>)
  1743034   12.811    0.000   12.811    0.000 {method 'flush' of '_io.TextIOWrapper' objects}
   249970    1.719    0.000   12.762    0.000 locator.py:268(_from_string)
  1996020    4.691    0.000   12.721    0.000 query.py:178(__init__)
    31677    0.415    0.000   12.698    0.000 problem_interaction_events.py:179(get_object)
   249886    2.615    0.000   12.335    0.000 python.py:240(build_time)
   882323    0.957    0.000   12.299    0.000 event_log_parser.py:16(parse_json_event)
   498089    1.983    0.000   12.007    0.000 lookups.py:230(get_prep_lookup)
   250000    0.882    0.000   11.923    0.000 helpers.py:26(get_uuid5)
115561857   11.231    0.000   11.231    0.000 {method 'append' of 'list' objects}
  1912046   11.015    0.000   11.075    0.000 decoder.py:343(raw_decode)
   871511    1.627    0.000   10.904    0.000 handlers.py:444(reopenIfNeeded)
   997928    5.440    0.000   10.610    0.000 query.py:1600(setup_joins)
       26    0.000    0.000   10.583    0.407 ssl_.py:355(ssl_wrap_socket)
       26    0.000    0.000   10.385    0.399 ssl_.py:481(_ssl_wrap_socket_impl)
       26    0.000    0.000   10.385    0.399 ssl.py:494(wrap_socket)
       26    0.001    0.000   10.385    0.399 ssl.py:983(_create)
       26    0.001    0.000   10.382    0.399 ssl.py:1302(do_handshake)
       26   10.381    0.399   10.381    0.399 {method 'do_handshake' of '_ssl._SSLSocket' objects}
  1745934    3.642    0.000    9.988    0.000 query.py:951(get_initial_alias)
2256765/1505537    3.096    0.000    9.988    0.000 functional.py:40(__get__)
   249917    1.122    0.000    9.664    0.000 query.py:1143(order_by)
   498088    3.798    0.000    9.603    0.000 lookups.py:260(batch_process_rhs)
   908743    9.412    0.000    9.412    0.000 {built-in method posix.stat}
   132919    0.833    0.000    9.361    0.000 video_events.py:136(get_object)
  4617931    6.746    0.000    9.277    0.000 serializable_base.py:151(_filter_none)
  2253750    2.494    0.000    9.229    0.000 _collections_abc.py:657(get)
  1995854    7.240    0.000    9.165    0.000 query.py:1497(names_to_path)
   250001    2.055    0.000    9.043    0.000 serializers.py:574(__init__)
  1745441    9.030    0.000    9.030    0.000 {method 'write' of '_io.TextIOWrapper' objects}
   249917    1.146    0.000    8.555    0.000 compiler.py:730(find_ordering_name)
   748006    0.798    0.000    8.470    0.000 where.py:32(split_having)
   500000    3.661    0.000    8.231    0.000 uuid.py:784(uuid5)
   748007    2.516    0.000    8.153    0.000 __init__.py:689(db_type)
   250000    1.162    0.000    8.058    0.000 context.py:54(__init__)
   748084    7.159    0.000    8.030    0.000 query.py:161(__init__)
   249969    1.194    0.000    8.012    0.000 models.py:131(get_prep_value)
   748007    1.261    0.000    7.803    0.000 cursors.py:351(_get_result)
   748009    2.570    0.000    7.783    0.000 query.py:1126(solve_lookup_type)
   872002    5.526    0.000    7.743    0.000 dispatcher.py:234(_live_receivers)
  1000081    5.751    0.000    7.728    0.000 uuid.py:132(__init__)
   748005    7.657    0.000    7.657    0.000 query.py:57(<listcomp>)
  1249540    2.606    0.000    7.600    0.000 copy.py:258(_reconstruct)
   250000    0.629    0.000    7.557    0.000 agent.py:174(account)
  1743022    0.916    0.000    7.509    0.000 __init__.py:639(formatMessage)
 51389025    7.490    0.000    7.490    0.000 {method 'get' of 'dict' objects}
 18726755    4.307    0.000    7.312    0.000 __init__.py:746(get_db_converters)
       26    0.000    0.000    7.274    0.280 connection.py:161(_new_conn)
       26    0.001    0.000    7.274    0.280 connection.py:37(create_connection)
   748005    1.439    0.000    7.216    0.000 query.py:1249(db)
  2244019    3.261    0.000    7.155    0.000 utils.py:21(__getattr__)
   249970    2.188    0.000    7.023    0.000 locator.py:172(__init__)
  2320918    3.836    0.000    6.804    0.000 os.py:670(__getitem__)
   748006    0.882    0.000    6.653    0.000 where.py:171(contains_aggregate)
  1743022    0.947    0.000    6.594    0.000 __init__.py:438(format)
   250000    0.609    0.000    6.463    0.000 agent_account.py:31(__init__)
  1009574    3.828    0.000    6.449    0.000 language_map.py:25(__init__)
   249916    1.083    0.000    6.436    0.000 query.py:287(__getitem__)
   748014    0.738    0.000    6.209    0.000 expressions.py:331(get_lookup)
   249886    2.551    0.000    6.202    0.000 time.py:75(parse_time)
   748007    6.075    0.000    6.075    0.000 {method 'store_result' of '_mysql.connection' objects}
   871515    3.723    0.000    6.049    0.000 __init__.py:1514(findCaller)
   748007    1.463    0.000    6.013    0.000 base.py:223(_prepare_cursor)
       25    0.035    0.001    5.830    0.233 amqp.py:452(send_task_message)
   748007    4.164    0.000    5.828    0.000 cursors.py:354(_post_get_result)
   748061    2.423    0.000    5.815    0.000 manager.py:177(__get__)
13813188/13811671    5.794    0.000    5.808    0.000 {method 'update' of 'dict' objects}
   132919    0.573    0.000    5.805    0.000 video_events.py:158(get_context_extensions)
       50    0.001    0.000    5.794    0.116 messaging.py:114(publish)
   748005    2.512    0.000    5.777    0.000 utils.py:237(_route_db)
1496013/748006    2.068    0.000    5.771    0.000 where.py:165(_contains_aggregate)
   249886    0.734    0.000    5.712    0.000 python.py:222(build_date)
  1510835    3.284    0.000    5.695    0.000 sync.py:497(get_current_task)
   871511    2.341    0.000    5.663    0.000 signals.py:17(_get_current_user_from_request)
  1743022    5.647    0.000    5.647    0.000 __init__.py:435(_format)
   748014    3.309    0.000    5.471    0.000 query_utils.py:174(get_lookup)
   748008    2.938    0.000    5.470    0.000 query.py:2462(update_join_types)
   748007    1.524    0.000    5.404    0.000 utils.py:95(inner)
       77    5.292    0.069    5.292    0.069 {method 'connect' of '_socket.socket' objects}
   145470    0.613    0.000    5.183    0.000 result.py:53(__init__)
   748006    5.162    0.000    5.162    0.000 {method 'describe' of '_mysql.result' objects}
1025067/1025053    1.015    0.000    5.042    0.000 {built-in method builtins.any}
   249969    0.249    0.000    4.984    0.000 models.py:68(_strip_value)
   249886    1.567    0.000    4.979    0.000 __init__.py:299(range_check_date)
   249283    0.725    0.000    4.970    0.000 context_activities.py:28(__init__)
  1496014    2.466    0.000    4.833    0.000 cursors.py:128(nextset)
   250000    1.034    0.000    4.763    0.000 transformer.py:135(get_context_extensions)
 20222768    4.744    0.000    4.744    0.000 query.py:2293(extra_select)
   249969    0.517    0.000    4.736    0.000 models.py:59(_strip_object)
1247843/997926    0.716    0.000    4.733    0.000 expressions.py:247(<listcomp>)
  5801797    4.703    0.000    4.703    0.000 {method 'match' of 're.Pattern' objects}
  6055047    1.708    0.000    4.696    0.000 abc.py:96(__instancecheck__)
   744357    1.026    0.000    4.649    0.000 user_util.py:24(get_all_retired_usernames)
  4807564    4.627    0.000    4.627    0.000 {method 'format' of 'str' objects}
  1743030    4.613    0.000    4.613    0.000 {built-in method time.strftime}
  1247938    2.232    0.000    4.577    0.000 where.py:142(clone)
1742335/748009    2.295    0.000    4.533    0.000 query.py:1108(resolve_lookup_value)
  1249540    2.538    0.000    4.526    0.000 {method '__reduce_ex__' of 'object' objects}
   249886    1.122    0.000    4.510    0.000 __init__.py:339(range_check_time)
 25202714    4.440    0.000    4.440    0.000 {method 'items' of 'dict' objects}
   748006    1.999    0.000    4.406    0.000 compiler.py:784(get_from_clause)
   748008    2.029    0.000    4.360    0.000 query.py:2440(__init__)
  1743048    4.289    0.000    4.289    0.000 {built-in method time.localtime}
   249970    1.009    0.000    4.289    0.000 __init__.py:269(__init__)
   250076    1.031    0.000    4.197    0.000 uuid.py:780(uuid4)
   748044    1.988    0.000    4.182    0.000 query_utils.py:68(__init__)
1755415/1750806    1.587    0.000    4.173    0.000 <frozen importlib._bootstrap>:1017(_handle_fromlist)
    16824    0.132    0.000    4.139    0.000 video_events.py:223(get_result)
   748007    1.451    0.000    4.063    0.000 base.py:259(create_cursor)
   748009    0.704    0.000    4.042    0.000 query.py:2455(add_votes)
   249917    0.802    0.000    4.041    0.000 compiler.py:771(_setup_joins)
      751    0.004    0.000    4.028    0.005 response.py:398(_decode)
      647    0.160    0.000    4.024    0.006 response.py:88(decompress)
   498088    1.218    0.000    3.973    0.000 lookups.py:45(batch_process_rhs)
   748007    1.319    0.000    3.962    0.000 __init__.py:674(db_type_parameters)
  1496012    1.262    0.000    3.925    0.000 cursors.py:78(close)
      647    3.864    0.006    3.864    0.006 {method 'decompress' of 'zlib.Decompress' objects}
   872697    1.361    0.000    3.798    0.000 posixpath.py:117(splitext)
8356/1747    0.023    0.000    3.786    0.002 <frozen importlib._bootstrap>:986(_find_and_load)
   748008    2.392    0.000    3.776    0.000 query.py:971(join)
7188/1232    0.016    0.000    3.767    0.003 <frozen importlib._bootstrap>:956(_find_and_load_unlocked)
  1244244    1.879    0.000    3.744    0.000 connections.py:266(literal)
  1496013    0.845    0.000    3.739    0.000 where.py:168(<genexpr>)
 5980/652    0.015    0.000    3.706    0.006 <frozen importlib._bootstrap>:650(_load_unlocked)
    52599    0.280    0.000    3.692    0.000 problem_interaction_events.py:73(get_object)
 5782/650    0.008    0.000    3.688    0.006 <frozen importlib._bootstrap_external>:837(exec_module)
  1496275    1.408    0.000    3.681    0.000 __init__.py:608(update)
 8000/633    0.003    0.000    3.672    0.006 <frozen importlib._bootstrap>:211(_call_with_frames_removed)
    48312    0.317    0.000    3.665    0.000 navigation_events.py:99(get_object)
   748006    2.218    0.000    3.614    0.000 cursors.py:368(fetchmany)
  2244029    2.786    0.000    3.613    0.000 tree.py:78(add)
   249886    0.312    0.000    3.608    0.000 iso8601.py:225(jsonify_datetime)
2778/1698    0.002    0.000    3.608    0.002 __init__.py:109(import_module)
3011/1698    0.002    0.000    3.606    0.002 <frozen importlib._bootstrap>:1002(_gcd_import)
      100    0.001    0.000    3.579    0.036 json.py:64(dumps)
   748009    0.690    0.000    3.547    0.000 query.py:376(_get_col)
   249886    1.982    0.000    3.517    0.000 date.py:59(parse_date)
  2644495    1.867    0.000    3.499    0.000 saferepr.py:149(_format_chars)
  1249535    1.333    0.000    3.456    0.000 copyreg.py:90(__newobj__)
   748005    1.806    0.000    3.397    0.000 lookups.py:210(get_db_prep_lookup)
   750735    2.125    0.000    3.393    0.000 options.py:365(swapped)
   749658    1.478    0.000    3.387    0.000 python.py:77(fractional_range_check)
  4240083    3.383    0.000    3.383    0.000 tree.py:21(__init__)
  1743022    1.178    0.000    3.369    0.000 __init__.py:633(usesTime)
  3486105    2.128    0.000    3.319    0.000 __init__.py:898(acquire)
       50    0.003    0.000    3.294    0.066 connection.py:513(_ensured)
       50    0.001    0.000    3.287    0.066 messaging.py:183(_publish)
  2573386    1.574    0.000    3.283    0.000 base_transformer.py:35(find_nested)
   249970    0.596    0.000    3.279    0.000 locator.py:127(parse_url)
   249886    3.172    0.000    3.277    0.000 {method 'isoformat' of 'datetime.datetime' objects}
  1692104    2.389    0.000    3.230    0.000 models.py:19(get_value_from_dotted_path)
   249886    2.114    0.000    3.218    0.000 base_transformer.py:188(del_none)
       50    0.001    0.000    3.200    0.064 base.py:591(basic_publish)
 13981460    3.165    0.000    3.165    0.000 {method 'copy' of 'dict' objects}
   499938    1.323    0.000    3.148    0.000 __init__.py:313(replace)
   873407    1.755    0.000    3.147    0.000 posixpath.py:140(basename)
   249917    1.724    0.000    3.090    0.000 expressions.py:1201(as_sql)
  1487564    1.023    0.000    3.062    0.000 verb.py:56(id)
   249886    0.826    0.000    3.021    0.000 python.py:405(build_timezone)
  6055047    2.981    0.000    2.988    0.000 {built-in method _abc._abc_instancecheck}
   496238    1.325    0.000    2.988    0.000 user_util.py:11(_compute_retired_hash)
  2320926    1.774    0.000    2.948    0.000 os.py:748(encode)
     1472    0.001    0.000    2.943    0.002 s3.py:332(iterate_container_objects)
        4    0.000    0.000    2.941    0.735 http.py:228(request)
        4    0.000    0.000    2.940    0.735 sessions.py:500(request)
   249917    1.069    0.000    2.873    0.000 compiler.py:769(<listcomp>)
 18726753    2.815    0.000    2.815    0.000 query.py:1019(join_parent_model)
   748008    1.444    0.000    2.773    0.000 base.py:544(validate_thread_sharing)
  8723935    1.932    0.000    2.753    0.000 saferepr.py:81(_chaindict)
1503531/1501065    2.245    0.000    2.752    0.000 deconstruct.py:14(__new__)
  8852431    2.716    0.000    2.723    0.000 {built-in method __new__ of type object at 0x7fca36b379a0}
  1999088    1.140    0.000    2.694    0.000 __init__.py:48(cast)
   249969    0.599    0.000    2.691    0.000 locator.py:339(for_branch)
        2    0.000    0.000    2.690    1.345 __init__.py:8(setup)
        1    0.000    0.000    2.686    2.686 startup.py:13(run)
        2    0.001    0.000    2.652    1.326 registry.py:61(populate)
        1    0.000    0.000    2.647    2.647 transform_tracking_logs.py:143(validate_source_and_files)
   748007    2.643    0.000    2.643    0.000 datastructures.py:260(__init__)
   871511    1.063    0.000    2.629    0.000 signals.py:26(_get_current_user_from_thread_locals)
   748006    1.708    0.000    2.621    0.000 query.py:854(reset_refcounts)
   249916    0.642    0.000    2.593    0.000 query.py:1229(ordered)
      821    0.006    0.000    2.578    0.003 __init__.py:1(<module>)
      131    0.002    0.000    2.566    0.020 models.py:1(<module>)
   249917    1.177    0.000    2.530    0.000 query.py:1985(add_ordering)
       50    0.000    0.000    2.498    0.050 messaging.py:247(_prepare)
       50    0.001    0.000    2.498    0.050 serialization.py:150(dumps)
 18726756    2.497    0.000    2.497    0.000 __init__.py:408(select_format)
  1743025    0.775    0.000    2.469    0.000 __init__.py:28(get_current_request)
  1521481    1.655    0.000    2.436    0.000 <frozen importlib._bootstrap>:389(parent)
  1510835    1.439    0.000    2.411    0.000 tasks.py:34(current_task)
   250000    0.284    0.000    2.405    0.000 base_transformer.py:77(transformer_version)
  5040929    2.391    0.000    2.391    0.000 {method 'split' of 'str' objects}
  2382351    1.873    0.000    2.372    0.000 threading.py:1306(current_thread)
 18726754    2.356    0.000    2.356    0.000 operations.py:564(get_db_converters)
24760470/24760401    2.289    0.000    2.355    0.000 {method 'extend' of 'list' objects}
   748266    1.990    0.000    2.332    0.000 __init__.py:540(__init__)
       25    0.044    0.002    2.304    0.092 redis.py:1007(_put)
   249917    1.011    0.000    2.288    0.000 lookups.py:289(process_rhs)
   249970    0.502    0.000    2.267    0.000 __init__.py:306(_unchecked_init)
   992478    1.301    0.000    2.260    0.000 __init__.py:1072(get_prep_value)
   748006    2.112    0.000    2.249    0.000 compiler.py:1115(apply_converters)
  1743022    2.245    0.000    2.245    0.000 __init__.py:364(getMessage)
   748007    0.840    0.000    2.243    0.000 connections.py:241(cursor)
5375368/5375316    2.208    0.000    2.208    0.000 {method 'encode' of 'str' objects}
  1999088    1.414    0.000    2.199    0.000 __init__.py:63(range_check)
   872697    1.687    0.000    2.198    0.000 genericpath.py:121(_splitext)
  1749790    1.600    0.000    2.196    0.000 __init__.py:330(__setattr__)
  1743022    1.340    0.000    2.191    0.000 __init__.py:427(usesTime)
  1050934    0.820    0.000    2.176    0.000 activity.py:57(id)
2535/1300    0.004    0.000    2.160    0.002 {built-in method builtins.__import__}
  1247938    1.278    0.000    2.153    0.000 tree.py:29(_new_instance)
9779546/9779533    2.124    0.000    2.124    0.000 {built-in method builtins.iter}
       77    0.001    0.000    2.001    0.026 socket.py:901(getaddrinfo)
       77    1.997    0.026    1.997    0.026 {built-in method _socket.getaddrinfo}
  1247855    1.268    0.000    1.984    0.000 expressions.py:165(__getstate__)
   499962    1.931    0.000    1.931    0.000 uuid.py:273(__str__)
  1492414    1.019    0.000    1.928    0.000 query.py:1116(<genexpr>)
   249880    1.475    0.000    1.905    0.000 events_router.py:31(configure_host)
      191    0.000    0.000    1.886    0.010 config.py:294(import_models)
   249886    1.074    0.000    1.871    0.000 timezone.py:14(parse_timezone)
   748014    1.430    0.000    1.859    0.000 query_utils.py:164(_get_lookup)
  1496016    1.615    0.000    1.845    0.000 query.py:835(demote_joins)
   997928    1.814    0.000    1.814    0.000 query.py:1704(trim_joins)
  3486105    1.328    0.000    1.808    0.000 __init__.py:905(release)
  3754860    1.796    0.000    1.796    0.000 {built-in method _asyncio.get_running_loop}
   748005    0.785    0.000    1.770    0.000 db_routers.py:25(db_for_read)
2962268/2573386    1.512    0.000    1.709    0.000 base_transformer.py:47(_find_nested)
  1492414    0.862    0.000    1.702    0.000 lookups.py:266(<genexpr>)
   748007    0.785    0.000    1.664    0.000 cursors.py:325(_fetch_row)
        1    0.000    0.000    1.660    1.660 base.py:410(check)
        1    0.000    0.000    1.660    1.660 registry.py:65(run_checks)
   871511    0.775    0.000    1.642    0.000 filters.py:14(filter)
   268902    0.680    0.000    1.608    0.000 helpers.py:174(convert_seconds_to_float)
1255235/1255232    0.846    0.000    1.602    0.000 {built-in method builtins.all}
      686    0.012    0.000    1.595    0.002 enrollment_events.py:28(get_object)
   249943    0.954    0.000    1.584    0.000 hashable.py:4(make_hashable)
   746208    1.135    0.000    1.569    0.000 datastructures.py:265(__getitem__)
   249886    1.232    0.000    1.554    0.000 python.py:102(_cast_to_fractional_component)
    19683    0.072    0.000    1.543    0.000 problem_interaction_events.py:135(get_object)
   350877    0.520    0.000    1.528    0.000 base_transformer.py:202(get_object_iri)
2263311/2262912    0.906    0.000    1.510    0.000 {built-in method builtins.issubclass}
   249970    0.752    0.000    1.501    0.000 __init__.py:222(get_namespace_plugin)
   996240    1.490    0.000    1.490    0.000 {built-in method _hashlib.openssl_sha1}
   249920    0.564    0.000    1.471    0.000 lookups.py:73(get_prep_lookup)
    48312    0.235    0.000    1.463    0.000 navigation_events.py:116(get_context_extensions)
   249886    0.452    0.000    1.449    0.000 calendar.py:120(monthrange)
  1496018    0.767    0.000    1.444    0.000 query.py:1175(check_filterable)
   250072    1.065    0.000    1.436    0.000 fields.py:369(bind)
  1997673    1.423    0.000    1.425    0.000 options.py:587(get_field)
   249916    0.455    0.000    1.425    0.000 query.py:1405(__instancecheck__)
    25268    0.020    0.000    1.422    0.000 {method 'join' of 'bytes' objects}
   755674    1.270    0.000    1.420    0.000 {built-in method builtins.sorted}
      178    0.000    0.000    1.416    0.008 retry.py:35(call_with_retry)
      102    0.001    0.000    1.408    0.014 client.py:1251(execute_command)
   249886    1.011    0.000    1.407    0.000 python.py:686(_distribute_microseconds)
   748007    1.404    0.000    1.404    0.000 cursors.py:65(__init__)
       28    0.000    0.000    1.402    0.050 models.py:887(content)
      106    0.001    0.000    1.401    0.013 response.py:789(read_chunked)
    15840    0.042    0.000    1.400    0.000 helpers.py:256(get_problem_block_id)
   743782    0.507    0.000    1.391    0.000 verb.py:76(display)
      102    0.000    0.000    1.385    0.014 client.py:1259(<lambda>)
      102    0.000    0.000    1.385    0.014 client.py:1230(_send_command_parse_response)
   249969    0.323    0.000    1.379    0.000 locator.py:318(version_agnostic)
   250000    0.326    0.000    1.378    0.000 transformer.py:88(get_timestamp)
  2244019    0.847    0.000    1.375    0.000 base.py:91(__getattr__)
  1242446    1.371    0.000    1.371    0.000 {method 'string_literal' of '_mysql.connection' objects}
   748007    0.982    0.000    1.366    0.000 base.py:588(make_cursor)
  1009574    0.816    0.000    1.343    0.000 language_map.py:34(<lambda>)
   499169    0.699    0.000    1.332    0.000 activity_definition.py:126(type)
   249916    0.687    0.000    1.332    0.000 operations.py:225(limit_offset_sql)
    31674    0.133    0.000    1.325    0.000 problem_interaction_events.py:249(get_result)
    19683    0.101    0.000    1.317    0.000 problem_interaction_events.py:146(get_result)
       25    0.000    0.000    1.313    0.053 core.py:2689(lpush)
        2    0.000    0.000    1.312    0.656 courses.py:1(<module>)
  4123914    1.301    0.000    1.301    0.000 {method 'replace' of 'str' objects}
   249886    0.278    0.000    1.296    0.000 python.py:65(year_range_check)
    15840    0.086    0.000    1.292    0.000 helpers.py:212(get_block_id_from_event_referrer)
   748005    0.864    0.000    1.279    0.000 compiler.py:1095(deferred_to_columns)
   253859    0.496    0.000    1.267    0.000 __init__.py:79(__getattr__)
    35517    0.099    0.000    1.263    0.000 result.py:75(score)
  1622233    1.262    0.000    1.262    0.000 __init__.py:1689(isEnabledFor)
     7142    0.030    0.000    1.255    0.000 {method 'read' of '_io.BufferedReader' objects}
   249969    0.474    0.000    1.243    0.000 __init__.py:161(__str__)
   499077    0.704    0.000    1.241    0.000 datastructures.py:10(__init__)
   748452    0.382    0.000    1.233    0.000 activity.py:77(object_type)
   249918    0.466    0.000    1.231    0.000 lookups.py:89(process_rhs)
      104    0.001    0.000    1.227    0.012 response.py:767(_handle_chunk)
      208    0.000    0.000    1.227    0.006 client.py:606(_safe_read)
   499885    0.773    0.000    1.223    0.000 setting_toggle.py:20(is_enabled)
  2788805    1.212    0.000    1.212    0.000 {method 'find' of 'str' objects}
   498088    0.799    0.000    1.208    0.000 datastructures.py:19(discard)
   871515    0.867    0.000    1.208    0.000 __init__.py:160(<lambda>)
   249886    1.206    0.000    1.206    0.000 utcoffset.py:13(__init__)
   499772    0.466    0.000    1.198    0.000 statement.py:79(id)
  3489707    1.195    0.000    1.195    0.000 {method 'acquire' of '_thread.RLock' objects}
269024/251682    0.869    0.000    1.187    0.000 copy.py:128(deepcopy)
   250000    0.669    0.000    1.185    0.000 registry.py:75(get_transformer)
   501004    0.985    0.000    1.179    0.000 fields.py:660(__new__)
   249283    0.711    0.000    1.179    0.000 typed_list.py:27(__init__)
  3740035    1.176    0.000    1.176    0.000 cursors.py:165(_get_db)
        1    0.000    0.000    1.156    1.156 cohorts.py:1(<module>)
   748008    1.154    0.000    1.154    0.000 base.py:529(allow_thread_sharing)
   873245    0.734    0.000    1.145    0.000 dispatcher.py:12(_make_id)
  4737961    1.129    0.000    1.129    0.000 query.py:297(get_meta)
   748006    1.124    0.000    1.124    0.000 compiler.py:29(__init__)
    35517    0.128    0.000    1.118    0.000 score.py:42(__init__)
  5741230    1.101    0.000    1.101    0.000 query.py:194(query)
  2658643    1.089    0.000    1.089    0.000 {method 'rfind' of 'str' objects}
14114/13154    0.185    0.000    1.086    0.000 {built-in method builtins.__build_class__}
  5517998    1.059    0.000    1.059    0.000 {method 'pop' of 'dict' objects}
        2    0.000    0.000    1.055    0.528 s3.py:379(get_container)
   249880    0.931    0.000    1.053    0.000 base.py:586(_get_pk_val)
   748005    1.051    0.000    1.051    0.000 compiler.py:1131(<listcomp>)
      252    0.001    0.000    1.047    0.004 client.py:1268(parse_response)
      253    0.000    0.000    1.046    0.004 connection.py:859(read_response)
   250000    0.228    0.000    1.045    0.000 base_transformer.py:121(extract_username_or_userid)
      253    0.001    0.000    1.045    0.004 connection.py:343(read_response)
      255    0.002    0.000    1.044    0.004 connection.py:191(_read_from_socket)
  353/253    0.001    0.000    1.043    0.004 connection.py:355(_read_response)
      353    0.001    0.000    1.042    0.003 connection.py:254(readline)
      255    1.041    0.004    1.041    0.004 {method 'recv' of '_socket.socket' objects}
   748008    0.889    0.000    1.031    0.000 query.py:768(table_alias)
        4    0.000    0.000    1.024    0.256 base.py:131(__init__)
   499169    0.402    0.000    1.014    0.000 activity.py:92(definition)
  1496014    1.003    0.000    1.003    0.000 {method 'next_result' of '_mysql.connection' objects}
  3430552    0.999    0.000    0.999    0.000 {built-in method builtins.vars}
   249886    0.868    0.000    0.998    0.000 calendar.py:113(weekday)
   748005    0.998    0.000    0.998    0.000 read_replica.py:125(db_for_read)
       12    0.000    0.000    0.997    0.083 models.py:906(text)
        6    0.000    0.000    0.997    0.166 models.py:790(apparent_encoding)
        6    0.000    0.000    0.997    0.166 __init__.py:30(detect)
        6    0.000    0.000    0.996    0.166 universaldetector.py:154(feed)
   249886    0.445    0.000    0.989    0.000 __init__.py:438(range_check_timezone)
   748005    0.985    0.000    0.985    0.000 db_routers.py:17(_use_blockstore)
        6    0.705    0.117    0.976    0.163 utf1632prober.py:187(feed)
   748007    0.732    0.000    0.965    0.000 cursors.py:377(fetchall)
    22362    0.139    0.000    0.964    0.000 video_events.py:281(get_result)
       53    0.001    0.000    0.959    0.018 api.py:1(<module>)
   249886    0.438    0.000    0.948    0.000 statement.py:104(object)
   499938    0.638    0.000    0.945    0.000 __init__.py:321(<dictcomp>)
   748267    0.925    0.000    0.926    0.000 {built-in method _collections._count_elements}
   871516    0.623    0.000    0.920    0.000 __init__.py:119(getLevelName)
   750717    0.518    0.000    0.916    0.000 __init__.py:1424(debug)
  4023370    0.916    0.000    0.916    0.000 {built-in method _thread.get_ident}
   249283    0.365    0.000    0.905    0.000 context_activities.py:67(parent)
   748005    0.710    0.000    0.898    0.000 query.py:2008(get_related_populators)
   748006    0.721    0.000    0.895    0.000 datastructures.py:166(as_sql)
   275998    0.893    0.000    0.893    0.000 {method 'search' of 're.Pattern' objects}
   748015    0.675    0.000    0.891    0.000 query.py:1396(_not_support_combined_queries)
   748006    0.731    0.000    0.889    0.000 compiler.py:706(get_distinct)
   871983    0.886    0.000    0.886    0.000 {built-in method posix.getpid}
   815570    0.880    0.000    0.880    0.000 {built-in method builtins.min}
   748006    0.880    0.000    0.880    0.000 {method 'fetch_row' of '_mysql.result' objects}
   249916    0.329    0.000    0.877    0.000 query.py:1865(is_empty)
    14169    0.059    0.000    0.865    0.000 navigation_events.py:53(get_object)
   250000    0.230    0.000    0.861    0.000 base_transformer.py:136(extract_sessionid)
   498088    0.705    0.000    0.861    0.000 lookups.py:217(<listcomp>)
   871918    0.597    0.000    0.860    0.000 posixpath.py:52(normcase)
   748007    0.674    0.000    0.855    0.000 lookups.py:135(contains_aggregate)
   994326    0.598    0.000    0.840    0.000 lookups.py:252(resolve_expression_parameter)
   249886    0.445    0.000    0.823    0.000 __init__.py:584(build_date)
  1499189    0.808    0.000    0.808    0.000 {method 'update' of 'set' objects}
      112    0.002    0.000    0.795    0.007 urls.py:1(<module>)
  1563744    0.789    0.000    0.789    0.000 {method 'rpartition' of 'str' objects}
    15834    0.035    0.000    0.782    0.000 problem_interaction_events.py:233(_get_interaction_type)
   249919    0.214    0.000    0.781    0.000 query.py:1638(final_transformer)
   748006    0.781    0.000    0.781    0.000 {method 'field_flags' of '_mysql.result' objects}
   249969    0.670    0.000    0.766    0.000 serializer_helpers.py:39(__init__)
  5499319    0.753    0.000    0.753    0.000 __init__.py:2149(get_internal_type)
   249970    0.535    0.000    0.741    0.000 locator.py:279(<dictcomp>)
   500001    0.430    0.000    0.739    0.000 uuid.py:278(bytes)
  4982475    0.738    0.000    0.738    0.000 __init__.py:1064(get_internal_type)
        1    0.000    0.000    0.737    0.737 urls.py:8(check_url_config)
   2832/1    0.001    0.000    0.737    0.737 urls.py:17(check_resolver)
    423/1    0.002    0.000    0.737    0.737 resolvers.py:414(check)
258964/255835    0.437    0.000    0.728    0.000 functional.py:244(inner)
   424785    0.726    0.000    0.726    0.000 extensions.py:28(__init__)
   498914    0.725    0.000    0.725    0.000 {method 'hexdigest' of '_hashlib.HASH' objects}
   748008    0.718    0.000    0.718    0.000 query.py:801(promote_joins)
   249969    0.496    0.000    0.714    0.000 locator.py:354(_to_string)
   891654    0.500    0.000    0.713    0.000 posixpath.py:41(_get_sep)
   250360    0.706    0.000    0.706    0.000 {method 'groupdict' of 're.Match' objects}
   748006    0.490    0.000    0.702    0.000 operations.py:329(compiler)
   250000    0.320    0.000    0.700    0.000 agent_account.py:48(name)
   748005    0.687    0.000    0.687    0.000 query.py:36(__init__)
     5782    0.030    0.000    0.686    0.000 <frozen importlib._bootstrap_external>:909(get_code)
  1496019    0.681    0.000    0.681    0.000 tree.py:60(__bool__)
       50    0.000    0.000    0.680    0.014 base.py:601(_inplace_augment_message)
    25001    0.145    0.000    0.679    0.000 parse.py:369(urlparse)
       50    0.000    0.000    0.678    0.014 base.py:785(encode_body)
       50    0.017    0.000    0.678    0.014 base.py:64(encode)
    26515    0.050    0.000    0.674    0.000 parse.py:662(parse_qs)
   250028    0.474    0.000    0.668    0.000 extension.py:333(__getitem__)
   996179    0.468    0.000    0.644    0.000 lookups.py:103(rhs_is_direct_value)
   871516    0.643    0.000    0.643    0.000 threading.py:1031(name)