Investigate why is the app so slow

[kubantjan]

Find out what are the bottlenecks. Propose solutions.

[kubantjan]

Sem prosim psat vystupy investigace @krllstdn

[krllstdn]

1. Optimise get_hla_antibodies_from_recipient_model() #1045

On the image generated with snakeviz library is shown the time that takes a function call. As can be seen the function _from_dict () takes too much time to complete. get_hla_antibodies_from_recipient_model() is used almost everywhere, so if we optimize it we might significantly increase speed of the whole application. On the image above are shown three endpoint calls that are made in calcucate page.

1. http://localhost:8080/v1/txm-event/3/patients/configs/default
2. http://localhost:8080/v1/txm-event/3/configuration/default
3. http://localhost:8080/v1/txm-event/3/matching/calculate-for-config

I decided to code something that will be faster than _from_dict() function. And so I did. You can see results in PR #1045

So maybe it is worth checking other cases of Dacite library usage. #1047

[kubantjan]

Zkontrolovat jestli nevolame neco zbytecne

[kubantjan]

Next steps:

1) zkontorlovan jestli nevolame veci zbytecne (get_hla_antibodies_from_recipient_model ta vypada ze ji volame treba v http://localhost:8080/v1/txm-event/3/patients/configs/default kde to vypada, ze to k nicemu neni? 2) zkontrolvoat jestli nevolame v endpoitech veci nekolikrat 3) get_hla_antibodies_from_recipient_model vypada ze fakt trva dlouho - udelat issue na jeji zrychleni 4) sem pridat profil toho kdyz otevru matchings stranku a patients stranku detailni

[krllstdn]

Bod 1:

1. V http://localhost:8080/v1/txm-event/3/configuration/default se vola cely get_txm_event_complete() i pres to ze by stacilo get_txm_event_base(). Na strance Matchings to zrychlilo ten endpoint call z 746ms (na obrazku) do 30ms
2. http://localhost:8080/v1/txm-event/3/patients/configs/default tam musi byt (jestli ji blokuju v Developer Tools tak hned vsechno padne)

Bod 2

V trech zminenych endpointech narocne veci se nevolaji vicekrat.

Bod 3:

1045

Bod 4

Matchings (po optimalizaci)

Patients (po optimalizaci)

[krllstdn]

@kubantjan chci opravit bod 1 v bode 1 v me odpovedi. Potrebuju na to vytvorit issue i PR? nebo staci proste pridat do #1046? tam opravy na jeden radek, ale vytvaret issue a PR mi pride zbytecny. Jak to udelat nejlip?

[kubantjan]

@krllstdn klidně to přidej

[kubantjan]

• rozpadenout ty 4 endpointy a shrnout co na nich trva dlouho
• muzes kratky zamysleni jestli se neco nevola zbytecne.

[krllstdn]

Matchings page

total time: 3,02s

1. Calculate for config

By function call in API

get_txm_event_complete()

• get_recipient_from_recipient_model() - by some reason there's called _get_base_patient_from_patient_model() that takes 0,5s. I don't see the reason why should it be called (all information is already in RecipientModel) - potentiallow hanging fruit
  • query to db takes 0,4s. it might be possible to optimise it, but it is complicated if even possible.
• _get_txm_event_from_txm_event_model - takes 1,17s . there's used for loop. so if we make a list comprehension we can see some performance boost. I tried, it is not so easy to create a comprehension loop, but it might be worth a try.
• get_hla_typing_from_patient_model() - two cases of dacite.from_dict() usage. low-hanging fruit it takes0.475s

  Solver itself.

  Probably it can be optimized.

2. http://localhost:8080/v1/txm-event/3/patients/configs/default (gets all patients for txm event 3 in 1,32s)

get_txm_event_complete()

the same as above. two "low-hanging fruits".

3. http://localhost:8080/v1/txm-event (gets all txm events without patients)

It could be faster, but I haven't seen nothing in particular that could be improved easily.

Other endpoints are already extremely fast.

Conclusions

Things that can be improved very fast and potentially bring further noticable iincrease in speed are

• get_hla_typing_from_patient_model() - two cases of dacite.from_dict() usage. Write some code that will do the same with list comprehension and it should be faster.
• get_recipient_from_recipient_model() - by some reason there's called _get_base_patient_from_patient_model() and seems that there's no point in that. It can be rewritten without _get_base_patient_from_patient_model() .

[krllstdn]

@kubantjan v conclusions jsou action items, ktere muzou byt udelane dost rychle

[kubantjan]

get_hla_typing_from_patient_model() vidím tam že píšeš že by šlo zrychlit ale nevidím kolik času to teď trvá

[krllstdn]

Opraveno. Trva to 0.475s. Skoro cely ten cas zabira from_dict(), tak ze teoriticky by se dalo optimalizovat a to by mohlo trvat 0.1-0.2 s. Je to dostatecny cas pro optimalizaci nebo na to vykaslame?

[kubantjan]

Jop to stojí za to udělaj issue a rovnou na něj mrkni

[krllstdn]

Patients page

Here it is a bit slower due to include-antibodies raw.

http://localhost:8080/v1/txm-event/3/patients/configs/default?include-antibodies-raw

this include-antibodies-raw makes it slower

• get_txm_event_complete()
  • 
• to_lists_for_fe()
  • 

Queries take the biggest chunk of its completion time, so probably no easy solutions here.

[krllstdn]

rewriting get_recipient_from_recipient_model() without patient_base had no impact on speed. And from point of readability it also does not make much sense, so this was a bad idea.

[krllstdn]

Dacite

dacite is also used in

• patient_upload_service.py

• scorer_service.py

• config_service.py. but it doesn't seem to have any significant impact on speed, so probably it doesn't matter

  GET.v1.txm-event.3.patients.configs.default

• from_dict porad nekde je a zere 0.5s

• response_ok probably could be faster, but it is flask's function so probably it can't be optimised by us

[kubantjan]

So the conclusion now is that if we want to focus on speedup we shall focus on sql queries. Is that right? @krllstdn ?

[krllstdn]

Yes, this is the slowest thing that there is.