Effective processing of data. (Probably making SQL queries)

[Molozey]

TEST CASE: 40_000 record updates about limited orderbook with size 2. In cycle with size 10_000 (4 line every loop)

REFERENCE TIME OF CYCLE WITHOUT PROCESSING DATA (no making queries):

---

Process records. Make record to db. size of one batch is 400:

---

Process records. No record to db. size of one batch is 400:

---

Process records. No record to db. size of one batch is 40:

---

Process records.Make record to db. size of one batch is 40:

---

[Molozey]

I tried to make this with numba, but it's not easy (how i get, numba doesn't support easy concatenation for strings). file with creation of query is: OrderBookScrapper/DataBase/mysqlRecording/postDataTemplateLimited.py. It iterates from all pandas dataframe line and construct "line query" from values.

If i disable iteration for lines:

Process records.Make record to db. size of one batch is 40 (ONLY FIRST LINE):