search

elixir-ecto / myxql

MySQL 5.5+ driver for Elixir
Apache License 2.0
273 stars 67 forks source link

Tests run slower after switching to myxql #60

Closed michaelst closed 4 years ago

michaelst commented 5 years ago

We are noticing a significant (8x) slow down on our tests with the myxql driver when compared to mariaex. Our average test time is 200s with mariaex but 1600s with myxql. My coworker also noticed a similar performance impact when trying out the postgrex driver.

Is this a problem with our configuration or is this a known problem?

wojtekmach commented 5 years ago

Thank you for the report. This is unexpected. One potential reason is we dont yet cache inserts so if you make a lot of those (eg using ex_machina and such) that could be a factor. If youre able to reproduce this in something I can run locally that would be appreciated, otherwise I’ll look into it in a couple of days.

Wiadomość napisana przez Michael St Clair notifications@github.com w dniu 21.05.2019, o godz. 13:35:

We are noticing a significant (8x) slow down on our tests with the myxql driver when compared to mariaex. Our average test time is 200s with mariaex but 1600s with myxql. My coworker also noticed a similar performance impact when trying out the postgrex driver.

Is this a problem with our configuration or is this a known problem?

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub, or mute the thread.

michaelst commented 5 years ago

We definitely do a ton of inserts on our tests, but how do you cache an insert? Not sure if I would be able to provide anything of our code for that.

wojtekmach commented 5 years ago

By caching I meant preparing a particular insert statement once and then executing it multiple times, instead of doing prepare+execute every time. We have an issue for this, but since youve run into a problem I’ll prioritize it.

Wiadomość napisana przez Michael St Clair notifications@github.com w dniu 21.05.2019, o godz. 16:05:

We definitely do a ton of inserts on our tests, but how do you cache an insert? Not sure if I would be able to provide anything of our code for that.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub, or mute the thread.

michaelst commented 5 years ago

Great thanks!

michaelst commented 5 years ago

So something interesting I found is that the tests run fine on my local machine and are not slowed down, but on our remote instances that run our automated tests they run extremely slow. I have tested this with 1.8.2 with both Erlang 21 and 22. My local machine is a Mac and the remote instances run Ubuntu 18.04. Are there any other data points you need that could help diagnose this problem?

josevalim commented 5 years ago

@michaelst are they slow for both mariaex and myxql or only myxql?

michaelst commented 5 years ago

Only myxql

josevalim commented 5 years ago

Yeah, then I would bet being the query cache as you have to do more round trips. I would disable the lru cache for mariaex and see how it degrades.

wojtekmach commented 5 years ago

@michaelst to make sure query cache (or lack thereof) is the culprit, could you try mariaex master and set prepare: :unnamed in your repo config? This disables LRU cache on Mariaex. Use the same option on MyXQL and compare results. Also, are you running latest MyXQL? What server version are you using?

michaelst commented 5 years ago

ok working on that now, which server specifically?

wojtekmach commented 5 years ago

Oh, I was just wondering what mysql server version you were testing it against already just in case that’s a factor.

On 28 May 2019, at 22:21, Michael St Clair notifications@github.com wrote:

ok working on that now, which server specifically?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/elixir-ecto/myxql/issues/60?email_source=notifications&email_token=AAASSJZVHLFN7FC35JVS2QLPXWH6DA5CNFSM4HONG4ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODWNKL6Q#issuecomment-496674298, or mute the thread https://github.com/notifications/unsubscribe-auth/AAASSJ2Q2TDBVI423VBJNRLPXWH6DANCNFSM4HONG4ZA.

michaelst commented 5 years ago

mysql: 5.7.26

all referencing GitHub master in mix.exs ecto-sql: 2ec71d74655cc076b6c965740a357dfe1505883a myxql: a0e9e53804f506e85600207729536426a038c103 mariaex: 5f4aa50dd4dc31fef99dea890ade21dfcea54f07

I ran the tests with this repo config

config :zipbooks, ZB.Repo,
  username: System.get_env("TESTING_DB_USERNAME") || "root",
  password: unless(System.get_env("TESTING_DB_PASSWORD") == "", do: System.get_env("TESTING_DB_PASSWORD")),
  database: System.get_env("TESTING_DB_DATABASE") || "zipbooks_test",
  hostname: System.get_env("TESTING_DB_HOSTNAME") || "127.0.0.1",
  pool: Ecto.Adapters.SQL.Sandbox,
  timeout: :infinity,
  ownership_timeout: :infinity,
  queue_target: 1_000,
  queue_interval: 5_000
  migration_timestamps: [inserted_at: :created_at],
  charset: "utf8mb4",
  collation: "utf8mb4_unicode_ci",
  pool_size: 10,
  prepare: :unnamed,
  log: false

Mariaex: Finished in 240.3 seconds MyXQL: I'm not sure what changed, but it has gotten worse, it has been almost an hour now and still not finished.

the only difference between the two runs was this change 

# use Ecto.Repo, otp_app: :zipbooks, adapter: Ecto.Adapters.MySQL
  use Ecto.Repo, otp_app: :zipbooks, adapter: Ecto.Adapters.MyXQL

Something I noticed is that with MyXQL we are getting deadlocks where we weren't before with Mariaex. Most of our tests we run with the async flag, with Mariaex there were no failures, but with MyXQL, 7 of the tests failed with a deadlock error. That may be contributing to the the extra time but definitely not all of it because the deadlocks only occur during the first 2 minutes.

Also, it seems like the tests get slower as it goes, the beginning tests seem to run just as fast as Mariaex, but then towards the end slow down considerably. Resource usage on the instance also seems to decrease as time goes on using very little memory and cpu.

michaelst commented 5 years ago

Here is a small section of tests that might help show the difference better 

mix test lib/zipbooks_web/controllers/transactions/chart_account_controller/ --trace
Excluding tags: [:skip]

ZB.Controllers.ChartAccount.IndexTest
  * test gets a list of chart accounts (8333.5ms)

ZB.Controllers.ChartAccount.UpdateTest
  * test change the parent and the change is propogated (2655.7ms)
  * test update a chart account (2792.1ms)
  * test cant update chart account with locked name (1313.1ms)
  * test cant archive a system chart account (1281.4ms)
  * test archive a chart account (1727.5ms)

ZB.Controllers.ChartAccount.ReassignTest
  * test reassign a chart account to an uncreated default (2006.2ms)
  * test reassign a chart account (1767.3ms)

ZB.Controllers.ChartAccount.ReconciliationMetaTest
  * test get reconciliation no permission (1199.7ms)
  * test get reconciliation meta (3679.3ms)

ZB.Controllers.ChartAccount.ShowTest
  * test get a chart account (2921.3ms)

ZB.Controllers.ChartAccount.CreateTest
  * test create using an uncreated system chart account as a template (2255.8ms)
  * test create a chart account with value and is_depreciable (2519.8ms)
  * test create a chart account (1951.6ms)

ZB.Controllers.ChartAccount.ImportTest
  * test import chart accounts csv (3182.3ms)

ZB.Controllers.ChartAccount.DeleteTest
  * test locked chart accounts archive instead of delete (1903.1ms)
  * test delete a chart account with no transactions (25491.2ms)
  * test deleting a chart account with transactions archives it (24777.2ms)

Finished in 91.9 seconds
18 tests, 0 failures

mix test lib/zipbooks_web/controllers/transactions/chart_account_controller/ --trace
Excluding tags: [:skip]

ZB.Controllers.ChartAccount.IndexTest
  * test gets a list of chart accounts (1481.4ms)

ZB.Controllers.ChartAccount.UpdateTest
  * test update a chart account (108.7ms)
  * test change the parent and the change is propogated (83.4ms)
  * test cant archive a system chart account (49.0ms)
  * test archive a chart account (57.5ms)
  * test cant update chart account with locked name (64.9ms)

ZB.Controllers.ChartAccount.ReconciliationMetaTest
  * test get reconciliation no permission (41.0ms)
  * test get reconciliation meta (151.4ms)

ZB.Controllers.ChartAccount.ReassignTest
  * test reassign a chart account to an uncreated default (85.6ms)
  * test reassign a chart account (58.6ms)

ZB.Controllers.ChartAccount.ShowTest
  * test get a chart account (155.8ms)

ZB.Controllers.ChartAccount.CreateTest
  * test create using an uncreated system chart account as a template (79.8ms)
  * test create a chart account with value and is_depreciable (88.2ms)
  * test create a chart account (68.5ms)

ZB.Controllers.ChartAccount.ImportTest
  * test import chart accounts csv (187.8ms)

ZB.Controllers.ChartAccount.DeleteTest
  * test locked chart accounts archive instead of delete (61.5ms)
  * test delete a chart account with no transactions (1346.3ms)
  * test deleting a chart account with transactions archives it (915.5ms)

Finished in 5.2 seconds
18 tests, 0 failures
josevalim commented 5 years ago

I was just going to ask you to run it with --trace, so great job. There is definitely something fundamental wrong here. MySQL is not safe to run in async: true and the sandbox because of deadlocks but I wouldn't expect the sandbox to behave so differently between adapters.

Btw, you can't reproduce this locally, right?

michaelst commented 5 years ago

Correct, locally it runs just as fast on both adapters. My local mysql version is 5.7.23 Homebrew

josevalim commented 5 years ago

And what is the OTP version? Using a more recent version causes the same errors?

michaelst commented 5 years ago

I have tried on OTP 21 and 22, same results

wojtekmach commented 5 years ago

Could you set protocol: :tcp on MyXQL repo config (by default we use UNIX domain socket which should be faster) just to get this one out of the way?

michaelst commented 5 years ago

same result

wojtekmach commented 5 years ago

@michaelst I was getting similar results when running benchmark on an Ubuntu VM, so it seems an OS might be a factor here.

We've added statement cache that made it so that on that benchmark myxql is as fast as mariaex.

Could you give myxql master a try?

michaelst commented 5 years ago

To isolate any other changes I ran things again with our latest master merged into the branch and the time is now cut in half (but still slower than the 5s it was before), not sure what changed. 

root@myxql-test:/srv/zipbooks/phoenix-api# mix test lib/zipbooks_web/controllers/transactions/chart_account_controller/ --trace
Excluding tags: [:skip]

ZB.Controllers.ChartAccount.IndexTest
  * test gets a list of chart accounts (2146.9ms)

ZB.Controllers.ChartAccount.UpdateTest
  * test archive a chart account (697.9ms)
  * test cant update chart account with locked name (619.2ms)
  * test cant archive a system chart account (597.8ms)
  * test update a chart account (1256.5ms)
  * test change the parent and the change is propogated (964.3ms)

ZB.Controllers.ChartAccount.ReassignTest
  * test reassign a chart account (885.2ms)
  * test reassign a chart account to an uncreated default (836.0ms)

ZB.Controllers.ChartAccount.ReconciliationMetaTest
  * test get reconciliation meta (1517.9ms)
  * test get reconciliation no permission (570.6ms)

ZB.Controllers.ChartAccount.ShowTest
  * test get a chart account (1326.2ms)

ZB.Controllers.ChartAccount.CreateTest
  * test create a chart account (696.8ms)
  * test create a chart account with value and is_depreciable (943.8ms)
  * test create using an uncreated system chart account as a template (822.0ms)

ZB.Controllers.ChartAccount.ImportTest
  * test import chart accounts csv (1019.5ms)

ZB.Controllers.ChartAccount.DeleteTest
  * test deleting a chart account with transactions archives it (13654.8ms)
  * test delete a chart account with no transactions (13921.8ms)
  * test locked chart accounts archive instead of delete (648.2ms)

Finished in 43.2 seconds
18 tests, 0 failures

Then I updated myxql from a0e9e53804f506e85600207729536426a038c103 -> 4a14841e13b94283c62db005550bd7ada1412bc8 and got about 3s faster 

root@myxql-test:/srv/zipbooks/phoenix-api# mix test lib/zipbooks_web/controllers/transactions/chart_account_controller/ --trace
Excluding tags: [:skip]

ZB.Controllers.ChartAccount.IndexTest
  * test gets a list of chart accounts (1935.5ms)

ZB.Controllers.ChartAccount.UpdateTest
  * test update a chart account (1236.4ms)
  * test change the parent and the change is propogated (808.0ms)
  * test cant archive a system chart account (557.5ms)
  * test archive a chart account (599.8ms)
  * test cant update chart account with locked name (571.2ms)

ZB.Controllers.ChartAccount.ReassignTest
  * test reassign a chart account to an uncreated default (828.8ms)
  * test reassign a chart account (824.1ms)

ZB.Controllers.ChartAccount.ReconciliationMetaTest
  * test get reconciliation no permission (557.9ms)
  * test get reconciliation meta (1469.3ms)

ZB.Controllers.ChartAccount.ShowTest
  * test get a chart account (1226.7ms)

ZB.Controllers.ChartAccount.CreateTest
  * test create using an uncreated system chart account as a template (698.2ms)
  * test create a chart account with value and is_depreciable (878.1ms)
  * test create a chart account (649.8ms)

ZB.Controllers.ChartAccount.ImportTest
  * test import chart accounts csv (1011.1ms)

ZB.Controllers.ChartAccount.DeleteTest
  * test locked chart accounts archive instead of delete (641.8ms)
  * test delete a chart account with no transactions (13203.9ms)
  * test deleting a chart account with transactions archives it (12832.3ms)

Finished in 40.6 seconds
18 tests, 0 failures

This was the Repo config it ran with 

[
  migration_timestamps: [inserted_at: :created_at],
  charset: "utf8mb4",
  collation: "utf8mb4_unicode_ci",
  pool_size: 10,
  log: false,
  username: "root",
  password: "root",
  database: "zipbooks_test",
  hostname: "127.0.0.1",
  pool: Ecto.Adapters.SQL.Sandbox,
  timeout: :infinity,
  ownership_timeout: :infinity,
  queue_target: 1000,
  queue_interval: 5000
]
wojtekmach commented 5 years ago

@michaelst thank you for following up!

the time is now cut in half (but still slower than the 5s it was before),

So it's still 40s on myxql vs 5s on mariaex? You mentioned you've merged a branch to your project's master branch, could you re-run the numbers on mariaex too?

Btw, out of curiosity, is switching adapters just a matter of changing adapter: between Ecto.Adapters.MyXQL and Ecto.Adapters.MySQL or do you need to perform any additional steps?

michaelst commented 5 years ago

Yes the 40s myxql (still 5s on local Mac), and still 5s mariaex.

I have a myxql branch, which I merged master into. One change that was merged that may explain the difference, not sure though, was we did a migration to our deleted tables. We have triggers on each of our tables for when a record is deleted we insert it into a deleted table first to keep the record. The migration removed the primary key from these tables leaving id only as an indexed column so that the unique constraint wouldn't be enforced.

Changing the adapter is the only change I make when testing myxql vs mariaex.

michaelst commented 5 years ago

I was getting similar results when running benchmark on an Ubuntu VM, so it seems an OS might be a factor here.

When you say similar results, do you mean you are also seeing the slowness on ubuntu?

wojtekmach commented 5 years ago

When you say similar results, do you mean you are also seeing the slowness on ubuntu?

Yes, exactly. I was able to observe slowness by starting a vagrant vm with ubuntu and running ecto_sql benchmarks, insert.exs in particular. On my VM myxql was about 10x slower than mariaex. Interestingly, when I switched pool_size from 10 to 1 on both repos, it was 100x slower. After the patch that introduced statement cache, the performance was comparable.

I just remembered, in order to use statement cache for inserts (for selects etc it works out of the box) in myxql there's a change required in ecto_sql so that's a very likely reason. I just pushed it to master (https://github.com/elixir-ecto/ecto_sql/commit/2b4ac614121a06aefeb8f55d94e8a2a697e8b0e4). Could you give ecto_sql master a try? Thank you again for helping us with debugging this.

michaelst commented 5 years ago

That has cut it down to 20s 

ZB.Controllers.ChartAccount.ImportTest
  * test import chart accounts csv (699.7ms)

ZB.Controllers.ChartAccount.UpdateTest
  * test cant archive a system chart account (258.3ms)
  * test update a chart account (527.1ms)
  * test change the parent and the change is propogated (331.5ms)
  * test cant update chart account with locked name (237.3ms)
  * test archive a chart account (280.7ms)

ZB.Controllers.ChartAccount.ShowTest
  * test get a chart account (538.3ms)

ZB.Controllers.ChartAccount.ReassignTest
  * test reassign a chart account (282.6ms)
  * test reassign a chart account to an uncreated default (276.9ms)

ZB.Controllers.ChartAccount.ReconciliationMetaTest
  * test get reconciliation meta (497.9ms)
  * test get reconciliation no permission (185.3ms)

ZB.Controllers.ChartAccount.IndexTest
  * test gets a list of chart accounts (508.6ms)

ZB.Controllers.ChartAccount.CreateTest
  * test create a chart account with value and is_depreciable (345.2ms)
  * test create using an uncreated system chart account as a template (275.5ms)
  * test create a chart account (279.1ms)

ZB.Controllers.ChartAccount.DeleteTest
  * test delete a chart account with no transactions (7473.3ms)
  * test locked chart accounts archive instead of delete (238.6ms)
  * test deleting a chart account with transactions archives it (7220.2ms)

Finished in 20.5 seconds
18 tests, 0 failures
wojtekmach commented 5 years ago

Thank you. One thing we've seen when running benchmarks on ubuntu is increased queue_time in sql logs, I wonder if that might be an issue still. I believe it's enough to set config :logger, level: :debug in tests to start seeing these logs. The total time will likely be slower due to writing to console but perhaps we'll see differences there?

edit: or, remove log: false from repo config.

michaelst commented 5 years ago

Ok another thing I just tested, is over the weekend I got docker setup for our dev environment. So I just ran in there using the elixir image, which appears to be using Debian Stetch, so maybe any linux environment as the results were the same.

michaelst commented 5 years ago

I removed log: false but didn't notice any logs, is there something else I am missing there?

wojtekmach commented 5 years ago

Have you set your logger level to :debug? You can check it with Application.get_env(:logger, :level).

On 6 Jun 2019, at 21:59, Michael St Clair notifications@github.com wrote:

I removed log: false but didn't notice any logs, is there something else I am missing there?

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/elixir-ecto/myxql/issues/60?email_source=notifications&email_token=AAASSJ6ATBVKIA54IFNFFRTPZFUDBA5CNFSM4HONG4ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXD7SXY#issuecomment-499644767, or mute the thread https://github.com/notifications/unsubscribe-auth/AAASSJ7NUUDVMPQLUWPAQQDPZFUDBANCNFSM4HONG4ZA.

michaelst commented 5 years ago

Ah yes, that was the problem, there was a warn override in test.exs, what am I looking for in the output?

wojtekmach commented 5 years ago

On 6 Jun 2019, at 22:08, Michael St Clair notifications@github.com wrote:

Ah yes, that was the problem, there was a warn override in test.exs, what am I looking for in the output?

Lines like the following:

20:15:12.989 [debug] QUERY OK db=0.1ms queue=0.1ms

queue value is important here. On my ubuntu benchmark I’ve seen differences between 0.1ms vs 50ms on myxql which explained parts the slowness. After the patch the difference is gone, but just want to cross this one off the list.

— You are receiving this because you commented. Reply to this email directly, view it on GitHub https://github.com/elixir-ecto/myxql/issues/60?email_source=notifications&email_token=AAASSJ44SLBXIE43F73D3VLPZFVD7A5CNFSM4HONG4ZKYY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGODXEAK5A#issuecomment-499647860, or mute the thread https://github.com/notifications/unsubscribe-auth/AAASSJ7IUTLQAK3JJSUIXFTPZFVD7ANCNFSM4HONG4ZA.

michaelst commented 5 years ago

I do see some queries with queueing but most under 0.5ms, some showing 1-3ms, and 8 queries around 45ms.

One thing I did notice watching the logs is that we have a long list of inserts that show up during the delete tests (we insert relational data to make sure the deletes working with foreign keys), however, none of these inserts had a queue, the 8 with the 45ms queues were almost all select queries.

wojtekmach commented 5 years ago

@michaelst unfortunately I'm not able to reproduce this, I'm getting similar numbers on my Mac and Ubuntu vm. Are you able to provide a minimal case that I could run locally that shows significant performance degradation? Otherwise, ping me at wojtek.mach [at] plataformatec.com.br if you'd be willing to share source code for your project.

michaelst commented 5 years ago

I'm not able to provide our source but I found a clue that may help

Mariaex

16:06:41.303 [debug] QUERY OK db=0.2ms
INSERT INTO `visibility__page_postal_code` (`page_id`,`postal_code`) VALUES (?,?) [187, "84006"]

16:06:41.303 [debug] QUERY OK db=0.1ms
INSERT INTO `visibility__page_postal_code` (`page_id`,`postal_code`) VALUES (?,?) [187, "84010"]

MyXQL

16:15:25.127 [debug] QUERY OK db=0.3ms
INSERT INTO `visibility__page_postal_code` (`page_id`,`postal_code`) VALUES (?,?) [198, "84006"]

16:15:25.170 [debug] QUERY OK db=0.4ms
INSERT INTO `visibility__page_postal_code` (`page_id`,`postal_code`) VALUES (?,?) [198, "84010"]

notice the timestamps, MyXQL does take twice as long, but the second query isn't kicked off until 43ms later where Mariaex is the same ms.

Here is the code where that is happening, the put_assoc adds a few hundred of those in the test, I only included two from the logs, but that time gap between the queries still happens between each query throughout the logs even though the query doesn't say it is taking that long and there is no queueing reported.

  def put_service_area_postal_codes(changeset) do
    postal_code = get_field(changeset, :service_area_postal_code)
    miles = get_field(changeset, :service_area_miles)

    postal_codes =
      from(pc in ZB.PostalCode, where: pc.postal_code in ^ZB.Page.Utils.postal_codes_close_to(postal_code, miles))
      |> ZB.Repo.all()

    put_assoc(
      changeset,
      :postal_codes,
      postal_codes
    )
  end
wojtekmach commented 5 years ago

@michaelst I think I've found the culprit, I believe we are not using cache properly on ecto_sql side when inserting associations. I thought I ruled this out but I must have made a mistake when trying that out.

Could you try running tests on this ecto_sql version?

{:ecto_sql, github: "elixir-ecto/ecto_sql", branch: "wm-insert-all-cache-statement"}
michaelst commented 5 years ago

Ok that brought our test suite to 400s compared to running about 250s with mariaex right now, I will spend some more time seeing if I can track down where else there might be some caching needed and update the example project I sent you.

wojtekmach commented 5 years ago

@michaelst Great! This branch might be helpful: https://github.com/elixir-ecto/myxql/tree/wm-debug, it'll print the given statement if ecto didn't set the cache key (and so the statement will have to be re-prepared which slows it down)

michaelst commented 5 years ago

So there are a ton of select queries that show up. Are those supposed to be cached?

Here are some of the queries I saw show up if they help at all.

      from(dc in "zb__dirty_caches", where: dc.id in ^ids)
      |> ZB.Repo.update_all(
        set: [locked_until: Timex.now() |> Timex.shift(minutes: 5) |> Utils.DateTime.to_string()],
        inc: [failure_count: 1]
      )
ZB.Repo.delete_all(from(t in "zb__dirty_caches", where: t.id in ^&1))

Also, if I call query directly, is there something I need to do for it to cache because I noticed a few of these? 

Ecto.Adapters.SQL.query!(Repo, query, [] )
josevalim commented 5 years ago

@michaelst you can explicitly provide a cache_statement: option for raw queries. Queries using IN cannot be efficiently cached in MySQL, so we don't.

michaelst commented 5 years ago

@josevalim I'm not seeing any documentation for that, is there anything specific that needs to be set for the value?

josevalim commented 5 years ago

@michaelst it should be documented on MyXQL but it is basically a unique name for that query to be used as a cached. For example:

cache_statement: "my_complex_query_foo_bar"
josevalim commented 5 years ago

Also, it is worth saying that we won't be equal to Mariaex because they have a LRU cache and we have basically a cache per source. It is extremely helpful in loops but not beyond that. But if you set cache_statement, then you can cache those extra queries too.

michaelst commented 5 years ago

ok that is helpful to know.

On the select queries that are not being cached, would that be expected?

josevalim commented 5 years ago

On the select queries that are not being cached, would that be expected?

Yes, because IN in MySQL has a variable number of parameters. This means that, for every possible number of parameters, we have to cache a different query, which would explode in memory. They are just hard to cache in general. If the number of parameters are fixed and known query building time, then you can rewrite the query without IN. In Postgrex we can bypass this by using arrays.

michaelst commented 5 years ago

Ok that makes sense. The one thing that still doesn't make sense to me is why we are only seeing this performance difference on linux and not mac.

michaelst commented 5 years ago

Using the cache statements on the raw queries helped a little.

Another thing I am noticing is that in the select queries there are a ton that don't have in but have WHERE (false)

josevalim commented 5 years ago

I believe that doing an IN on an empty list becomes a WHERE FALSE.

wojtekmach commented 5 years ago

We don't have plans to ship with an LRU cache so some performance hit might be expected, but in those cases as mentioned in this issue the solution would be to explicitly set :cache_statement option to avoid re-preparing the query.

I'm gonna close this issue due to inactivity, but if folks could give us benchmarks we can run ourselves I'd be happy to optimise further.

Thanks a lot @michaelst!

1player commented 5 years ago

Sorry for resurrecting this thread, I've noticed the same behaviour and query caching doesn't seem to explain it. Most of the slowdown is in commit phase.

mariaex:

14:24:39.660 [debug] QUERY OK db=0.1ms
begin []
14:24:39.660 [debug] QUERY OK db=0.2ms
INSERT INTO `racing_selections` (`cluster`,`encoded_key`,`event_id`,`market`,`withdrawn`,`inserted_at`,`updated_at`) VALUES (?,?,?,?,?,?,?) ["b5073fca-80c5-4d16-98ba-38e0e1f6c387", "Horse", 722, "win_each_way", false, ~N[2019-09-13 13:24:39], ~N[2019-09-13 13:24:39]]
14:24:39.661 [debug] QUERY OK source="racing_selections" db=0.3ms
UPDATE `racing_selections` AS r0 SET r0.`cluster` = ? WHERE (r0.`id` IN (?,?,?)) ["b5073fca-80c5-4d16-98ba-38e0e1f6c387", 594, 593, 592]
14:24:39.661 [debug] QUERY OK source="racing_selections" db=0.2ms
UPDATE `racing_selections` AS r0 SET r0.`cluster` = ? WHERE (false) [nil]
14:24:39.661 [debug] QUERY OK db=0.1ms
commit []

myxql:

14:25:33.111 [debug] QUERY OK db=0.3ms
begin []
14:25:33.112 [debug] QUERY OK db=0.2ms
INSERT INTO `racing_selections` (`cluster`,`encoded_key`,`event_id`,`market`,`withdrawn`,`inserted_at`,`updated_at`) VALUES (?,?,?,?,?,?,?) ["e11aab1f-e48b-4b59-9c27-649ea8292e1f", "Horse", 1030, "win_each_way", false, ~N[2019-09-13 13:25:33], ~N[2019-09-13 13:25:33]]
14:25:33.113 [debug] QUERY OK source="racing_selections" db=0.4ms
UPDATE `racing_selections` AS r0 SET r0.`cluster` = ? WHERE (r0.`id` IN (?,?,?)) ["e11aab1f-e48b-4b59-9c27-649ea8292e1f", 859, 858, 857]
14:25:33.155 [debug] QUERY OK source="racing_selections" db=0.4ms
UPDATE `racing_selections` AS r0 SET r0.`cluster` = ? WHERE (false) [nil]
14:25:33.201 [debug] QUERY OK db=46.1ms
commit []

The last commit on MyXQL takes 46ms while it's 0.1ms on mariaex. I don't think it's a query caching problem.

EDIT: mind you, it doesn't happen on all commits, and yes, these are "raw" queries.