elixir-sqlite / ecto_sqlite3

An Ecto SQLite3 adapter.
https://hexdocs.pm/ecto_sqlite3
MIT License
300 stars 45 forks source link

Update insertion benchmark #144

Closed gmile closed 5 months ago

gmile commented 6 months ago

This refreshes insertion benchmark stats + does a couple of minor fixes to the bench README.md.

I couldn't update all benchmarks due to a failure that I haven't had time to look into. Here's a full failure output:

``` Operating System: macOS CPU Information: Apple M3 Max Number of Available Cores: 16 Available memory: 128 GB Elixir 1.16.2 Erlang 26.2.4 JIT enabled: true Benchmark suite executing with the following configuration: warmup: 2 s time: 5 s memory time: 0 ns reduction time: 0 ns parallel: 1 inputs: Big 1 Million, Date attr, Medium 100 Thousand, Small 1 Thousand, Time attr, UUID attr Estimated total run time: 2 min 6 s Benchmarking MyXQL Loader with input Big 1 Million ... Benchmarking MyXQL Loader with input Date attr ... Benchmarking MyXQL Loader with input Medium 100 Thousand ... Benchmarking MyXQL Loader with input Small 1 Thousand ... Benchmarking MyXQL Loader with input Time attr ... Benchmarking MyXQL Loader with input UUID attr ... Benchmarking Pg Loader with input Big 1 Million ... Benchmarking Pg Loader with input Date attr ... Benchmarking Pg Loader with input Medium 100 Thousand ... Benchmarking Pg Loader with input Small 1 Thousand ... Benchmarking Pg Loader with input Time attr ... Benchmarking Pg Loader with input UUID attr ... Benchmarking SQLite3 Loader with input Big 1 Million ... Benchmarking SQLite3 Loader with input Date attr ... Benchmarking SQLite3 Loader with input Medium 100 Thousand ... Benchmarking SQLite3 Loader with input Small 1 Thousand ... Benchmarking SQLite3 Loader with input Time attr ... 09:27:11.506 [error] Task #PID<0.352.0> started from #PID<0.98.0> terminating ** (FunctionClauseError) no function clause matching in Calendar.ISO.parse_time/1 (elixir 1.16.2) lib/calendar/iso.ex:297: Calendar.ISO.parse_time(~T[21:25:04.361140]) (elixir 1.16.2) lib/calendar/time.ex:255: Time.from_iso8601/2 (ecto_sqlite3 0.15.1) lib/ecto/adapters/sqlite3/codec.ex:83: Ecto.Adapters.SQLite3.Codec.time_decode/1 (ecto 3.11.1) lib/ecto/type.ex:932: Ecto.Type.process_loaders/3 (ecto 3.11.1) lib/ecto/schema/loader.ex:71: anonymous fn/5 in Ecto.Schema.Loader.unsafe_load/4 (elixir 1.16.2) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3 (elixir 1.16.2) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2 (benchee 1.3.0) lib/benchee/benchmark/collect/time.ex:17: Benchee.Benchmark.Collect.Time.collect/1 Function: #Function<2.121299024/0 in Benchee.Utility.Parallel.map/2> Args: [] ** (EXIT from #PID<0.98.0>) an exception was raised: ** (FunctionClauseError) no function clause matching in Calendar.ISO.parse_time/1 (elixir 1.16.2) lib/calendar/iso.ex:297: Calendar.ISO.parse_time(~T[21:25:04.361140]) (elixir 1.16.2) lib/calendar/time.ex:255: Time.from_iso8601/2 (ecto_sqlite3 0.15.1) lib/ecto/adapters/sqlite3/codec.ex:83: Ecto.Adapters.SQLite3.Codec.time_decode/1 (ecto 3.11.1) lib/ecto/type.ex:932: Ecto.Type.process_loaders/3 (ecto 3.11.1) lib/ecto/schema/loader.ex:71: anonymous fn/5 in Ecto.Schema.Loader.unsafe_load/4 (elixir 1.16.2) lib/enum.ex:2528: Enum."-reduce/3-lists^foldl/2-0-"/3 (elixir 1.16.2) lib/enum.ex:1700: Enum."-map/2-lists^map/1-1-"/2 (benchee 1.3.0) lib/benchee/benchmark/collect/time.ex:17: Benchee.Benchmark.Collect.Time.collect/1 ```
warmwaffles commented 6 months ago

@gmile you've stumbled on a bug.

How did you run the benchmarks. Were you running MySQL and Postgres locally vs running them in docker?

Overall thanks for bumping the benchmarks!

gmile commented 6 months ago

@warmwaffles to your question:

How did you run the benchmarks. Were you running MySQL and Postgres locally vs running them in docker?

I ran MySQL in a container, but PostgreSQL locally (natively, running the the host OS).

Your question made me realise its not fair to compare results of SQLite3 and PostgreSQL running on my computer natively with results for MySQL running inside a container, e.g. via virtualization which has a performance penalty. I'm using orbstack which claims to be faster the Docker, but still. I'll install MySQL on my mac natively and will regenerate the benchmark once again.

On a separately note, to clarify existence of commit https://github.com/elixir-sqlite/ecto_sqlite3/pull/144/commits/a9902f38ba8699a989eacd4b96b111aa49bbe361. Using docker image for MySQL 5.7 didn't work for me, because there's no arm64 OCI manifest generated for 5.7 version, but there's for 8.

gmile commented 6 months ago

I was able to reproduce the exception mentioned in my message with this self-contained Elixir script:

```elixir Mix.install([:ecto_sqlite3]) Application.put_env(:foo, Repo, pool_size: 1, database: ":memory:") defmodule Repo do use Ecto.Repo, adapter: Ecto.Adapters.SQLite3, otp_app: :foo end defmodule Migration0 do use Ecto.Migration def change do create table("posts") do add(:title, :string) add(:published_at, :time) end end end defmodule Post do use Ecto.Schema schema "posts" do field(:title, :string) field(:published_at, :time) end end defmodule Main do import Ecto.Query, warn: false def reproduce do :ok = Repo.__adapter__().storage_up(Repo.config()) {:ok, _pid} = Supervisor.start_link([Repo], strategy: :one_for_one) Ecto.Migrator.run(Repo, [{0, Migration0}], :up, all: true, log_migrations_sql: :debug) Repo.load(Post, %{title: "New blog post", published_at: ~T[21:25:04.361140]}) end end Main.reproduce() ```

If I make a change to a script - it will no longer raise:

diff --git a/time-issue-repro.exs b/time-issue-repro.exs
index 8643d65..52e173e 100644
--- a/time-issue-repro.exs
+++ b/time-issue-repro.exs
@@ -38,7 +38,7 @@ defmodule Main do

     Ecto.Migrator.run(Repo, [{0, Migration0}], :up, all: true, log_migrations_sql: :debug)

-    Repo.load(Post, %{title: "New blog post", published_at: ~T[21:25:04.361140]})
+    Repo.load(Post, %{title: "New blog post", published_at: "21:25:04.361140")
   end
 end
gmile commented 6 months ago

@warmwaffles I added this commit https://github.com/elixir-sqlite/ecto_sqlite3/pull/144/commits/bceff9a7bf38518b875e41bc6c13f631e76598f2 just to see if I am able to get pass the issue when running the benchmark

It work, so not only "insert" part, but all benchmarks were regenerated successfully. However, I don't believe my fix is correct. For example, running benchmark having %Date{} struct as a value appears to run just fine without any changes:

https://github.com/elixir-sqlite/ecto_sqlite3/blob/86b76bb3996221d1ef2d9dd021b9b261c48bdf45/bench/scripts/micro/load_bench.exs#L33-L34

I am not sure why explicit handling of %Time{} is necessary in principle 🤔

warmwaffles commented 5 months ago

Sorry this took so long to merge. It completely slipped through on me. Feel free to ping mention me if you see another PR for these libraries fall through.

gmile commented 5 months ago

@warmwaffles noted, and no worries, I kind of forgot about this one too 🙈 Thanks for merging! I just hope this change is not going to break anything https://github.com/elixir-sqlite/ecto_sqlite3/pull/144/commits/bceff9a7bf38518b875e41bc6c13f631e76598f2 🙏

warmwaffles commented 5 months ago

It shouldn't. You added a test for it and I think it was just an oversight by me initially.