search

vitessio / vitess

Vitess is a database clustering system for horizontal scaling of MySQL.
http://vitess.io
Apache License 2.0
18.18k stars 2.06k forks source link

unifying OLAP and OLTP modes #12357

Open frouioui opened 1 year ago

frouioui commented 1 year ago

Over time we have enhanced OLAP support to be at parity with OLTP support. Related PRs: #6077 #6205 #6655 #6691 #6692 #6949 #7752 #8018 #8139 #8722 #9115 #9253 #9484 #10946 #11090 #11517

However, we still have duplication at the code level. At some point we want to refactor the code to eliminate as much of the duplication as possible.

techbelle commented 1 year ago

I have a quick question for the group; I've been reading the Vitess documentation and even though it says that the default workload type is OLTP, and that there is a general execution timeout set as a result of having the workload type = OLTP, I haven't found anything that says what the execution timeout settings are for OLAP, OLTP, or DBA. Can someone please guide me to wherever this is documented OR the actual place in the code? I found an old PDF of the documentation (https://vitess.io/files/version-pdfs/Vitess-Docs-8.0-10-27-2020.pdf) that says

Anyway based on the above, I can't tell if this issue of unifying the modes would mean that the execution timeout value becomes configurable or if it means that all modes become n seconds? also can't tell what impact would be in terms of rows scanned?

deepthi commented 1 year ago

Hi @techbelle, thanks for the feedback. This issue was probably confusing because it had only a title and no description. I've added a description that documents the scope of work.

Re documentation, here's what I see in the MySQL Compatibility page on the website

Killing running queries

Vitess does not yet support killing running queries via the KILL command through VTGate. Vitess does have strict query timeouts for OLTP workloads (see below). If you need to kill a query, you can connect to the underlying MySQL shard instance and run KILL from there.

Workload

By default, Vitess sets some intentional restrictions on the execution time and number of rows that a query can return. This default workload mode is called OLTP. This can be disabled by setting the workload to OLAP:

So when it says "see below", it is just referring to the next section, which pretty much just says "intentional restrictions blah blah".

What you have discovered is accurate, the queryserver-config-query-timeout flag which defaults to 30 seconds controls how long OLTP queries can run. There is no timeout on OLAP read queries. In addition to this, there are flags that control how long transactions are allowed to stay open in OLTP and OLAP modes. Both of those default to 30 seconds as well.

With DBA mode, no timeout is enforced on transactions and they are allowed to stay open forever (subject to MySQL level idle connection timeouts / vttablet shutdown). There is some debate about whether this is actually a good idea and whether we should stop allowing these in future.

Hope this helps. I started off trying to update the website docs to answer your questions, but couldn't find a good place for this. Maybe we need to write a new page!

Default flag values in code: https://github.com/vitessio/vitess/blob/467972a523a85f2aa8f9a532eda441379e51471b/go/vt/vttablet/tabletserver/tabletenv/config.go#L500

Transaction timeouts (0 means no timeout): https://github.com/vitessio/vitess/blob/800c6b089f0606cdff06d54aa7874c7f9b66d748/go/vt/vttablet/tabletserver/tabletenv/config.go#L437

techbelle commented 1 year ago

wow, thank you for this incredibly thorough reply ! 💯 makes total sense. FWIW, i only even heard of this feature because another engineer was telling me how great it is. That's an anecdote, so I can't speculate on the usage or the lovability of the feature overall (nor do I have any opinion) but it does seem very useful and plan to tinker with it.

thanks again!