Add support for MySQL 8.0

jeremycole commented 3 years ago

Add support for MySQL 8.0. There are a number of changes needed to support MySQL 8.0, and some new data structures to implement:

[x] Support new headers in FSP pages
[x] Support for reconstructing full BLOB content
[ ] Support new page types:
- [ ] UNKNOWN (13)
- [ ] COMPRESSED (14)
- [ ] ENCRYPTED (15)
- [ ] COMPRESSED_AND_ENCRYPTED (16)
- [ ] ENCRYPTED_RTREE (17)
- [x] SDI_BLOB (18) (aka BLOB)
- [ ] SDI_ZBLOB (19) (aka ZBLOB)
- [ ] LEGACY_DBLWR (20)
- [ ] RSEG_ARRAY (21)
- [ ] LOB_INDEX (22)
- [ ] LOB_DATA (23)
- [ ] LOB_FIRST (24)
- [ ] ZLOB_FIRST (25)
- [ ] ZLOB_DATA (26)
- [ ] ZLOB_INDEX (27)
- [ ] ZLOB_FRAG (28)
- [ ] ZLOB_FRAG_ENTRY (29)
- [x] SDI (17853) (aka INDEX)
- [ ] RTREE (17854)
[x] Support for new data dictionary structures:
- [x] Read SDI information from InnoDB tablespace files
- [x] Using SDI to make record describers
- [x] Refactor current data dictionary to support old and new
[ ] Support new mysql data dictionary tables
- [x] https://github.com/jeremycole/innodb_ruby/issues/205
- [x] https://github.com/jeremycole/innodb_ruby/issues/207
[ ] Support for LOB
[ ] Support for Compressed LOB (ZLOB)
[ ] Support for online-added columns?
[ ] Support for undo tablespaces
[ ] ???

Sub-issue list:

PR list:

utdrmac commented 2 years ago

Related?

# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
    16: from /usr/local/bin/innodb_space:23:in `<main>'
    15: from /usr/local/bin/innodb_space:23:in `load'
    14: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
    13: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
    12: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
    11: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
    10: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
     9: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
     8: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
     7: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
     6: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
     5: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
     4: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
     3: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
     2: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
     1: from /usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/usr/local/share/gems/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055a140fed580> (NoMethodError)

8.0.28

lhsaq2009 commented 1 year ago

[root@MySQL80 mysql]# innodb_space -s ibdata1 system-spaces
name                            pages       indexes
Traceback (most recent call last):
    16: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `<main>'
    15: from /root/.gem/ruby/2.6.0/bin/innodb_space:23:in `load'
    14: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:1714:in `<top (required)>'
    13: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:192:in `system_spaces'
    12: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `block in system_spaces'
    11: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `to_a'
    10: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/bin/innodb_space:188:in `each'
     9: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:341:in `each_index'
     8: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:321:in `each_index_root_page_number'
     7: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:425:in `each_index_by_space_id'
     6: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:355:in `each_index'
     5: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:324:in `each_record_from_data_dictionary_index'
     4: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/data_dictionary.rb:287:in `data_dictionary_index'
     3: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `index'
     2: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/space.rb:312:in `new'
     1: from /root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:19:in `initialize'
/root/.gem/ruby/2.6.0/gems/innodb_ruby-0.12.0/lib/innodb/index.rb:34:in `page': undefined method `record_describer=' for #<Innodb::Page::FspHdrXdes:0x000055b0ae47b968> (NoMethodError)
[root@MySQL80 mysql]#

8.0.18

jeremycole commented 1 year ago

I filed MySQL Bug 111079: InnoDB SDI index description has fields out-of-order as implementing SDI caused me to find a subtle issue with the fields being out of order internally.

jeremycole commented 1 year ago

I filed MySQL Bug 111112: JSON SDI default_value in columns object includes trailing zero bytes as implementing SDI caused me to notice that the included default_value field includes trailing zero bytes (seems to be encoding a raw buffer from C++).

jeremycole commented 1 week ago

Good news! The main branch now has support for the new SDI data dictionary, e.g.:

$ bundle exec bin/innodb_space -s spec/data/sakila/8.0 -T sakila/actor -I PRIMARY index-recurse
ROOT NODE #4: 200 records, 7507 bytes
  RECORD: (actor_id=1) → (first_name="PENELOPE", last_name="GUINESS", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=2) → (first_name="NICK", last_name="WAHLBERG", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=3) → (first_name="ED", last_name="CHASE", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=4) → (first_name="JENNIFER", last_name="DAVIS", last_update="2006-02-15 11:34:33")
  RECORD: (actor_id=5) → (first_name="JOHNNY", last_name="LOLLOBRIGIDA", last_update="2006-02-15 11:34:33")

You may notice that it can't read all columns from all tables, in particular there is some weirdness with DECIMAL type at the moment, and only partial support for ENUM and SET. While innodb_ruby has never really supported those types in the past, from the new data dictionary it now knows the proper types of columns, so it's trying, and kinda failing. I'll keep working on that.

In the mean time, feel free to play around and file bug reports, send pull requests, etc. The basics should all be there.

jeremycole / innodb_ruby

Add support for MySQL 8.0 #97