search

dolthub / dolt

Dolt – Git for Data
Apache License 2.0
17.97k stars 514 forks source link

Panic when running `dolt sql-server` in a directory with old and new format dolt databases #3880

Closed timsehn closed 2 years ago

timsehn commented 2 years ago

Tip of main.

Before running dolt sql-server

dolt $ ls -al
total 48
drwxr-xr-x  70 timsehn  staff  2240 Jul 19 09:34 .
drwxr-xr-x   6 timsehn  staff   192 Oct 18  2021 ..
-rw-r--r--@  1 timsehn  staff  6148 Jul  5 13:54 .DS_Store
-rw-r--r--   1 timsehn  staff  5514 Jul 11 12:47 .sqlhistory
drwxr-xr-x   5 timsehn  staff   160 Mar  9 11:20 Org-record-backup-3
drwxr-xr-x   7 timsehn  staff   224 Apr 21 12:49 Source3
drwxr-xr-x   4 timsehn  staff   128 Jul 11 14:06 backup-example
drwxr-xr-x   3 timsehn  staff    96 Jul 11 11:53 backup-restore
drwxr-xr-x   3 timsehn  staff    96 Jul 11 11:41 backups
drwxr-xr-x   5 timsehn  staff   160 Oct 22  2021 basketball
drwxr-xr-x   4 timsehn  staff   128 Aug 19  2021 bats-working
drwxr-xr-x   3 timsehn  staff    96 Apr  7 14:42 ckeran
drwxr-xr-x   4 timsehn  staff   128 Jun 17  2021 datasets
drwxr-xr-x   3 timsehn  staff    96 Aug  4  2021 demo
drwxr-xr-x   3 timsehn  staff    96 Jul 12 15:26 dirty-branch
drwxr-xr-x   3 timsehn  staff    96 Aug 30  2021 disqo-demo
drwxr-xr-x   5 timsehn  staff   160 Jun 29 13:05 docs
drwxr-xr-x   3 timsehn  staff    96 Sep 30  2021 dolt-test
drwxr-xr-x   3 timsehn  staff    96 Jun 21  2021 doltdb
drwxr-xr-x   5 timsehn  staff   160 Feb  7 12:02 drop-test
drwxr-xr-x   3 timsehn  staff    96 Nov 16  2021 earnings
drwxr-xr-x   3 timsehn  staff    96 Jun  7 16:32 easy_panic
drwxr-xr-x   3 timsehn  staff    96 Mar 30 14:33 flashbots
drwxr-xr-x   4 timsehn  staff   128 Mar  8 12:53 flvoters
drwxr-xr-x   3 timsehn  staff    96 Jun  7 16:28 getting_started
drwxr-xr-x   4 timsehn  staff   128 Jun  2 17:44 hospital-price-transparency-v3
drwxr-xr-x   4 timsehn  staff   128 Jul  6 15:49 json-example
drwxr-xr-x   4 timsehn  staff   128 Aug 12  2021 lesson-1
drwxr-xr-x   4 timsehn  staff   128 Aug 20  2021 menus
drwxr-xr-x   3 timsehn  staff    96 Dec  9  2021 military-demo
drwxr-xr-x   5 timsehn  staff   160 Jul 11 11:26 options
drwxr-xr-x   4 timsehn  staff   128 Jul 15 09:35 read_replica
drwxr-xr-x   5 timsehn  staff   160 Jul 15 09:35 replication_example
drwxr-xr-x   4 timsehn  staff   128 Jan  7  2022 simping101
drwxr-xr-x   3 timsehn  staff    96 Dec 15  2021 stocks
drwxr-xr-x   4 timsehn  staff   128 Mar  9 11:24 tablename-dash-bug
drwxr-xr-x   6 timsehn  staff   192 Jun 10 15:08 test-as-of
drwxr-xr-x   6 timsehn  staff   192 Mar 21 13:26 test-busteroni
drwxr-xr-x   3 timsehn  staff    96 Aug 26  2021 test-clobber
drwxr-xr-x   7 timsehn  staff   224 Mar 16 10:29 test-comments
drwxr-xr-x   4 timsehn  staff   128 May 26 14:16 test-concurrency
drwxr-xr-x   4 timsehn  staff   128 Jun 15 16:14 test-conflict-resolve
drwxr-xr-x   5 timsehn  staff   160 Jun 13 15:19 test-conflicts
drwxr-xr-x   5 timsehn  staff   160 Jan 28 13:49 test-constraints
drwxr-xr-x   4 timsehn  staff   128 Jan 27 08:49 test-create-db
drwxr-xr-x   9 timsehn  staff   288 Jun 14 15:22 test-csv-json-issue
drwxr-xr-x   4 timsehn  staff   128 May 10 16:39 test-default
drwxr-xr-x   3 timsehn  staff    96 Nov 17  2021 test-dolt-branch
drwxr-xr-x   4 timsehn  staff   128 May 17 10:20 test-dolt-stored-w-prepareds
drwxr-xr-x   2 timsehn  staff    64 May 11 14:41 test-drop
drwxr-xr-x   4 timsehn  staff   128 Mar 23 10:26 test-dumps
drwxr-xr-x   4 timsehn  staff   128 Jun  7 16:26 test-information_schema.statitics
drwxr-xr-x   3 timsehn  staff    96 Dec  9  2021 test-lower
-rw-r--r--   1 timsehn  staff   570 Jun  9 18:17 test-maria-prepareds.py
drwxr-xr-x   6 timsehn  staff   192 May 16 12:58 test-mv-table
-rw-r--r--   1 timsehn  staff   578 Jun  9 18:44 test-mysql-prepareds.py
drwxr-xr-x   9 timsehn  staff   288 Jan 28 14:19 test-nickname
drwxr-xr-x   3 timsehn  staff    96 Nov  2  2021 test-null-pk-add
drwxr-xr-x   2 timsehn  staff    64 Mar  1 08:54 test-perms
drwxr-xr-x   4 timsehn  staff   128 Mar 29 09:07 test-pull
drwxr-xr-x   5 timsehn  staff   160 Jan 25 15:51 test-python-bats
drwxr-xr-x   4 timsehn  staff   128 May  5 14:07 test-shrink-col
drwxr-xr-x   7 timsehn  staff   224 Nov 16  2021 test-status-issue
drwxr-xr-x   3 timsehn  staff    96 Feb 14 15:23 test-triggers
drwxr-xr-x   5 timsehn  staff   160 Jun  6 15:42 test-tutorial
drwxr-xr-x   3 timsehn  staff    96 Jun 16 10:20 test-unscoped-diff
drwxr-xr-x   4 timsehn  staff   128 Jun 16 13:04 test-working-set-cli-server-interaction
drwxr-xr-x   5 timsehn  staff   160 May 27 13:35 us-housing-prices
drwxr-xr-x   5 timsehn  staff   160 Jul 20 10:51 us-jails
drwxr-xr-x  10 timsehn  staff   320 Nov  2  2021 us-schools

Run dolt sql-server:

dolt $ dolt sql-server
Starting server with Config HP="localhost:3306"|T="28800000"|R="false"|L="info"
panic: unknown message id 

goroutine 1 [running]:
github.com/dolthub/dolt/go/store/prolly/message.GetKeysAndValues({0xc02c1ac02b?, 0x190?, 0x10154e5?})
    /Users/timsehn/liquidata/git/dolt/go/store/prolly/message/message.go:56 +0x54d
github.com/dolthub/dolt/go/store/prolly/tree.NodeFromBytes(...)
    /Users/timsehn/liquidata/git/dolt/go/store/prolly/tree/node.go:101
github.com/dolthub/dolt/go/store/datas.parse_storeroot({0xc02c1ac003, 0x1a8, 0x1a8}, {0x2d7c468?, 0xc02c1a8000})
    /Users/timsehn/liquidata/git/dolt/go/store/datas/refmap.go:44 +0x1dd
github.com/dolthub/dolt/go/store/datas.(*database).loadDatasetsRefmap(0xc02c1a6030, {0x2d7af30, 0xc000c7c0c0}, {0x60, 0x7c, 0x37, 0xdd, 0xb4, 0x4a, 0x5, ...})
    /Users/timsehn/liquidata/git/dolt/go/store/datas/database_common.go:116 +0x14c
github.com/dolthub/dolt/go/store/datas.(*database).Datasets(0xc02c1a6030, {0x2d7af30, 0xc000c7c0c0})
    /Users/timsehn/liquidata/git/dolt/go/store/datas/database_common.go:156 +0x165
github.com/dolthub/dolt/go/store/datas.(*database).GetDataset(0xc000f86c90?, {0x2d7af30, 0xc000c7c0c0}, {0xc000610ae0, 0x1b})
    /Users/timsehn/liquidata/git/dolt/go/store/datas/database_common.go:179 +0x192
github.com/dolthub/dolt/go/libraries/doltcore/doltdb.(*DoltDB).ResolveWorkingSet(0xc000bbc050, {0x2d7af30, 0xc000c7c0c0}, {{0xc000d2a630?, 0x29c8280?}})
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/doltdb/doltdb.go:361 +0xd4
github.com/dolthub/dolt/go/libraries/doltcore/env.WorkingSet({0x2d7af30, 0xc000c7c0c0}, 0xc02c194020?, {0x2d7d400?, 0xc000b96130?})
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/env/environment.go:561 +0x5f
github.com/dolthub/dolt/go/libraries/doltcore/env.(*DoltEnv).WorkingSet(0xc000b92f30, {0x2d7af30, 0xc000c7c0c0})
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/env/environment.go:552 +0x6c
github.com/dolthub/dolt/go/libraries/doltcore/env.Load({0x2d7af30, 0xc000c7c0c0}, 0x2a96bb8, {0x2d93078?, 0xc02c01a0a0?}, {0xc000052f00, 0x13}, {0x29dd828, 0x7})
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/env/environment.go:164 +0x51e
github.com/dolthub/dolt/go/libraries/doltcore/env.MultiEnvForDirectory.func1({0xc000620240?, 0x29d7bac?}, 0x40?, 0x0?)
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/env/multi_repo_env.go:288 +0xe9
github.com/dolthub/dolt/go/libraries/utils/filesys.(*localFS).Iter(0xc000050359?, {0x29d7bac?, 0x2d5bf68?}, 0x0, 0xc000c7c440)
    /Users/timsehn/liquidata/git/dolt/go/libraries/utils/filesys/localfs.go:110 +0x1ba
github.com/dolthub/dolt/go/libraries/doltcore/env.MultiEnvForDirectory({0x2d7af30?, 0xc000c7c0c0}, {0x2d83138?, 0xc00011bc50}, {0x2d93078?, 0x4115d30}, {0x52005b8?, 0x20?}, 0x0, 0xc00011c5a0)
    /Users/timsehn/liquidata/git/dolt/go/libraries/doltcore/env/multi_repo_env.go:276 +0x305
github.com/dolthub/dolt/go/cmd/dolt/commands/sqlserver.Serve({0x2d7af30, 0xc000c7c0c0}, {0x29dd828, 0x7}, {0x2d96da8, 0xc00091e2d0}, 0x1?, 0xc00011c5a0)
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/commands/sqlserver/server.go:100 +0x58e
github.com/dolthub/dolt/go/cmd/dolt/commands/sqlserver.startServer({0x2d7af30, 0xc000c7c0c0}, {0x29dd828, 0x7}, {0xc0006d9e10, 0xf}, {0xc00004e050, 0x0, 0x0}, 0xc00011c5a0, ...)
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/commands/sqlserver/sqlserver.go:219 +0x5aa
github.com/dolthub/dolt/go/cmd/dolt/commands/sqlserver.SqlServerCmd.Exec({{0x29dd828?, 0xc0000bbaa0?}}, {0x2d7c040?, 0xc0005d34c0}, {0xc0006d9e10, 0xf}, {0xc00004e050, 0x0, 0x0}, 0xc00011c5a0)
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/commands/sqlserver/sqlserver.go:173 +0x285
github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.handleCommand({{0x29d9529, 0x4}, {0x29f4131, 0x11}, {0x0, 0x0}, {0xc0001c7200, 0x2e, 0x2e}, 0x0}, ...)
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/cli/command.go:237 +0x502
github.com/dolthub/dolt/go/cmd/dolt/cli.SubCommandHandler.Exec({{0x29d9529, 0x4}, {0x29f4131, 0x11}, {0x0, 0x0}, {0xc0001c7200, 0x2e, 0x2e}, 0x0}, ...)
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/cli/command.go:186 +0x47f
main.runMain()
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/dolt.go:366 +0x6c5
main.main()
    /Users/timsehn/liquidata/git/dolt/go/cmd/dolt/dolt.go:138 +0x19

After, shit out a .doltcfg directory:

dolt $ ls -al .doltcfg/
total 0
drwxr-xr-x   2 timsehn  staff    64 Jul 21 08:48 .
drwxr-xr-x  70 timsehn  staff  2240 Jul 21 08:49 ..
timsehn commented 2 years ago

Andy claims I have a repo that's new format somehow.

timsehn commented 2 years ago

Indeed:

dolt $ find . | grep manifest | xargs grep __DOLT_1__  
./test-concurrency/.dolt/noms/manifest:5:__DOLT_1__:17h1e5bfuollhti6t4ro2r0g5310p30c:c1u3fndk982oqcndrfc7o118pe59dlbu:00000000000000000000000000000000:6893nt1teg7mf3qstjefu9vblo37fuff:2:fq5a1sbquehigit0qsecmqn584r171tq:2:m5mo6qk2fjtfvk601d452884u1u6va1b:1:ls0mre9s6hnfn1nqbac5scmglnioa6p1:5:97e85vc26fb7demsm6dk5n3shol86d7e:7:mh0tumb76lfqjngu2rrgmn82emubhine:7:q6dc6luekf22at1umnlqph5fa7ceesg8:7:nj9djqc736i130b0ivi9uagjtk9c3qel:5:u9psr92v13l6ofgvnloiqr7r94v40nld:5:181t2jpjokuo7ikph3q1eikljjf1jve0:6:96cq0ra8fa2oi9s1lvpeltsngjjrtl2g:2:0f8qldn7834fpuai3p72b81s2iv6r867:1:to6t90qicthka6a1p6p2ctmf2b7qnsaf:2:gm4j144kkh4qugmtmh2n64rmm7d0djuk:1:k3d5lffc7daqtk71ovn3hohmkod3v7dh:3

This should not panic. It should error "Cannot mix old and new format databases when running a dolt sql-server`

timsehn commented 2 years ago

This errors now. No panic. Resolving.