After upgrade from 2.11 vshard won't work until restart

[yngvar-antonsson]

Steps to reproduce:

1) Set up any 2.11 cluster. I used cartridge. If you need an example, see here.

2) Setup 3.1 vshard cluster. Don't forget to use uuids from your snaps.

---
credentials:
  users:
    admin:
      password: secret-cluster-cookie
      roles:
      - super
      - sharding
groups:
  default:
    replicasets:
      s-1:
        sharding:
          roles:
          - storage
        database:
          replicaset_uuid: e95af557-d80a-4446-b8e2-377ec3c421be
        leader: s-1-1
        instances:
          s-1-1:
            iproto:
              listen:
              - uri: localhost:3302
            database:
              instance_uuid: a472f21d-ad28-4c21-83c4-97c01875b6ee
      s-2:
        sharding:
          roles:
          - storage
        database:
          replicaset_uuid: 424cb45c-ab53-4773-8ecc-92f64ca3432d
        leader: s-2-1
        instances:
          s-2-1:
            iproto:
              listen:
              - uri: localhost:3304
            database:
              instance_uuid: 878631e3-1ca1-4e6a-958f-5996307c33df
      router:
        sharding:
          roles:
          - router
        database:
          replicaset_uuid: 8e5a9f64-5617-48e7-be86-685f4f807973
        leader: router-1
        instances:
          router-1:
            iproto:
              listen:
              - uri: localhost:3301
            database:
              instance_uuid: f0aa9cb7-fdd5-4ea6-8fe4-c9a697c0fe26
sharding:
  bucket_count: 30000
iproto:
  advertise:
    peer:
      login: admin
    sharding:
      login: admin
replication:
  failover: manual
...

3) Run tarantool 3.1

TT_SNAPSHOT_DIR=./tmp/data/myapp.router TT_WAL_DIR=./tmp/data/myapp.router/ tarantool --name router-1 --config result-config.yml &
TT_SNAPSHOT_DIR=./tmp/data/myapp.s1-master TT_WAL_DIR=./tmp/data/myapp.s1-master/ tarantool --name s-1-1 --config result-config.yml &
TT_SNAPSHOT_DIR=./tmp/data/myapp.s2-master TT_WAL_DIR=./tmp/data/myapp.s2-master/ tarantool --name s-2-1 --config result-config.yml &

4) Upgrade schema on every node:

box.schema.upgrade()
box.snapshot()

5) VShard doesn't work. Storages aren't configured, discovery isn't on.

require'vshard'.storage.info()

---
- error:
    reason: storage is not configured
    code: 33
    type: ShardingError
    message: 'Storage is disabled: storage is not configured'
    name: STORAGE_IS_DISABLED
...

6) Now if you restart the cluster, vshard configures storages and routers. (Sometimes you can restart the router only, but I can't guarantee the same result)

[Totktonada]

On the first glance, the problem is on the tarantool's vshard configuration code side, because (AFAIR) the status in config:info() is startup_in_progress in Igor's scenario. It shouldn't occur.

I'll try to reproduce it and will return with details.

[Totktonada]

<...> the status in config:info() is startup_in_progress in Igor's scenario. It shouldn't occur.

The reason is https://github.com/tarantool/tarantool/issues/9849.

[Totktonada]

https://github.com/tarantool/tarantool/issues/9849 is solved now. The fix will be released in 3.2.0 and 3.1.2 versions.

I hope that it resolves the most of the confusion with the upgrading. If there are more problems, please, let me know.

[sergepetrenko]

@Totktonada, can we close this then?

[Totktonada]

@sergepetrenko I've asked Igor about it. Waiting for the answer.

[yngvar-antonsson]

Closed, thank you!