Bug Report: Snapshot keyspace restores always restore from latest backup

[pbibra]

Overview of the Issue

Context: https://vitess.slack.com/archives/C0PQY0PTK/p1683010265492109

When specifying a timestamp for a snapshot keyspace, we expect all tablets coming up in that keyspace to use the backup at or before the snapshot time. https://vitess.io/docs/17.0/reference/features/recovery/

There seems to be a regression where the restore code is always using the latest backup regardless of what the snapshot time is.

Reproduction Steps

1. Pull the latest from repo.
2. make build & cd examples/local
3. ./101_initial_cluster
4. From GetTablets pick a replica
5. vtctldclient --server localhost:15999 Backup zone1-0000000101

6. vtctldclient --server localhost:15999 CreateKeyspace --type=SNAPSHOT --base-keyspace=commerce --snapshot-timestamp=2023-07-19T18:16:00Z pitrexample                                                            
   Successfully created keyspace pitrexample. Result:
   {
   "name": "pitrexample",
   "keyspace": {
   "served_froms": [],
   "keyspace_type": 1,
   "base_keyspace": "commerce",
   "snapshot_time": {
     "seconds": "1689790560",
     "nanoseconds": 0
   },
   "durability_policy": "none",
   "throttler_config": null,
   "sidecar_db_name": "_vt"
   }
   }

7. Create another backup for a later timestamp vtctldclient --server localhost:15999 Backup zone1-0000000101

8. ls vtdataroot/backups/commerce/0                                                                                                                                                                                             
   2023-07-19.181518.zone1-0000000100 2023-07-19.181623.zone1-0000000100

9. update vttablet-up.sh to have --init_db_name_override vt_commerce
10. Run CELL=zone1 TABLET_UID=300 ../common/scripts/mysqlctl-up.sh
11. Run CELL-zone1 KEYSPACE=pitrexample TABLET_UID=300 ../common/scripts/vttablet-up.sh
12. Navigate to localhost:15300/debug/vars and check RestoredBackupTime

Binary Version

latest on main - also in release 14

Operating System and Environment details

macOS

Log Fragments

No response

[deepthi]

Related: https://github.com/vitessio/vitess/pull/8824 which has the change that broke this.