ppy / osu-infrastructure

40 stars 5 forks source link

Infrastructure deployment tasks for path-to-ranking #23

Open peppy opened 1 year ago

peppy commented 1 year ago

From a high level, tracking the order of deployment tasks that have dependencies on other changes.

### Tasks
- [ ] https://github.com/ppy/osu-infrastructure/issues/22
- [x] Apply https://github.com/ppy/osu-infrastructure/issues/15 (required for `osu-queue-score-statistics` changes in `master` for rank count increase efficiency)
- [ ] https://github.com/ppy/osu/issues/24229
- [ ] https://github.com/ppy/osu-infrastructure/issues/16
- [ ] https://github.com/ppy/osu-infrastructure/issues/13
- [ ] https://github.com/ppy/osu-infrastructure/issues/21
- [ ] https://github.com/ppy/osu-elastic-indexer/pull/153
- [ ] https://github.com/ppy/osu-infrastructure/issues/25
- [x] Update `osu-queue-score-statistics` after next nuget build (mod multiplier changes)
- [x] Check on `created_at` being `0000-00-00 00:00:00` after import (https://github.com/peppy/osu-queue-score-statistics/commit/f1cc5d9c883894fd534266a4a38605c25ef781fe)
- [ ] https://github.com/ppy/osu-queue-score-statistics/issues/63
- [ ] https://github.com/ppy/osu-queue-score-statistics/issues/141
- [x] Remove `updated_at` column from `scores` table (pending on osu-web to stop using it)
- [x] Consider moving `scores` to block storage
- [x] Re-import stable scores to `scores` table
- [ ] Update stable score import watcher with new command style and pp processing considerations
- [ ] Add datadog coverage of queue lengths, indexing, etc.
- [x] Profile and optimise live `osu-server-spectator`
- [ ] Recreate `score_tokens` (removed to disable score submission via non-`lazer.` endpoint) and don’t forget partitioning
- [ ] Switch osu-web to new branch
- [ ] Consider dropping `score_process_history` (deemed likely unnecessary)
- [ ] Migrate old lazer scores across (including multiplayer linking for playlists)
- [ ] Consider migrating pinned scores
- [ ] Consider removing modded entries from `osu_beatmap_difficulty` (deemed likely unnecessary)
- [ ] Re-run indexing on ES index for all rulesets

Testing notes for self:

To reset everything:

# nuke all indices
osu.ElasticIndexer# dotnet run index nuke

# view all indices
curl -X GET "localhost:9200/_cat/indices?v&pretty"

# restart es service
systemctl restart elasticsearch

truncate table scores;truncate table score_legacy_id_map;truncate table score_performance;

Ongoing stuff which needs to be run:


# osu-queue-store-statistics
cd ~/repos/osu-queue-score-statistics/osu.Server.Queues.ScoreStatisticsProcessor
git fetch; git reset --hard peppy/new-table-names
SCHEMA=20231208 dotnet run queue import-high-scores --start-id 0

# osu-elastic-indexer
cd ~/repos/osu-elastic-indexer/osu.ElasticIndexer
git fetch --all; git reset --hard peppy/new-table-names
SCHEMA=20231208 dotnet run queue watch
peppy commented 9 months ago

As a heads up, I've migrated all my remaining tasks from my private todo list to this issue, so it should be up-to-date with everything remaining.