dtinth
commented
1 year ago Reconciliation script
require 'json'
# Load player list from .data/Player.json
players = JSON.parse(File.read('.data/Player (1).json'))
# Load legacy user list from .data/LegacyUser.json
legacy_users = JSON.parse(File.read('.data/LegacyUser.json'))
legacy_users_by_username = legacy_users.map { |u| [u['username'], u] }.to_h
# Load Auth0 user export from .data/bemuse.ndjson
auth0_users = File.read('.data/bemuse.ndjson').split("\n").map { |line| JSON.parse(line) }
auth0_users_by_id = auth0_users.map { |u| [u['Id'], u] }.to_h
auth0_users_by_nickname = auth0_users.map { |u| [u['Nickname'], u] }.to_h
# Load ranking entries
ranking_entries = JSON.parse(File.read('.data/RankingEntry.json'))
ranking_entries_by_player_id = ranking_entries.group_by { |e| e['playerId'] }
# Map each player to their email address
player_email = {}
used = {}
stats = {auth0: 0, legacy: 0, missing_safe: 0, missing_unsafe: 0}
players.each do |player|
id = player['_id']
linked_to = player['linkedTo']
auth0_user = auth0_users_by_id[linked_to] || auth0_users_by_nickname[id]
if auth0_user
player_email[id] = auth0_user['Email']
stats[:auth0] += 1
else
legacy_user = legacy_users_by_username[player['playerName']]
if legacy_user
player_email[id] = legacy_user['email']
stats[:legacy] += 1
else
if ranking_entries_by_player_id[id]
player_email[id] = "#{id}@reserved.bemuse.ninja"
stats[:missing_unsafe] += 1
p [player, ranking_entries_by_player_id[id].size]
else
stats[:missing_safe] += 1
end
end
end
if player_email[id]
(used[player_email[id]] ||= []) << player
end
end
used.select { |email, players| players.size > 1 }.each do |email, players|
puts email
players.each do |pl|
id = pl['_id']
n = (ranking_entries_by_player_id[id] || []).size
p [id, pl['playerName'], pl['linkedTo'], n]
if n == 0
player_email.delete(id)
end
end
end
p stats
File.write('.data/player-email.json', JSON.pretty_generate(player_email))Script to import into Firebase
process.env.GOOGLE_APPLICATION_CREDENTIALS = '.data/service-account.json';
import admin from 'firebase-admin'
import fs from 'fs'
import crypto from 'crypto'
const data = Object.entries(JSON.parse(fs.readFileSync('.data/player-email.json', 'utf8')))
admin.initializeApp()
function toFirebaseUser ([id, email]) {
const uid = 'M' + crypto.createHash('md5').update(id).digest('hex')
return {
uid,
email,
emailVerified: false,
// Generate a random password
passwordHash: crypto.createHash('sha256').update(crypto.randomBytes(32)).digest(),
passwordSalt: crypto.randomBytes(16),
}
}
// Loop 1000 users at a time and create them
for (let i = 0; i < data.length; i += 1000) {
const users = data.slice(i, i + 1000).map(toFirebaseUser)
const result = await admin.auth().importUsers(users, {
hash: {
algorithm: 'SHA256',
rounds: 1
}
})
console.log(JSON.stringify(result, null, 2))
for (const error of result.errors || []) {
const u = users[error.index]
console.log('==> Error importing user:', u, error)
}
}Script to generate commands to insert links into MongoDB
import fs from 'fs'
import crypto from 'crypto'
const data = Object.entries(JSON.parse(fs.readFileSync('.data/player-email.json', 'utf8')))
for (const [id] of data) {
const uid = 'M' + crypto.createHash('md5').update(id).digest('hex')
console.log(`db.FirebasePlayerLink.updateOne({ _id: '${id}' }, { $set: { firebaseUid: '${uid}' } }, { upsert: true });`)
}
Scoreboard server crashed due to outdated SSL certificates. It cannot be redeployed because Heroku stack is deprecated and its free tier is no more. Attempting to convert Auth0 custom database into an normal Auth0 database failed (see below). With that, I create an entirely new scoreboard system using Next.js with TypeScript. It will use the same database, but users will be migrated to Firebase Auth (so there is less headache due to Auth0’s increasingly complex system[^headache]). Everyone has to reset their password because exporting password hashes in Auth0 requires a paid plan.
Stats
[^headache]: I honestly tried to stay on Auth0 so that I don’t have to spend effort on migration. Reinventing wheels is not fun. First there’s confusion between Auth0.js and auth0-spa-js. I heard that
auth0-spa-jsis now recommended for SPAs and comes with built-in TypeScript definitions. Bemuse is an SPA, so I tried to migrate to it, but then it does not support the auth flows used in Bemuse without a significant rewrite. Ok, I stay withauth0-js, an older library which does not ship built-in type definitions. Now, the database connection is using a custom database. I used that because Bemuse’s online ranking service used to be hosted on Parse, which was shut down. So the users was exported from Parse and a Legacy User endpoints were created to facilitate trickle migration. Anyways, this was done back in 2017 and the custom database feature is no longer on Auth0’s free plan. Now when the old Heroku scoreboard server crashed, it also brought down the Legacy User endpoints, which caused all sign-ups to fail. Now that a vast majority of players have been migrated to Auth0 user store, I decided to use Auth0 user store exclusively, and get rid of legacy users (as they are no longer playing Bemuse). So I tried to disable the Use my own database option (so that the only source of truth for users is Auth0’s user store) but then I got an error: “options.enabledDatabaseCustomizationmust betrueifoptions.import_modeistrue”. Now that I am no longer importing users (most data is now in Auth0), I went ahead and disabled the Import Users to Auth0 option to resolve the previous error. Now I try to disable Use my own database again, this time it errors out with “This database contains users. You cannot change"options.enabledDatabaseCustomization"setting.” There is also a notice that said “You won't be able to read/delete the users if disabled. Delete all users before doing so.” What? To finish the migration I have to delete all users? This is ridiculous. Now I have a custom database that no longer works (and is no longer part of Auth0’s free plan). Anyways, now that most users are in Auth0 user store, I thought maybe I can create another Database connection that is not using the custom database feature. However, this does not work because user accounts are local to the database connection they originated from. Now one can not simply move users from one database connection to the other. We had to export users from one connection and import it back to the other connection. But this causes all passwords to be reset. To migrate to another connection with user passwords intact requires opening a support ticket which requires a paid plan. At this point I gave up and decided to switch to Firebase Authentication because its system is much simpler and does not impose a user limit. Firebase SDKs also ship with built-in TypeScript definitions, unlike Auth0.js which requires installing@types/auth0-jsseparately. It is only much much later that I found in the sea of documentation pages that the correct way to turn a custom database connection into a non-custom database connection is to adjust custom scripts so that a custom DB behaves like a non-custom DB but TOO LATE, I already migrated everything to Firebase Auth.