This pull request updates our migrate-links and migrate-scripts to run a "catchup" after our initial data migration.
How should this be reviewed?
I've split this work into one commit per script:
🐍 I added a list-keys.py script in 493e785. This lists out any shortlink keys that would be generated between the two given Bertly "counters", using Python's short_url module. We'll then use this in our new link migration script...
🔗 Updates the link migration script to read that list of keys in f2c8daa (rather than iterating over the whole database with SCAN). This uses gen-readlines to efficiently read each line from the file we created above.
🐁 Updates the click migration script to only return clicks after a given date in b87f10b. This allows us to specifically migrate clicks we know were created after the initial migration (rather than iterating over millions that we know already exist).
🍃 Finally, I removed an old helper that we're no longer using in 4d89841.
Any background context you want to provide?
This is the last major step before we can flip the switch for Bertly 2.0 in production!
What's this PR do?
This pull request updates our
migrate-linksandmigrate-scriptsto run a "catchup" after our initial data migration.How should this be reviewed?
I've split this work into one commit per script:
🐍 I added a
list-keys.pyscript in 493e785. This lists out any shortlink keys that would be generated between the two given Bertly "counters", using Python'sshort_urlmodule. We'll then use this in our new link migration script...🔗 Updates the link migration script to read that list of keys in f2c8daa (rather than iterating over the whole database with
SCAN). This usesgen-readlinesto efficiently read each line from the file we created above.🐁 Updates the click migration script to only return clicks after a given date in b87f10b. This allows us to specifically migrate clicks we know were created after the initial migration (rather than iterating over millions that we know already exist).
🍃 Finally, I removed an old helper that we're no longer using in 4d89841.
Any background context you want to provide?
This is the last major step before we can flip the switch for Bertly 2.0 in production!
Relevant tickets
References Pivotal #173478226.
Checklist