Open maxachis opened 3 months ago
automation.pdap.io now has two builds: One for Sandbox, and one for Stage. They should occur around the same time, but this allows us to be more flexible if we need to in the future.
Next up is the question of dummy data.
In some cases, we may not need any dummy data and can simply use data from the tables -- tables such as zip_codes
and state_names
don't contain any sensitive information, so we could just use those.
Then there are some cases where we simply don't need to include all that data -- for example, quick_search_query_logs
, as previously discussed, has an oversized amount of content that we don't need and which will slow things down.
We also want to be mindful that our schemas will change, so the more dummy data we believe we should possess, the more dummy data to modify if we adjust our schemas.
Additionally, some newly-created tables, such as requests_v2
, do not have any data at all, and adding some dummy data will help us test it out ahead of bringing it into production.
For now, I think we can get away with a handful of dummy data in sensitive tables such as users
, access_tokens
, session_tokens
, and data_requests
, empty tables such as requests_v2
, and importing en-masse from other tables, while ignoring quick_search_query_logs
.
Scripts modified so that sandbox database has non-sensitive data added from production. Next up is dummy data.
dummy_data
folder in repository, with empty csvs for each table to fill with dummy dataAdditionally, in the course of developing this logic, I came to the conclusion that Python would be preferable to shell scripts, and made an issue accordingly.
@josh-chamberlain I'll need to be made a member of the Notion workspace to add a "Testing" page with information on the Stage and Sandbox databases ⛑️
Nice work! I like your optionals, too.
Discussed previously in #340, here is the implementation!
TODO
Sandbox
database in Digital Ocean, with same specs aspdap-db-dev
(which unfortunately cannot be easily renamed).Prod-To-Dev-Migration
job in automation.pdap.io to reflect its new behavior.Optional/Debatable TODO
pdap-db-dev
and build new databasepdap-db-stg
, redirecting all stage connections to this database.