basedosdados / backend

Backend da BD
https://backend.basedosdados.org/graphql
GNU General Public License v3.0
8 stars 1 forks source link

feat: populate staging and dev with prod data #636

Closed jhonylucas74 closed 1 month ago

jhonylucas74 commented 1 month ago

Purpose

Sync the data from production with the dev and staging environments to allow tests to be run on these projects in conditions closer to reality.

Description

In this PR, I have created two commands that can also be used in local development. One of them retrieves all the necessary data from Metabase in production. This will save all the relevant tables as JSON files in a folder to be used.

The second command cleans the database (except for things related to users and Stripe) and loads all the tables from Metabase into the database.

This PR also includes a cron task that, for testing purposes, will run every 2 hours but the final goal is to run it once a week. A custom model has also been added to the admin to control and monitor the tasks executed.

Checklist

Testing and evidence

image

Next steps

Test only in staging for now. If successful, deploy to dev.