Seed Data Guidelines

[vgeorge]

Background

The data used by the web application is read-only, but the underlying model is under active development and will change over time. To ensure the platform remains stable during data updates, we need to implement a robust data management strategy.

Versioning Approach

A simplified versioning approach can help managing changes in the seed data packages and maintain stability.

If a breaking change occurs, such as a change in column type or name, a new version of the data files will be placed in a new folder with a versioned name. Adding new columns is also considered a breaking change because it might cause the ingest process to fail.

New versions will start with the suffix -dev to indicate they are under active development. Once a version has been sufficiently tested, the -dev suffix will be removed, indicating it is a final version with no further breaking changes.

Example:

• Current version: v0
  • Folder name: v0
  • Files: admin_centroids.gpkg, admin_limits.gpkg
• During development: v1-dev
  • Folder name: v1-dev
  • Files: admin_centroids.gpkg, admin_limits.gpkg, airports.gpkg
• After testing: v1
  • Folder name: v1
  • Files: admin_centroids.gpkg, admin_limits.gpkg, airports.gpkg

Data Management

Data files will be stored in the cloud service like Google Drive or S3. The web app will implement a basic check in the seed process to ensure the data version matches the required version. This will help us avoid any potential issues caused by incompatible data updates.

To Discuss

• The data package can become quite large. How would this impact the versioning approach? We might need to discard older versions.
• It looks like using Goggle Drive would be easier for managing the data as most of the team should familiar with it. It there any reason we should not use it?
• Are there any other questions or concerns not addressed by this proposed approach?

cc @oliverroick @wrynearson @cameronkruse

[oliverroick]

This is a great idea @vgeorge.

It looks like using Goggle Drive would be easier for managing the data as most of the team should familiar with it. It there any reason we should not use it?

What I can see is that we need access to the data when we want to automate deployments, I'm not sure Google Drive allows programatic access. I think better ways to manage this are:

• S3 and we give the deployment script read access to the bucket.
• Commit it to Git using Large File Storage, which stores files on a server instead as part of the repository, which should keep the repo size in check.