Create migration toolkit

[GoogleCodeExporter]

Idea
If we have changed models in application, this can create a conflict between 
old and new model structure.

For example, if we have added new required property to model, and have several 
records in database - we see error message like this "Data is incorrect". We 
can't load data from datastore to model instance in this case.

To fix this problem, we should create migration toolkit for convert already 
exists data in datastore from old to new model format.

I think that we need to add versions for each model record. For example, we 
have Article model version #1 with fields: name (StringProperty), author 
(UserProperty).

In next release, we make changes in Article model, and assign version 2. We 
have added new property "category" (StringProperty). We should detect what 
changes made with our model from version 1 to 2.

To do this we want to create special file "versions.py" in each application. 
This file contain information about current version of models in current 
application, and also history of changes between all versions of each model.

GAE framework automatically migrate data from old to new format. Also, on this 
step GAE framework create backup for all old models data, archivate this data 
and send to administrator email. After this GAE framework make changes with 
datastore.

I think, that we can create file "versions.py" in format:

# migrate data from specified version
# if return False - than this is newest version of model
def upgrade_Article(from_version):
  if from_version == 1:
    # we have added new property "category"
    records = Article.all()
    for record in records:
      record.category = "Main category"
      record.put()
    return True
  elif from_version == 2:
    # have changed type of property "category" from StringProperty to ReferenceProperty
    # create default category
    main_category = ArticleCategory(name="Default")
    records = Article.all()
    for record in records:
      record.category = main_category
      record.put()
    return True
  return False

Recommendations
We can store information about version of each saved record in hidden field 
(for example named "version").

Imagine next use case. User use blog app v1 and added 100 blog posts. In the 
future user upgrade blog application to v3. We try load some blog entities and 
see, that version of blob entity is not 3 (created in the older version of blog 
app). In this case we run migration of this record. For example, we can delete 
some properties.

How to detect, what changes we need do with old record for migrate to new style 
record? We can find this in file "apps/app_name/history.py", where we have 
described how to migrate data from one version to another. This file should 
contain separate function for migrate data from one version to next. For 
example:

# migrate data from v1 to v2
from gae.db import migrate
def migrate_record(record, from_version, to_version):
  # migrate version from 1 to 2
  if from_version == 1:
    migrate(record,
      add={"subscribers": 0})
  # from 2 to 3 version
  if from_version <= 2:
    migrate(record,
      delete=("not_used_property"),
      # rename property from "admin" to "manager"
      rename={"admin": "manager"})

Original issue reported on code.google.com by anton.danilchenko on 28 Jan 2011 at 8:33