migration story: python version agnotic interface

[zevdg]

So the best case scenario for migrating large python projects is to find versions of all your dependencies that are compatible with both python 2 and 3 so that you can test your python2/3 migration specific changes in isolation from changes to your dependencies and how you call them. The migration story looks something like this.

1. update your dependencies to py2/3 compatible versions
2. modernize your own code
3. incrementally roll-out the py3 version of your application
4. once py3 deployment is stable, py3 only dependencies can be safely adopted

Since this project was written from the ground up using python 3 only features, AND with a different API from python-ndb, AND the old library will not be modernized, that migration story isn't possible. Instead, we have something like

1. update dependencies except python-ndb to py2/3 compatible versions
2. modernize your own code AND migrate from python-ndb to endpoints-proto-datastore
3. incrementally roll-out the py3 version of your application
4. once py3 deployment is stable, other py3 only dependencies can be safely adopted

This isn't ideal because issues with the py2->py3 migration aren't isolated from the python-ndb -> endpoints-proto-datastore migration, but it's doable. However, when ndb is a transitive dependency and many apps rely on the intermediate dependency, this solution becomes harder. Throw in one or two more dependencies like this one and it's a nightmare. The right thing to do is to have a language agnostic interface at some level. It's still not great, because you're still mixing a drastic dependency change in with the py3 migration, but it becomes becomes tenable. You can make the intermediate lib the language agnostic layer, but that only solves the problem for that one library. The better solution is to create a new generic layer that sits right on top of python-ndb and/or endpoints-proto-datastore and change all callers of python-ndb to use that until the apps that depend on them have finished migrating.

This new layer needs an interface flexible enough that it could be implemented using python-ndb or endpoints-proto-datastore. Ideally, this interface would be totally identical to either the old or the new API (aka a shim), but that isn't strictly necessary.

All that said, my actual questions are:

• Do you have any plans of creating a migration layer like this?
• If not, do you know of another project that is?
• If not
  • Could a shim work, or do we need a 3rd interface?
  • Roughly what order of magnitude would you expect this work to be? (e.g. trivial, moderate, very difficult, nigh impossible)

[dhermes]

This library can be updated to run in both Python 2 and 3 with six (I have quite a bit of experience doing this).

The ndb (and possibly other App Engine) imports can be put in a "cage" that does a try / except and provides a standard interface. Not sure how possible this will be, I haven't been following python-ndb since leaving the project

[zevdg]

sorry. i meant to put this issue on https://github.com/googleapis/python-ndb. I'm going to update and repost it over there.