Migrating Geonode 2.0 Application (Postgresql Table) data to Geonode 2.4

[donaldbales]

I'm working on migrating data for John Jediny's nepanode.anl.gov site, which is Geonode 2.0, to a new site that will use the latest version Geonode 2.4. From my perspective, there are three parts to this process: 1) Migrate Geonode 2.0's application data, typically found in Postgresql database "geonode", schema "geonode" to Geonode 2.4's application database "geonode", schema "public". 2) Migrate GeoServer's upload data store, typically found in Postgresql database "geonode_upload", schema "public" to Geonode 2.4's application database "datastore", schema, "public". 3) Migrate GeoServer's data directory.

At this point in time, I'm working on step 1. To that end, I've written a Python program named "createmigrate", that opens two connections, one for the source (Geonode 2.0) database and a second for the destination (Geonode 2.4) database. Next it compares the tables in each, providing a list of tables in the source, but not in the destination, a second list of tables that exist in the destination, but not in the source, and a third list of tables that exist in both. The it writes a set of "migrate.py" Python programs that can perform the migration. Why write a set of programs? Because each program has a section where select values are assigned to insert values, and it is there that a programmer can customize each program, if needed, to perform translations, ID substitutions, assign default values, etc..

For Geonode 2.0 to 2.4 here is the lists it produced:

Tables in src not in dst:

auth_user auth_user_groups auth_user_user_permissions base_thumbnail django_comment_flags django_comments notification_noticequeuebatch notification_noticesetting notification_noticetype people_role people_role_permissions security_genericobjectrolemapping security_objectrole security_objectrole_permissions security_userobjectrolemapping south_migrationhistory tagging_tag tagging_taggeditem taggit_templatetags_amodel

I've removed from this list captcha and zinnia tables that are unique modifications to NEPAnode.

Tables in dst not in src:

account_signupcodeextended base_license celery_taskmeta celery_tasksetmeta djcelery_crontabschedule djcelery_intervalschedule djcelery_periodictask djcelery_periodictasks djcelery_taskstate djcelery_workerstate groups_groupinvitation groups_groupmember groups_groupprofile guardian_groupobjectpermission guardian_userobjectpermission layers_layerfile layers_uploadsession maps_mapsnapshot people_profile_groups people_profile_user_permissions services_service services_servicelayer services_serviceprofilerole services_webserviceharvestlayersjob services_webserviceregistrationjob tastypie_apiaccess tastypie_apikey

Tables in both:

account_account account_accountdeletion account_emailaddress account_emailconfirmation account_signupcode account_signupcoderesult actstream_action actstream_follow agon_ratings_overallrating agon_ratings_rating announcements_announcement announcements_dismissal auth_group auth_group_permissions auth_permission avatar_avatar base_contactrole base_link base_region base_resourcebase base_resourcebase_regions base_restrictioncodetype base_spatialrepresentationtype base_topiccategory dialogos_comment django_admin_log django_content_type django_session django_site documents_document layers_attribute layers_layer layers_layer_styles layers_style maps_map maps_maplayer people_profile taggit_tag taggit_taggeditem upload_upload upload_uploadfile user_messages_message user_messages_thread user_messages_userthread

First issue found: I have found that Geonode 2.0's auth_user and people_profile tables have been merged in Geonode 2.4, and accordingly, I modified the generated program migrate_people_profile.py.

Second issue found: I have found that there are a set of configuration and/or code tables, where the "meaning" of the ID values has not been preserved from Geonode 2.0 to Geonode 2.4. May I suggest that in the future greater attention be paid to the reuse of such IDs? The only solution I have at this time is to build a mapping table from the old values to the new values, and swapping the values on-the-fly in the migration programs. That is a lot of extra work that may negative impact the adoption of Geonode going forward, and I don't want to see that happen.

Third issue: concerning the list of Geonode 2.0 tables that are not in Geonode 2.4, do you have any documentation or explanation as to why? Were these table renamed, merged, or did they become obsolete?

Fourth issue: concerning the list of Geonode 2.4 tables that are not in Geonode 2.0, do you have any documentation or explanation as to why? Are these table renamed, merged, or really new?

John Jediny and I would like to share our migration work so others who are in a similar situartion may benefit. How do we go about doing that within the structure of your project?

[capooti]

I fixed with my last commit the problem with value 'None' in object_pk field for some records. This was generated by GeoNode 2.0 leaving orphan records in its security tables.

[jj0hns0n]

@capooti @simod can we close this one now?

[capooti]

Never came with a PR, but the code mentioned in the thread has been used for a couple of migrations already.

On Wed, Nov 18, 2015 at 2:55 PM, Jeffrey Johnson notifications@github.com wrote:

@capooti https://github.com/capooti @simod https://github.com/simod can we close this one now?

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-157843573.

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

[abersh]

Hi simod !!! I'm configure django 1.6.1 and download geonode source code I want to run this code in to my django but there is no manage.py what shall i do please !!!

[simod]

@abersh hi, you can follow the docs in order to install GeoNode. I suggest you to start with the quick install guide as per http://www.geonode.org. here additional docs http://docs.geonode.org/en/master/tutorials/install_and_admin/geonode_install/index.html#geonode-install.

Please use the mailing lists to ask for support. This is not the right place. :)