Migrating Geonode 2.0 Application (Postgresql Table) data to Geonode 2.4

[donaldbales]

I'm working on migrating data for John Jediny's nepanode.anl.gov site, which is Geonode 2.0, to a new site that will use the latest version Geonode 2.4. From my perspective, there are three parts to this process: 1) Migrate Geonode 2.0's application data, typically found in Postgresql database "geonode", schema "geonode" to Geonode 2.4's application database "geonode", schema "public". 2) Migrate GeoServer's upload data store, typically found in Postgresql database "geonode_upload", schema "public" to Geonode 2.4's application database "datastore", schema, "public". 3) Migrate GeoServer's data directory.

At this point in time, I'm working on step 1. To that end, I've written a Python program named "createmigrate", that opens two connections, one for the source (Geonode 2.0) database and a second for the destination (Geonode 2.4) database. Next it compares the tables in each, providing a list of tables in the source, but not in the destination, a second list of tables that exist in the destination, but not in the source, and a third list of tables that exist in both. The it writes a set of "migrate.py" Python programs that can perform the migration. Why write a set of programs? Because each program has a section where select values are assigned to insert values, and it is there that a programmer can customize each program, if needed, to perform translations, ID substitutions, assign default values, etc..

For Geonode 2.0 to 2.4 here is the lists it produced:

Tables in src not in dst:

auth_user auth_user_groups auth_user_user_permissions base_thumbnail django_comment_flags django_comments notification_noticequeuebatch notification_noticesetting notification_noticetype people_role people_role_permissions security_genericobjectrolemapping security_objectrole security_objectrole_permissions security_userobjectrolemapping south_migrationhistory tagging_tag tagging_taggeditem taggit_templatetags_amodel

I've removed from this list captcha and zinnia tables that are unique modifications to NEPAnode.

Tables in dst not in src:

account_signupcodeextended base_license celery_taskmeta celery_tasksetmeta djcelery_crontabschedule djcelery_intervalschedule djcelery_periodictask djcelery_periodictasks djcelery_taskstate djcelery_workerstate groups_groupinvitation groups_groupmember groups_groupprofile guardian_groupobjectpermission guardian_userobjectpermission layers_layerfile layers_uploadsession maps_mapsnapshot people_profile_groups people_profile_user_permissions services_service services_servicelayer services_serviceprofilerole services_webserviceharvestlayersjob services_webserviceregistrationjob tastypie_apiaccess tastypie_apikey

Tables in both:

account_account account_accountdeletion account_emailaddress account_emailconfirmation account_signupcode account_signupcoderesult actstream_action actstream_follow agon_ratings_overallrating agon_ratings_rating announcements_announcement announcements_dismissal auth_group auth_group_permissions auth_permission avatar_avatar base_contactrole base_link base_region base_resourcebase base_resourcebase_regions base_restrictioncodetype base_spatialrepresentationtype base_topiccategory dialogos_comment django_admin_log django_content_type django_session django_site documents_document layers_attribute layers_layer layers_layer_styles layers_style maps_map maps_maplayer people_profile taggit_tag taggit_taggeditem upload_upload upload_uploadfile user_messages_message user_messages_thread user_messages_userthread

First issue found: I have found that Geonode 2.0's auth_user and people_profile tables have been merged in Geonode 2.4, and accordingly, I modified the generated program migrate_people_profile.py.

Second issue found: I have found that there are a set of configuration and/or code tables, where the "meaning" of the ID values has not been preserved from Geonode 2.0 to Geonode 2.4. May I suggest that in the future greater attention be paid to the reuse of such IDs? The only solution I have at this time is to build a mapping table from the old values to the new values, and swapping the values on-the-fly in the migration programs. That is a lot of extra work that may negative impact the adoption of Geonode going forward, and I don't want to see that happen.

Third issue: concerning the list of Geonode 2.0 tables that are not in Geonode 2.4, do you have any documentation or explanation as to why? Were these table renamed, merged, or did they become obsolete?

Fourth issue: concerning the list of Geonode 2.4 tables that are not in Geonode 2.0, do you have any documentation or explanation as to why? Are these table renamed, merged, or really new?

John Jediny and I would like to share our migration work so others who are in a similar situartion may benefit. How do we go about doing that within the structure of your project?

[simod]

Hi @donaldbales thanks for reporting this, amazing work! I try to explain the reasons of al those changes here.

Tables in 2.0 and not in 2.4.

The biggest changes at database level is regarding the permissions system, in 2.4 we have introduced a user based permission system based on djangoguardian that allows us to define more granular permissions and users to define their own if needed. This change impacted the security tables, all that security* and some people_* , some of them became useless. We also swapped the django_auth_user with the people_profile table that now inherits from django_auth. This because we found no reasons to keep two tables to store users information and at the same time the info on the default django user table were not enough. The thumbnail table is gone and the information of the thumbnail is now just in the base_link table. The taggit tables should actually still be there. The notifications tables are just optional, there notification app is commented in settings. The south tables are gone as they were the old migrations.

Tables in 2.4 and not in 2.0

Groups tables are the new support for groups permissions. The groups are the standard django_group and the permissions are assigned by django_guardian to them. On top of the django_group there is the group_groupprofile which is the table used as Group in geonode. The permissions to the anonymous user (that now is a user called AnonymousUser) are granted to a django_group called "anonymous" which does not have a group_groupprofile associated (does not show up in geonode) and, by default, every user belongs to this group.

Celery tables (queue system) are new same as tastypie (APIs) and services(remote services).

Some of the tables that are in both may have slightly changed.

Although this is not a complete explanation I hope that it helps, don't hesitate to as more questions and again thanks for the work.

[alexpaulooliveira]

Hello everybody,

My name is Alex. I work at Embrapa Coastal Tablelands (Brazilian Agricultural Research Corporation). We need to migrate some data from version 2.0 to version 2.4 of GeoNode. Mr. Donald Bales has explained that there are significant differences between these two versions of the database (2.0 and 2.4)

If you have already taken the time to understand where each column of each table in the source version (2.0) needs to be copied to the target version (2.4), I agree to write, and deliver to the masters, the script (in PL/pgSQL or Python, as suggested by the masters) who will do all bank charge in version 2.0 to the bank in version 2.4.

Thank you for interaction. Best Regards, Alex.

[capooti]

here is a migration script that should be able to migrate any GeoNode 2.0 database instance to GeoNode 2.4: https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/index.rst

feedback is welcome, when we are sure everything works correctly, I will create a PR with it.

[donaldbales]

Congrats on the new book.

Sent from my iPhone

On Jun 18, 2015, at 7:42 AM, Paolo Corti notifications@github.com wrote:

here is a migration script that should be able to migrate any GeoNode 2.0 database instance to GeoNode 2.4: https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/index.rst

feedback is welcome, when we are sure everything works correctly, I will create a PR with it.

— Reply to this email directly or view it on GitHub.

[davicustodio]

Hi Paolo, me and Alex Oliveira are working on the same mission, to migrate a GeoNode 2.0 database with over 1000 layers for the GeoNode 2.4 database.

Giving feedback on our experience:

1 - I installed the 2.4 GeoNode 2 - I made backups 3 - I run a "GeoNode syncdb" 4 - I edited the "gn_migration.cfg" as:

[Db20] dbname = geonode20 host = localhost user = postgres password = *****

[DB24] dbname = geoNode host = localhost user = postgres password = ****

[Path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = /etc/geonode/local_settings.py

.. Just I was not sure if properly configured the session [path]?

5 - I run the ./migrate.sh

The script began to work writing the log on screen ... and after about 2 minutes, he began printing errors and exceptions like this:

<Class 'psycopg2.IntegrityError'> insert or update on table "layers_attribute" violates foreign key constraint "layers_attribute_layer_id_fkey" DETAIL: Key (layer_id) = (554) is not present in table "layers_layer". layer_id, attribute, description, attribute_label, attribute_type, visible, display_order, count, min, max, average, median, stddev, sum, unique_values, last_stats_updated from layers_attribute (904, 'Base' None, 'Base', 'xsd: double' True, 8, 0, 'NA', 'NA', 'NA', 'NA', 'NA', 'NA' ' 'datetime.datetime (2015, 3, 13, 11, 52, 38, 543 701, TzInfo = psycopg2.tz.FixedOffsetTimezone (offset = -180, name = None)))

.. Were many such errors (psycopg2.IntegrityError)

I do not understand why they are happening, as the GeoNode 2.4 database should be ready after the "geoNode syncdb".

I messed up at some step? Do you have any idea about the problem?

Thanks

On Thu, Jun 18, 2015 at 2:30 PM, Donald J. Bales notifications@github.com wrote:

Congrats on the new book.

Sent from my iPhone

On Jun 18, 2015, at 7:42 AM, Paolo Corti notifications@github.com wrote:

here is a migration script that should be able to migrate any GeoNode 2.0 database instance to GeoNode 2.4:

https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/index.rst

feedback is welcome, when we are sure everything works correctly, I will create a PR with it.

— Reply to this email directly or view it on GitHub.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-113228386.

Davi de O. Custódio

[capooti]

Hi Davi apparently the error is in migrate_attributes.py and it is because a layer failed to be imported in migrate_layers.py. Could you run the script from scratch logging to a file and send it to me? Even better, could you share with me the backup of your django database, this way I could test myself? thanks

[davicustodio]

Paolo, I am sending to your email, dump my GeoNode 2.0 database for you to test. Thank you for your help.

On Mon, Jun 29, 2015 at 4:35 AM, Paolo Corti notifications@github.com wrote:

Hi Davi apparently the error is in migrate_attributes.py and it is because a layer failed to be imported in migrate_layers.py. Could you run the script from scratch logging to a file and send it to me? Even better, could you share with me the backup of your django database, this way I could test myself? thanks

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-116500124.

Davi de O. Custódio

[capooti]

@davicustodio I had a look to your database, the problem is that you customized the base_restrictioncodetype table adding one record in 2.0, and now there is not such a record. Look at this line: https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/migrate_resourcebases.py#L53 the script does not handle still this siutation. In order to complete your task, make sure the tables are the same (same number of records and ids), and you will see that the script correctly work - I can see all of the 1020 layers now in my test instance. Even better if you have time, try to add a migrate_restrictioncodetypes.py script in the procedure (and add a get_restrictioncodetypeid_by_oldid method in utils.py) to handle the situation and submit a PR :) We would need to do the same with the base_license table.

[davicustodio]

Paolo, I removed the record in question, the base_restrictioncodetype table table and several errors that were happening disappeared. Still, after the executed script, appeared in 1020 the available layers, but with one problem: when you click the title of the layer to access your details, nothing happens. For all layers of the names of links GeoNode this using: http://192.168.157.153/layers/?limit=100&offset=0, it will not go anywhere.

In the log they are still showing a large number of such errors:

<Class 'psycopg2.IntegrityError'> duplicate key value violates unique constraint "guardian_userobjectpermission_user_id_permission_id_object__key" DETAIL: Key (user_id, permission_id, object_pk) = (24, 118, None) already exists. select id, user_id, timezone, language from account_account (10006, 41, 216, 'Administrative', 'Adriana') (10006, 41, 215, 'Read / Write', 'adriana') Id for user adriana was 10006 and is now 24 Content type id 41 is now 41 Content type id is layer is 41 ['View_resourcebase', 'download_resourcebase', 'change_resourcebase_metadata', 'change_layer_data', 'change_layer_style'] Content type id is resourcebase is 38 Inserting permission view_resourcebase for user adriana for content None INSERT INTO guardian_userobjectpermission (permission_id, content_type_id, object_pk, user_id) VALUES (115, 38, 'None', 24)

<Class 'psycopg2.IntegrityError'> duplicate key value violates unique constraint "guardian_userobjectpermission_user_id_permission_id_object__key" DETAIL: Key (user_id, permission_id, object_pk) = (24, 115, None) already exists. select id, user_id, timezone, language from account_account (10006, 41, 215, 'Read / Write', 'adriana') Content type id is resourcebase is 38 Inserting permission download_resourcebase for user adriana for content None INSERT INTO guardian_userobjectpermission (permission_id, content_type_id, object_pk, user_id) VALUES (117, 38, 'None', 24)

I do not know if they are interfering directly in trouble with the link of the names of layers.

Thanks for your help.

On Tue, Jun 30, 2015 at 7:12 AM, Paolo Corti notifications@github.com wrote:

@davicustodio https://github.com/davicustodio I had a look to your database, the problem is that you customized the base_restrictioncodetype table adding one record in 2.0, and now there is not such a record. Look at this line: https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/migrate_resourcebases.py#L53 the script does not handle still this siutation. In order to complete your task, make sure the tables are the same (same number of records and ids), and you will see that the script correctly work

• I can see all of the 1020 layers now in my test instance. Even better if you have time, try to add a migrate_restrictioncodetypes.py script in the procedure (and add a get_restrictioncodetypeid_by_oldid method in utils.py) to handle the situation and submit a PR :) We would need to do the same with the base_license table.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-117111123.

Davi de O. Custódio

[capooti]

oh I see what it is causing the problem, sorry about it. A few minutes ago I made a commit with some corrections, basically the main problem was that a script was updating layers, but without geoserver running it was failing. I also fixed a couple of other problems, I should have tested this more extensively, you are my alpha tester :D Do a git pull now and restart the procedure following the updated instructions, it should work. And thanks for testing it!

[davicustodio]

Paolo, did git pull, performed one geonode syncdb, and restarted the migration process. After that, I checked the GeoNode, and all users were migrated, but 0 layers appeared. You could perform the migration using the bank that sent you, after we did the last commit ?

thanks

On Thu, Jul 2, 2015 at 7:23 AM, Paolo Corti notifications@github.com wrote:

oh I see what it is causing the problem, sorry about it. A few minutes ago I made a commit with some corrections, basically the main problem was that a script was updating layers, but without geoserver running it was failing. I also fixed a couple of other problems, I should have tested this more extensively, you are my alpha tester :D Do a git pull now and restart the procedure following the updated instructions, it should work. And thanks for testing it!

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-117992215.

Davi de O. Custódio

[capooti]

please note that, as suggested in the updated instruction, now the migration process consists of the following steps:

1. run migrate.py
2. restore geoserver
3. run migrate2.py

it looks like you are on step 1? Therefore you don't see your layers, as they have been loaded in db with default permissions: in this case only if you are authenticated you will be able to see them. Basically, you need to login in GeoNode :)

After running migrate2.py script, permissions will be correctly setup. Let me know if this works.

By the way, I did only step 1 with your backup, I cannot process further because I would need the geoserver data directory to run step 2.

[AnnalisaS]

I've run first version of the procedure and I loaded all layers, maps and people_profile on new postgresql but no groups nor permissions. I haven't run migrate.sh 'cause my ubuntu shell raised error here https://github.com/capooti/geonode/blob/migration_from20_to_24/scripts/migrations/migrate20to24/migrate.sh#L5 , so I run scripts separately. First error raised on "create_auth_group_and_update_res.py" then other errors from others scripts. Then I migrated Geoserver data directory and layer on postgresql and tried to run the first script of the new migrate2.sh : create_auth_group_and_update_res.py If I set "settings_path = /etc/geonode/local_settings.py" in the file "gn_migration.cfg", script raised this error: ... ImportError: Could not import settings '/etc/geonode/local_settings.py' (Is it on sys.path? Is there an import error in the settings file?): Import by filename is not supported.

If I set "settings_path = geonode.local_settings", script raised this error: Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py", line 12, in from guardian.shortcuts import get_objects_for_group File "/usr/lib/python2.7/dist-packages/guardian/shortcuts.py", line 16, in from guardian.core import ObjectPermissionChecker File "/usr/lib/python2.7/dist-packages/guardian/core.py", line 8, in from guardian.utils import get_identity File "/usr/lib/python2.7/dist-packages/guardian/utils.py", line 24, in from guardian.conf import settings as guardian_settings File "/usr/lib/python2.7/dist-packages/guardian/conf/settings.py", line 11, in raise ImproperlyConfigured("In order to use django-guardian's " django.core.exceptions.ImproperlyConfigured: In order to use django-guardian's ObjectPermissionBackend authorization backend you have to configure ANONYMOUS_USER_ID at your settings module

Any suggestions ? Thanks

[capooti]

do you have the /etc/geonode/local_settings.py file? can you try with settings_path = /etc/geonode/settings.py" ? sorry but I am not familiar with the configuration from package

[AnnalisaS]

Yes I have the /etc/geonode/local_settings.py file. The /etc/geonode/settings.py file doesn't exists. One settings.py file is on /usr/local/lib/python2.7/dist-packages/geonode. Here there are both settings.py and local_settings.py.

Both settings gave same error: ImportError: Could not import settings '/usr/local/lib/python2.7/dist-packages/geonode/local_settings.py' (Is it on sys.path? Is there an import error in the settings file?): Import by filename is not supported.

Maybe I've to add the path to PATH variable of env?

2015-07-08 11:31 GMT+02:00 Paolo Corti notifications@github.com:

do you have the /etc/geonode/local_settings.py file? can you try with settings_path = /etc/geonode/settings.py" ? sorry but I am not familiar with the configuration from package

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-119520162.

[capooti]

for sure the one in dist-packages is not to be used. my setup is the following (not using package, though):

geonode_path = /home/capooti/git/github/geonode settings_path = geonode.local_settings

I think your problem could not be related to the settings_path but to the geonode_path: how did you set it?

[AnnalisaS]

I set : [path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = /usr/local/lib/python2.7/dist-packages/geonode/local_settings.py

Or: [path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = geonode.local_settings

The second raised another error: Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py", line 12, in from guardian.shortcuts import get_objects_for_group File "/usr/lib/python2.7/dist-packages/guardian/shortcuts.py", line 16, in from guardian.core import ObjectPermissionChecker File "/usr/lib/python2.7/dist-packages/guardian/core.py", line 8, in from guardian.utils import get_identity File "/usr/lib/python2.7/dist-packages/guardian/utils.py", line 24, in from guardian.conf import settings as guardian_settings File "/usr/lib/python2.7/dist-packages/guardian/conf/settings.py", line 11, in raise ImproperlyConfigured("In order to use django-guardian's " django.core.exceptions.ImproperlyConfigured: In order to use django-guardian's ObjectPermissionBackend authorization backend you have to configure ANONYMOUS_USER_ID at your settings module

But if I print settings with "geonode print_settings" command, I find ANONYMOUS_USER_ID = -1 Do I send you my settings? Or should I install my geonode_project (https://github.com/GeoNode/geonode-project/tree/master) and use my new dir (home/my_geonode) as geonode_path?

[simod]

@AnnalisaS I think that settings_path should be = geonode.settings. this will also import local_settings.

[capooti]

@AnnalisaS @davicustodio did you manage to migrate the database?

[AnnalisaS]

I'm going to reply ... I set : [path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = geonode.settings (thanks simod :-) )

Then I run "create_auth_group_and_update_res.py" script for all layer and map. It finished late yesterday and now I see it gave the same error for all layer: Ex: Updating layer Inquadramenti 1:10000 < class 'socket.error' >

No error for maps. But now I see all layers as geonode user. Now I run others scripts and report result!

I've a proxy under the old server and now I've to make it point to new server. On geoserver proxy base url is already set to proxy: http://geomap.arpa.veneto.it/geoserver/ Then I have to run geonode-updateip geomap.arpa.veneto.it isn't it? Now there are some misconfigured links between geoserver and geonode.

[capooti]

It looks like geonode cannot connect to geoserver. before running the create_auth_group_and_update_res.py make sure that you can successfully log in geonode and geoserver and that in geoserver there are all the layers.

[AnnalisaS]

Now proxy (geomap) point to new server (bender2). I can log in geoserver on http://geomap.arpa.veneto.it/geoserver/ as admin and as geonode. I see all layers. Proxy base url is http://geomap.arpa.veneto.it/geoserver/ But http://geomap.arpa.veneto.it/ gives "Internal server error", even after updateip. On DB, links appears changed to geomap. http://bender2.arpa.veneto.it/ instead appears correctly, but, for example, identify tool doesn't work.

[davicustodio]

Hello Paolo. I'm sorry to have not returned our conversation before. I've been on vacation and just returned yesterday. I resumed the migration of my database, and how you had suggested, performed the following steps:

1 - run migrate.py 2 - restore geoserver 3 - run migrate2.py

Before, I did a git pull to update my repository in order to receive the migrate2.py.

In step 1, I received several errors like:

<Class 'psycopg2.IntegrityError'> insert or update on table "maps_maplayer" violates foreign key constraint "maps_maplayer_map_id_fkey" DETAIL: Key (map_id) = (1048) is not present in table "maps_map". select map_id, stack_order, format, name, opacity, styles, transparent, fixed, "group", visibility, ows_url, layer_params, source_params, location from maps_maplayer (1755, 1, None, 'OpenStreetMap', 1.0, None, False, True, 'background', False, None, '{"selected": false, "title": "OpenStreetMap" "args": ["OpenStreetMap "]," type ":" OpenLayers.Layer.OSM "} ',' {'ptype": "gxp_olsource", "id": "0"}', False) Resource id 1755 is now 1048

.. And after finishing, I checked the process base_resourcebase table and all the layers were imported .. but, upon entering the GeoNode (while logged in as administrator), I realized that appeared only 2 imported layers. (We have about 1000 layers). Before I update the migration scripts with the latest version, it was working.

Go on and restored the GeoServer, and performed step 3.

Right at the beginning of the process, it happened the following error:

Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File "/usr/lib/python2.7/dist-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File "/usr/lib/python2.7/dist-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File "/usr/lib/python2.7/dist-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File "/usr/lib/python2.7/dist-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 54, in getattr self._setup (name) File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings '/etc/geonode/local_settings.py' (Is it on sys.path Is there an import error in the settings file?): Import by filename is not supported.

I suspected a problem in local_settings.py address, but looked in gn_migration.cfg and looks like this:

[Path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = /etc/geonode/local_settings.py

Thereafter, a number of errors appear as:

('Rio Branco', 'Rio Branco') <Class 'psycopg2.IntegrityError'> duplicate key value violates unique constraint "taggit_tag_name_key" DETAIL: Key (name) = (Nazaré Paulista © municipality) already exists.

Returning to GeoNode again appear only 2 layers.

Do you think the problem in step 1 may be related to the latest changes you made in the script?

If you are willing and have some time, I can send you my base again.

Thanks.

On Thu, Jul 9, 2015 at 7:44 AM, Annalisa Schiavon notifications@github.com wrote:

Now proxy (geomap) point to new server (bender2). I can log in geoserver on http://geomap.arpa.veneto.it/geoserver/ as admin and as geonode. I see all layers. Proxy base url is http://geomap.arpa.veneto.it/geoserver/ But http://geomap.arpa.veneto.it/ gives "Internal server error", even after updateip. On DB, links appears changed to geomap. http://bender2.arpa.veneto.it/ instead appears correctly, but, for example, identify tool doesn't work.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-119906735.

Davi de O. Custódio

[davicustodio]

Hello Paolo. Please excuse me and also ignore my last message. I found I was running the script with that record more than I had in base_restrictioncodetype table. sorry

After deleting the record, I got all the layers appear in GeoNode the layers list.

Then restored the GeoServer, and run the migrate2.sh

I noticed a problem: the list of layers by clicking on one of them to access the details, the link is not working. All links of layers are appearing all this: http://192.168.157.153/layers/?limit=100&offset=0

Attached sending the log generated by migrate.sh and migrate2.sh

On Tue, Aug 4, 2015 at 4:54 PM, Davi Custodio davicustodio@gmail.com wrote:

Hello Paolo. I'm sorry to have not returned our conversation before. I've been on vacation and just returned yesterday. I resumed the migration of my database, and how you had suggested, performed the following steps:

1 - run migrate.py 2 - restore geoserver 3 - run migrate2.py

Before, I did a git pull to update my repository in order to receive the migrate2.py.

In step 1, I received several errors like:

<Class 'psycopg2.IntegrityError'> insert or update on table "maps_maplayer" violates foreign key constraint "maps_maplayer_map_id_fkey" DETAIL: Key (map_id) = (1048) is not present in table "maps_map". select map_id, stack_order, format, name, opacity, styles, transparent, fixed, "group", visibility, ows_url, layer_params, source_params, location from maps_maplayer (1755, 1, None, 'OpenStreetMap', 1.0, None, False, True, 'background', False, None, '{"selected": false, "title": "OpenStreetMap" "args": ["OpenStreetMap "]," type ":" OpenLayers.Layer.OSM "} ',' {'ptype": "gxp_olsource", "id": "0"}', False) Resource id 1755 is now 1048

.. And after finishing, I checked the process base_resourcebase table and all the layers were imported .. but, upon entering the GeoNode (while logged in as administrator), I realized that appeared only 2 imported layers. (We have about 1000 layers). Before I update the migration scripts with the latest version, it was working.

Go on and restored the GeoServer, and performed step 3.

Right at the beginning of the process, it happened the following error:

Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File "/usr/lib/python2.7/dist-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File "/usr/lib/python2.7/dist-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File "/usr/lib/python2.7/dist-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File "/usr/lib/python2.7/dist-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 54, in getattr self._setup (name) File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File "/usr/lib/python2.7/dist-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings '/etc/geonode/local_settings.py' (Is it on sys.path Is there an import error in the settings file?): Import by filename is not supported.

I suspected a problem in local_settings.py address, but looked in gn_migration.cfg and looks like this:

[Path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = /etc/geonode/local_settings.py

Thereafter, a number of errors appear as:

('Rio Branco', 'Rio Branco') <Class 'psycopg2.IntegrityError'> duplicate key value violates unique constraint "taggit_tag_name_key" DETAIL: Key (name) = (Nazaré Paulista © municipality) already exists.

Returning to GeoNode again appear only 2 layers.

Do you think the problem in step 1 may be related to the latest changes you made in the script?

If you are willing and have some time, I can send you my base again.

Thanks.

On Thu, Jul 9, 2015 at 7:44 AM, Annalisa Schiavon < notifications@github.com> wrote:

Now proxy (geomap) point to new server (bender2). I can log in geoserver on http://geomap.arpa.veneto.it/geoserver/ as admin and as geonode. I see all layers. Proxy base url is http://geomap.arpa.veneto.it/geoserver/ But http://geomap.arpa.veneto.it/ gives "Internal server error", even after updateip. On DB, links appears changed to geomap. http://bender2.arpa.veneto.it/ instead appears correctly, but, for example, identify tool doesn't work.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-119906735.

Davi de O. Custódio

Davi de O. Custódio

[capooti]

hi @davicustodio from the logs you sent me it appears that the create_auth_group_and_update_res.py script, that should have been run by migrate2, was not launched. Is this possible? Can you try to run it manually? Are you using the latest version of the code? (I made some commits after submitting the code)

[davicustodio]

We performed manually create_auth_group_and_update_res.py and really he's in trouble. The error message says:

"ObjectPermissionBackend authorization backend you have to configure ANONYMOUS_USER_ID at your settings module"

I tried using ANONYMOUS_USER_ID = 1 in local_settings.py, and then rerun the create_auth_group_and_update_res.py and the error message has changed to:

DeprecationWarning: django.utils.simplejson is deprecated; use json instead. from django.utils import simplejson the json

Creating the authenticated group ... Now adding all users to the group ... Adding to the admin group Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 26, in gp.join (profile) File "/usr/local/lib/python2.7/dist-packages/geonode/groups/models.py", line 135, in join if user == user.get_anonymous (): AttributeError 'Profile' object has no attribute 'get_anonymous'

You have no idea what can be?

On Tue, Aug 11, 2015 at 6:14 AM, Paolo Corti notifications@github.com wrote:

hi @davicustodio https://github.com/davicustodio from the logs you sent me it appears that the create_auth_group_and_update_res.py script, that should have been run by migrate2, was not launched. Is this possible? Can you try to run it manually? Are you using the latest version of the code? (I made some commits after submitting the code)

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-129784893.

Davi de O. Custódio

[capooti]

you shouldn't add the ANONYMOUS_USER_ID setting, as it is already in place: https://github.com/GeoNode/geonode/blob/master/geonode/settings.py#L397

are you running this on the server using GeoNode 2.4? Can you run a pip freeze and paste here the results?

[davicustodio]

Yes, at least I think so, I am using a 2.4 GeoNode

Below the result of the freeze pip:

Django == 1.6.1 GDAL == 1.10.1 GeoNode == 2.4b24 Landscape-Client == 14:12 MultipartPostHandler == 0.1.0 OWSLib == 0.8.10 PAM == 0.4.2 Paver == 1.2.1 Pillow == 2.3.0 PyYAML == 3:10 Shapely == 1.3.0 South == 0.7.5 Twisted-Core == 13.2.0 Unidecode == 12:04:14 AMQP == 1.3.3 anyjson == 0.3.3 apt-xapian-index == 12:45 argparse == 1.2.1 awesome-slugify == 1.6.2 beautifulsoup4 == 4.2.1 billiard == 3.3.0.15 celery == 3.1.6 chardet == 2.0.1 cl == 0.0.3 Colorama == 0.2.5 ConfigObj == 4.7.2 dialogues == 0.4 django-activity-stream == 0.4.5beta1 django-appconf == 0.6 django-autocomplete-light == 1.4.14 django-bootstrap3-datetimepicker == 2.2.3 django-celery == 3.1.16 django-downloadview == 1.2 django-extensions == 1.2.5 django-forms-bootstrap == 3.0.1 django-friendly-tag-loader == 1.1 django-GeoExplorer == 4.0.4 django-guardian == 1.2.0 django-jsonfield == 0.9.12 django-leaflet == 0.13.7 django-modeltranslation == 0.8b3 django-mptt == 0.6.0 django-nose == 1.2 django-pagination == 1.0.5 django-polymorphic == 0.5.6 django-taggit == 0.11.2 django-taggit-templatetags == 0.4.6dev django-tastypie == 0.9.15 django-templatetag-sugar == 0.1 geolinks == 0.0.1 GeoNode-agon-ratings == 0.3.1 GeoNode-announcements == 1.0.6 GeoNode-arcrest == 10.2 GeoNode-avatar == 2.1.5 GeoNode-notification == 1.1.1 GeoNode-user-accounts == 1.0.11 GeoNode-user-messages == 0.1.3 gisdata == 0.5.4 gsconfig == 0.6.13 gsimporter == 0.1 html5lib == 0999 httplib2 == 0.8 kombu == 3.0.7 librabbitmq == 1.0.3 lxml == 3.3.3 == mailer 0.7 nose == 1.3.1 numpy == 1.8.2 pinax-theme-bootstrap == 3.0a11 pinax-theme-bootstrap-account == 1.0b2 psycopg2 == 2.4.5 pyOpenSSL == 12:13 PyCrypto == 2.6.1 pycsw == 1.10.0 pyparsing == 2.0.1 pyproj == 1.8.9 pyserial == 2.6 python-apt == 0.9.3.5ubuntu1 python-dateutil == 1.5 python-debian-0.1.21 == nmu2ubuntu2 python-memcached == 1:53 python-mimeparse == 0.1.4 pytz == 2012c regex == 02/16/2014 requests == 2.2.1 six == 1.5.2 ssh-import-id == 3:21 transifex-client == 00:10 urllib3 == 1.7.1 virtualenv == 1.11.4 wheel == 0.24.0 wsgiref == 0.1.2 zope.interface == 4.0.5

On Wed, Aug 12, 2015 at 9:48 AM, Paolo Corti notifications@github.com wrote:

you shouldn't add the ANONYMOUS_USER_ID setting, as it is already in place: https://github.com/GeoNode/geonode/blob/master/geonode/settings.py#L397

are you running this on the server using GeoNode 2.4? Can you run a pip freeze and paste here the results?

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130288967.

Davi de O. Custódio

[capooti]

Could you try to run the migration script within my branch? (it looks like you are just using the migration directory from your existing 2.4 installation) Easiest way to do this is:

• create a virtualenv on the server and activate it
• git clone my branch and run: pip install -e geonode
• copy your local_settings.py file from your GN installation
• then try again running the script create_auth_group_and_update_res.py

Let me know p

[davicustodio]

Paolo, I'm sorry but I do not have much experience with Python development environments.

I performed all procedures:

• Create a virtualenv on the server and activate it
• Clone the branch migration_from20_to_24 and install Geonode
• Copy local_settings.py file from my installation

But when running migrate2.sh, I received errors. Log attached.

thank you.

On Thu, Aug 13, 2015 at 10:02 AM, Paolo Corti notifications@github.com wrote:

Could you try to run the migration script within my branch? (it looks like you are just using the migration directory from your existing 2.4 installation) Easiest way to do this is:

• create a virtualenv on the server and activate it
• git clone my branch and run: pip install -e geonode
• copy your local_settings.py file from your GN installation
• then try again running the script create_auth_group_and_update_res.py

Let me know p

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130662264.

Davi de O. Custódio

Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py", line 69, in if DEFAULT_CACHE_ALIAS not in settings.CACHES: File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 54, in getattr self._setup(name) File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings(settings_module) File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 132, in init % (self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (Is it on sys.path? Is there an import error in the settings file?): No module named local_settings Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/migrate_contactroles.py", line 2, in import utils File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/utils.py", line 1, in import psycopg2 ImportError: No module named psycopg2 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/migrate_tags.py", line 2, in import utils File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/utils.py", line 1, in import psycopg2 ImportError: No module named psycopg2 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/migrate_regions.py", line 2, in import utils File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/utils.py", line 1, in import psycopg2 ImportError: No module named psycopg2 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/migrate_user_permissions.py", line 2, in import utils File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/utils.py", line 1, in import psycopg2 ImportError: No module named psycopg2 Traceback (most recent call last): File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/migrate_group_permissions.py", line 2, in import utils File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24/utils.py", line 1, in import psycopg2 ImportError: No module named psycopg2 (migracao)root@ide-cnptia:/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/scripts/migrations/migrate20to24#

[capooti]

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

[davicustodio]

Ok, I managed to install psycopg and gdal. Also could run migrate.sh. But when running create_auth_group_and_update_res.py get the error:

Traceback (most recent call last): File "./create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 54, in getattr self._setup (name) File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (? Is it on sys.path Is there an import error in the settings file?): No module named local_settings

On Thu, Aug 13, 2015 at 11:01 AM, Paolo Corti notifications@github.com wrote:

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130684073.

Davi de O. Custódio

[capooti]

Davi I am in holidays and without computer. The problem is that the script cannot find your settings. Check the migration configuration script Cheers On Aug 13, 2015 5:29 PM, "Davi Custodio" notifications@github.com wrote:

Ok, I managed to install psycopg and gdal. Also could run migrate.sh. But when running create_auth_group_and_update_res.py get the error:

Traceback (most recent call last): File "./create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 54, in getattr self._setup (name) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (? Is it on sys.path Is there an import error in the settings file?): No module named local_settings

On Thu, Aug 13, 2015 at 11:01 AM, Paolo Corti notifications@github.com wrote:

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130684073.

Davi de O. Custódio

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130695803.

[davicustodio]

Paolo, reviewed the script configuration, and again ran the create_auth_group_and_update_res.py. The error changed to:

Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 15, in from guardian.shortcuts import get_objects_for_group File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/shortcuts.py", line 16, in from guardian.core import ObjectPermissionChecker File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/core.py", line 8, in from guardian.utils import get_identity File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/utils.py", line 24, in guardian.conf import settings from the guardian_settings File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/conf/settings.py", line 11, in raise ImproperlyConfigured ("In order to use django-guardian's" django.core.exceptions.ImproperlyConfigured: In order to use django-guardian's authorization ObjectPermissionBackend backend you have to configure ANONYMOUS_USER_ID at your settings module

I do not want to disrupt your vacation. Enjoy them and when you return can take up the matter.

Good holiday!

On Sat, Aug 15, 2015 at 4:56 AM, Paolo Corti notifications@github.com wrote:

Davi I am in holidays and without computer. The problem is that the script cannot find your settings. Check the migration configuration script Cheers

On Aug 13, 2015 5:29 PM, "Davi Custodio" notifications@github.com wrote:

Ok, I managed to install psycopg and gdal. Also could run migrate.sh. But when running create_auth_group_and_update_res.py get the error:

Traceback (most recent call last): File "./create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 54, in getattr self._setup (name) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (? Is it on sys.path Is there an import error in the settings file?): No module named local_settings

On Thu, Aug 13, 2015 at 11:01 AM, Paolo Corti notifications@github.com wrote:

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

— Reply to this email directly or view it on GitHub <https://github.com/GeoNode/geonode/issues/2067#issuecomment-130684073 .

Davi de O. Custódio

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-130695803.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-131313973.

Davi de O. Custódio

[davicustodio]

Paolo, you tell me if the above problem is related to the fact that all the links of the layers appear exactly as http://192.168.157.153/layers/?limit=100&offset=200 after migration?

As the restored geoserver:

1 - We performed migrate.sh 2 - I run GeoNode updatelayers (all layers have been processed) 3 - I run the migrate2.sh 4 - I rebooted the apache2 service and the tomcat7

.. But unfortunately all links to the layers appear as http://192.168.157.153/layers/?limit=100&offset=200

On Mon, Aug 17, 2015 at 9:20 AM, Davi Custodio davicustodio@gmail.com wrote:

Paolo, reviewed the script configuration, and again ran the create_auth_group_and_update_res.py. The error changed to:

Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 15, in from guardian.shortcuts import get_objects_for_group File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/shortcuts.py", line 16, in from guardian.core import ObjectPermissionChecker File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/core.py", line 8, in from guardian.utils import get_identity File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/utils.py", line 24, in guardian.conf import settings from the guardian_settings File "/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/conf/settings.py", line 11, in raise ImproperlyConfigured ("In order to use django-guardian's" django.core.exceptions.ImproperlyConfigured: In order to use django-guardian's authorization ObjectPermissionBackend backend you have to configure ANONYMOUS_USER_ID at your settings module

I do not want to disrupt your vacation. Enjoy them and when you return can take up the matter.

Good holiday!

On Sat, Aug 15, 2015 at 4:56 AM, Paolo Corti notifications@github.com wrote:

Davi I am in holidays and without computer. The problem is that the script cannot find your settings. Check the migration configuration script Cheers

On Aug 13, 2015 5:29 PM, "Davi Custodio" notifications@github.com wrote:

Ok, I managed to install psycopg and gdal. Also could run migrate.sh. But when running create_auth_group_and_update_res.py get the error:

Traceback (most recent call last): File "./create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 5, in from django.contrib.auth.models import Group File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py", line 6, in from django.middleware.csrf import rotate_token File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py", line 14, in from django.utils.cache import patch_vary_headers File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py", line 26, in from django.core.cache import get_cache File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py", line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 54, in getattr self._setup (name) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 49, in _setup self._wrapped = Settings (settings_module) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py", line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (? Is it on sys.path Is there an import error in the settings file?): No module named local_settings

On Thu, Aug 13, 2015 at 11:01 AM, Paolo Corti <notifications@github.com

wrote:

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

— Reply to this email directly or view it on GitHub < https://github.com/GeoNode/geonode/issues/2067#issuecomment-130684073>.

Davi de O. Custódio

— Reply to this email directly or view it on GitHub <https://github.com/GeoNode/geonode/issues/2067#issuecomment-130695803 .

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-131313973.

Davi de O. Custódio

Davi de O. Custódio

[capooti]

Yes You need to run successfully that script, otherwise this is expected P On Aug 21, 2015 5:28 PM, "Davi Custodio" notifications@github.com wrote:

Paolo, you tell me if the above problem is related to the fact that all the links of the layers appear exactly as http://192.168.157.153/layers/?limit=100&offset=200 after migration?

As the restored geoserver:

1 - We performed migrate.sh 2 - I run GeoNode updatelayers (all layers have been processed) 3 - I run the migrate2.sh 4 - I rebooted the apache2 service and the tomcat7

.. But unfortunately all links to the layers appear as http://192.168.157.153/layers/?limit=100&offset=200

On Mon, Aug 17, 2015 at 9:20 AM, Davi Custodio davicustodio@gmail.com wrote:

Paolo, reviewed the script configuration, and again ran the create_auth_group_and_update_res.py. The error changed to:

Traceback (most recent call last): File "create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py", line 15, in from guardian.shortcuts import get_objects_for_group File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/shortcuts.py", line 16, in from guardian.core import ObjectPermissionChecker File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/core.py", line 8, in from guardian.utils import get_identity File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/utils.py", line 24, in guardian.conf import settings from the guardian_settings File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/guardian/conf/settings.py", line 11, in raise ImproperlyConfigured ("In order to use django-guardian's" django.core.exceptions.ImproperlyConfigured: In order to use django-guardian's authorization ObjectPermissionBackend backend you have to configure ANONYMOUS_USER_ID at your settings module

I do not want to disrupt your vacation. Enjoy them and when you return can take up the matter.

Good holiday!

On Sat, Aug 15, 2015 at 4:56 AM, Paolo Corti notifications@github.com wrote:

Davi I am in holidays and without computer. The problem is that the script cannot find your settings. Check the migration configuration script Cheers

On Aug 13, 2015 5:29 PM, "Davi Custodio" notifications@github.com wrote:

Ok, I managed to install psycopg and gdal. Also could run migrate.sh. But when running create_auth_group_and_update_res.py get the error:

Traceback (most recent call last): File "./create_auth_group_and_update_res.py", line 10, in from geonode.groups.models import GroupProfile File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/geonode/geonode/groups/models.py",

line 5, in from django.contrib.auth.models import Group File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/contrib/auth/init.py",

line 6, in from django.middleware.csrf import rotate_token File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/middleware/csrf.py",

line 14, in from django.utils.cache import patch_vary_headers File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/utils/cache.py",

line 26, in from django.core.cache import get_cache File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/core/cache/init.py",

line 69, in DEFAULT_CACHE_ALIAS if not in settings.CACHES: File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py",

line 54, in getattr self._setup (name) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py",

line 49, in _setup self._wrapped = Settings (settings_module) File

"/usr/local/lib/python2.7/dist-packages/geonode/scripts/migrations/migrate20to24/migracao/local/lib/python2.7/site-packages/django/conf/init.py",

line 132, in init % (Self.SETTINGS_MODULE, e) ImportError: Could not import settings 'geonode.local_settings' (? Is it on sys.path Is there an import error in the settings file?): No module named local_settings

On Thu, Aug 13, 2015 at 11:01 AM, Paolo Corti < notifications@github.com

wrote:

You just need to install psycopg2 (and maybe gdal) in your virtualenv. Try running:

pip install psycopg2==2.5.3 pip install gdal

hope it helps

Paolo Corti Geospatial software developer web: http://www.paolocorti.net twitter: @capooti skype: capooti

— Reply to this email directly or view it on GitHub < https://github.com/GeoNode/geonode/issues/2067#issuecomment-130684073>.

Davi de O. Custódio

— Reply to this email directly or view it on GitHub < https://github.com/GeoNode/geonode/issues/2067#issuecomment-130695803 .

— Reply to this email directly or view it on GitHub <https://github.com/GeoNode/geonode/issues/2067#issuecomment-131313973 .

Davi de O. Custódio

Davi de O. Custódio

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-133442535.

[AnnalisaS]

Davi, I had the error on ANONYMOUS_USER_ID when I've setted: settings_path = /usr/local/lib/python2.7/dist-packages/geonode/local_settings.py

Then I've setted: [path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = geonode.settings

This with standard installation (but to latest version -> ppa:geonode/testing), not developer, and without template-project

Now I've other errors, maybe with migration of data-directory of Geoserver ...

[davicustodio]

Annalisa, thank you for your help. I found that my problem was in use:

settings_path = geonode.local_settings

instead :

settings_path = geonode.settings

Once adjusted it all worked and managed to make the migration happen 100%.

Thanks also to Paolo for their patience in trying to find solutions to the problems encountered.

They can count on me to help anything.

On Mon, Sep 14, 2015 at 9:36 AM, Annalisa Schiavon <notifications@github.com

wrote:

Davi, I had the error on ANONYMOUS_USER_ID when I've setted: settings_path = /usr/local/lib/python2.7/dist-packages/geonode/local_settings.py

Then I've setted: [path] geonode_path = /usr/local/lib/python2.7/dist-packages/geonode settings_path = geonode.settings

This with standard installation (but to latest version -> ppa:geonode/testing), not developer, and without template-project

Now I've other errors, maybe with migration of data-directory of Geoserver ...

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-140056207.

Davi de O. Custódio

[AnnalisaS]

Glad to know it! Simod's suggested so some posts ago!

[davicustodio]

Oh yes, really Simone had already spoken about it. sorry

But, it is still not 100%. To restart my Google Chrome, I realized that something is still wrong. Even restarting Apache, the home page of GeoNode displays an error:

invalid literal for int () with base 10: 'None'

Attached the complete error log.

I tried to interpret the log, trying to find which table I should look for a value 'None' but could not find.

I ask your help again.

Thank you.

On Tue, Sep 15, 2015 at 8:33 AM, Annalisa Schiavon <notifications@github.com

wrote:

Glad to know it! Simod's suggested so some posts ago!

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-140359569.

Davi de O. Custódio

[simod]

@davicustodio can you paste the log to a pastebin?

[AnnalisaS]

I've the similar problem: migration seems ok, but after a restart of apache something is still wrong. I've reinstalled all geonode and have to relaunch migration script.

[capooti]

I also experienced that error because for some reasons, after the migration process, I had some records in the guardian_userobjectpermission and the guardian_groupobjectpermission tables with a null value for the object_pk field. I think I got this behavior at the beginning of my development, then I did not see it anymore. I would be grate if you could investigate why this happened. After removing that records, GeoNode will works well again.

[davicustodio]

Paolo really the two tables that you mentioned were with records with the value 'None' in object_pk field. After deleting these records the error disappear. I have 1042 layers. I noticed a certain slowness in GeoNode (different from other instances that have not received a migration and have fewer layers) after migration.

On Tue, Sep 15, 2015 at 10:03 AM, Paolo Corti notifications@github.com wrote:

I also experienced that error because for some reasons, after the migration process, I had some records in the guardian_userobjectpermission and the guardian_groupobjectpermission tables with a null value for the object_pk field. I think I got this behavior at the beginning of my development, then I did not see it anymore. I would be grate if you could investigate why this happened. After removing that records, GeoNode will works well again.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-140381007.

Davi de O. Custódio

[davicustodio]

More about the slowness: by clicking on any layer is expected to more than 20 seconds to the details page appears layer. I suspected the delay of GeoServer, but investigating the "Layer Preview" within the GeoServer environment all happens without any delay. I monitored the apache.log while waiting the 20 seconds, and nothing was reported.

On Tue, Sep 15, 2015 at 10:23 AM, Davi Custodio davicustodio@gmail.com wrote:

Paolo really the two tables that you mentioned were with records with the value 'None' in object_pk field. After deleting these records the error disappear. I have 1042 layers. I noticed a certain slowness in GeoNode (different from other instances that have not received a migration and have fewer layers) after migration.

On Tue, Sep 15, 2015 at 10:03 AM, Paolo Corti notifications@github.com wrote:

I also experienced that error because for some reasons, after the migration process, I had some records in the guardian_userobjectpermission and the guardian_groupobjectpermission tables with a null value for the object_pk field. I think I got this behavior at the beginning of my development, then I did not see it anymore. I would be grate if you could investigate why this happened. After removing that records, GeoNode will works well again.

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-140381007.

Davi de O. Custódio

Davi de O. Custódio

[capooti]

Thanks @davicustodio, I will try to figure out why that records with None values are appearing.

Regarding the slowness: which GeoNode version are you using? beta 26 would help: https://github.com/GeoNode/geonode/blob/fb74a6c7df743748656864e8cea0bc3533c95fa6/package/debian/changelog#L405

@simod at some point we should consider to use direct foreign keys in django-guardian

[capooti]

sorry, that optimization is already in beta 25

[simod]

@davicustodio the slowness of the detail page is weird, it happened to someone already but because they modified the detail page. Although the problem might be on the list page due to the tough /profiles/api (owners filter). Try to deactivate it from the settings https://github.com/GeoNode/geonode/blob/master/geonode/settings.py#L843 and if that's the case try this branch https://github.com/simod/geonode/tree/fast-owners-api

[capooti]

@simod correct, I think in my case that fixed the slow layer list page response. Then the problem may be something different

[davicustodio]

Paolo and Simone,

I updated to the beta 26 and the problem with the delay is solved.

thanks

On Tue, Sep 15, 2015 at 11:10 AM, Paolo Corti notifications@github.com wrote:

@simod https://github.com/simod correct, I think in my case that fixed the slow layer list page response. Then the problem may be something different

— Reply to this email directly or view it on GitHub https://github.com/GeoNode/geonode/issues/2067#issuecomment-140405386.

Davi de O. Custódio