BULK UPLOAD: Pemba (11 datasets)

sophialinn commented 5 years ago

Data and metadata for Pemba (11 datasets) are ready for upload from here: https://drive.google.com/drive/folders/1HvWeYs04CgklQl4JqB4VTmKZOgtEubAr?usp=sharing

kalxas commented 5 years ago

Finished processing 11 layers in 610.0 seconds.

8 Created layers
0 Updated layers
0 Skipped layers
3 Failed layers
55.454545 seconds per layer

kalxas commented 5 years ago

Error log:

/data/Pemba/Pemba Data/Pem_Education.shp                                                    
================                                                                             
Traceback (most recent call last):                                                           
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 830, in upload 
    charset=charset)                                                                      
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 599, in file_up
load                                                                                         
    defaults=defaults                                               
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in mana
ger_method                                                                                   
    return getattr(self.get_queryset(), name)(*args, **kwargs)      
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get_or_create                                                                                   
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_or_create                                                                                  
    return self._create_object_from_params(lookup, params)   
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _create_object_from_params
    obj = self.create(**params)                              
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in cre
ate                                                           
    return super(MultilingualQuerySet, self).create(**kwargs)                                
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in creat
e                                                                                            
    obj.save(force_insert=True, using=self.db)                                               
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 83, in save      
    return super(PolymorphicModel, self).save(*args, **kwargs)                               
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save  
    force_update=force_update, update_fields=update_fields)                             
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_b
ase                                                                                          
    update_fields=update_fields, raw=raw, using=using,                                       
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in s
end                                                                                       
    for receiver in self._live_receivers(sender)                                             
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper   
    return func(*args, **kwargs)                                                             
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 87, in geo
server_post_save                                                                             
    producer.geoserver_upload_layer(payload)                                                 
  File "</usr/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-3>", line 2, in g
eoserver_upload_layer                                                                        
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/producer.py", line 71, in sy
nc_if_local_memory                                                                           
    worker.run(timeout=broker_socket_timeout)                                               
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):                                             
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_event
s
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 963, i$
 drain_events
    get(self._deliver, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, $
n _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, i$
 drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, i$
 _poll
    return cycle.get(callback)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, i$
 _get_and_deliver
    callback(message, queue)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, i$
 _deliver
    callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, i$
 _callback
    return callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_ca$
lback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/consumer.py", line 104, in $
n_geoserver_messages
    geoserver_post_save_local(layer)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 143, in g$
oserver_post_save_local
    charset=instance.charset)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/upload.py", line 140, in ge$
server_upload
    workspace=workspace)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/helpers.py", line 1352, in $
create_db_featurestore
    raise GeoNodeException(msg)
GeoNodeException: An exception occurred loading data to PostGIS- Failed to add data to store
geonode_data : 500, java.io.IOException: transform error

/data/Pemba/Pemba Data/Pem_Road_Network.shp
================
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 830, in upload
    charset=charset)
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 599, in file_up
load
    defaults=defaults
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in mana
ger_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get
_or_create
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_o
r_create
    return self._create_object_from_params(lookup, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _crea
te_object_from_params
    obj = self.create(**params)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in cre
ate
    return super(MultilingualQuerySet, self).create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in creat
e
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 83, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_b
ase
    update_fields=update_fields, raw=raw, using=using,
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in s
end
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 87, in geo
server_post_save
    producer.geoserver_upload_layer(payload)
  File "</usr/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-3>", line 2, in g
eoserver_upload_layer
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/producer.py", line 71, in sy
nc_if_local_memory
    worker.run(timeout=broker_socket_timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_event
s
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 963, in
 drain_events
    get(self._deliver, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, i
n _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in
 drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in
 _poll
    return cycle.get(callback)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in
 _get_and_deliver
    callback(message, queue)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, in
 _deliver
    callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in
 _callback
    return callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_cal
lback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/consumer.py", line 104, in o
n_geoserver_messages
    geoserver_post_save_local(layer)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 143, in ge
oserver_post_save_local
    charset=instance.charset)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/upload.py", line 140, in geo
server_upload
    workspace=workspace)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/helpers.py", line 1352, in _
create_db_featurestore
    raise GeoNodeException(msg)
GeoNodeException: An exception occurred loading data to PostGIS- Failed to add data to store
geonode_data : 500, java.io.IOException: transform error

/data/Pemba/Pemba Data/Pem_Health_and_Medical.shp                                   [55/1938]
================                                                                             
Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 830, in upload
    charset=charset)                                                                         
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 599, in file_up
load                                   
    defaults=defaults                                                                     
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in mana
ger_method                                                                                   
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get
_or_create                                                                                   
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_o
r_create                                                                                     
    return self._create_object_from_params(lookup, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _crea
te_object_from_params                                                                     
    obj = self.create(**params)                  
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in cre
ate              
    return super(MultilingualQuerySet, self).create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in creat
e        
    obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 83, in save      
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)                                  
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_b
ase                                                                 
    update_fields=update_fields, raw=raw, using=using,                                  
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in s
end                                                                                          
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)                                                          
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 87, in geo
server_post_save                                                                             
    producer.geoserver_upload_layer(payload)
  File "</usr/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-3>", line 2, in g
eoserver_upload_layer                                                                        
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/producer.py", line 71, in sy
nc_if_local_memory      
    worker.run(timeout=broker_socket_timeout)                                                
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume       
    conn.drain_events(timeout=safety_interval)          
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_event
s
    get(self._deliver, timeout=timeout)                                                     
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get   
    return self.fun(resource, callback, **kwargs)                                            
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, i
n _drain_channel                                                                          
    return channel.drain_events(callback=callback, timeout=timeout)                          
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in
 drain_events                                                 
    return self._poll(self.cycle, callback, timeout=timeout)                                 
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in
 _poll                                                              
    return cycle.get(callback)                                                               
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get   
    return self.fun(resource, callback, **kwargs)         
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in
 _get_and_deliver                                                                         
    callback(message, queue)                     
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, in
 _deliver        
    callback(message)                                        
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in
 _callback
    return callback(message)                  
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_cal
lback                                                         
    return on_m(message) if on_m else self.receive(decoded, message)                       
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive     
    [callback(body, message) for callback in callbacks]                                      
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/consumer.py", line 104, in o
n_geoserver_messages                                                                    
    geoserver_post_save_local(layer)                                                         
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper   
    return func(*args, **kwargs)                
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 143, in ge
oserver_post_save_local                                                                   
    charset=instance.charset)                                                                
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/upload.py", line 140, in geo
server_upload                               
    workspace=workspace)                                                                     
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/helpers.py", line 1352, in _
create_db_featurestore                                                                       
    raise GeoNodeException(msg)
GeoNodeException: An exception occurred loading data to PostGIS- Failed to add data to store 
geonode_data : 500, java.io.IOException: transform error

kalxas commented 5 years ago

Pemba: Communication : Metadata updated!
pemba: education : Failed to update
Exception: Layer matching query does not exist.
Pemba: Emergency Shelter and Services : Metadata updated!
Pemba: Energy : Metadata updated!
pemba: health and medical : Failed to update
Exception: Layer matching query does not exist.
Pemba: Open Space : Metadata updated!
Pemba: Religious : Metadata updated!
pemba: road network : Failed to update
Exception: Layer matching query does not exist.
Pemba: Solid Waste : Metadata updated!
Pemba: Water and Sanitation : Metadata updated!
Pemba: City Boundary : Metadata updated!

sophialinn commented 5 years ago

I'm checking on the failed layers...

kalxas commented 5 years ago

thanks!

kalxas commented 5 years ago

Some comments on the failing layers:

Pem_Education.shp: Has some geometries outside the Pemba bbox
Pem_Road_Network.shp: Although it says road network it is a point layer. Also some of the field names do not make sense
Pem_Health_and_Medical.shp: Has some geometries outside the Pemba bbox

state-hiu / 2C-GeoNode

BULK UPLOAD: Pemba (11 datasets) #200