search

state-hiu / 2C-GeoNode

3 stars 9 forks source link

BULK UPLOAD: Santa Fe (37 datasets) #206

Closed sophialinn closed 5 years ago

sophialinn commented 5 years ago

Data and metadata for Santa Fe are ready for upload from here:

https://drive.google.com/drive/folders/1uvAYldAa0VCurFwD1-_2prRXbJ-zvT8q?usp=sharing

NOTE: They seem to be continuing to upload data and metadata! To avoid confusion, data to be uploaded (for this issue) is located in the Santa Fe Data (github issue #206) READY FOR UPLOAD folder, and metadata is in the Santa Fe Metadata Spreadsheet (github issue #206) READY FOR UPLOAD.

kalxas commented 5 years ago 
Finished processing 36 layers in 7319.0 seconds.

34 Created layers
0 Updated layers
0 Skipped layers
2 Failed layers
203.305556 seconds per layer
kalxas commented 5 years ago 
Detailed report of failures:                                                       [115/1847]

/data/Santa_Fe/SNF_Community_centers.shp                                                     
================                                                                             
Traceback (most recent call last):                                                           
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 830, in upload 
    charset=charset)                                                                         
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 599, in file_u$
load                                                                                      
    defaults=defaults                                                                        
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in man$
ger_method                                                       
    return getattr(self.get_queryset(), name)(*args, **kwargs)                               
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in ge$
_or_create                                                          
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)                         
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_$
r_create                                                             
    return self._create_object_from_params(lookup, params)                                   
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _cre$
te_object_from_params                                                 
    obj = self.create(**params)                                                              
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in cr$
ate                                                                  
    return super(MultilingualQuerySet, self).create(**kwargs)                                
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in crea$
e                                                                   
    obj.save(force_insert=True, using=self.db)                                            
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 83, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)                               
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save
    force_update=force_update, update_fields=update_fields)                                  
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 833, in save_$
ase                                                   
    update_fields=update_fields,                                                             
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in $
end                                             
    for receiver in self._live_receivers(sender)                                          
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/models.py", line 540, in pre_s$
ve_layer                                                                                     
    base_file, info = instance.get_base_file()                                
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/models.py", line 251, in get_b$
se_file                                                                                      
    self)                                                                                 
  File "/usr/local/lib/python2.7/site-packages/geonode/utils.py", line 980, in check_shp_col$
mnnames           
    return fixup_shp_columnnames(inShapefile, layer.charset)            
  File "/usr/local/lib/python2.7/site-packages/geonode/utils.py", line 1064, in fixup_shp_co$
umnnames                                        
    "Could not decode SHAPEFILE attributes by using the specified charset '{}'.".format(chars
et))
GeoNodeException: Could not decode SHAPEFILE attributes by using the specified charset 'UTF-8
'.
/data/Santa_Fe/SNF_MDT.tif                                                                   
================                                                                             
Traceback (most recent call last):                                                           
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 830, in upload 
    charset=charset)                                                                         
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 599, in file_up
load                                                                                         
    defaults=defaults                                                                     
  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 85, in mana
ger_method                                                                                   
    return getattr(self.get_queryset(), name)(*args, **kwargs)   
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get
_or_create                                                                                   
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 466, in get_o
r_create                                                                                     
    return self._create_object_from_params(lookup, params)           
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 500, in _crea
te_object_from_params                                                                        
    obj = self.create(**params)                                       
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in cre
ate                                                                                          
    return super(MultilingualQuerySet, self).create(**kwargs)        
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 394, in creat
e                                                                                            
    obj.save(force_insert=True, using=self.db)                      
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 83, in save   
    return super(PolymorphicModel, self).save(*args, **kwargs)                         
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 808, in save  
    force_update=force_update, update_fields=update_fields)                                
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 848, in save_b
ase                                                                                          
    update_fields=update_fields, raw=raw, using=using,
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 193, in s
end                                                                                          
    for receiver in self._live_receivers(sender)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)                                                             
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 87, in geo
server_post_save                                                              
    producer.geoserver_upload_layer(payload)                                                 
  File "</usr/local/lib/python2.7/site-packages/decorator.pyc:decorator-gen-3>", line 2, in g
eoserver_upload_layer                                                                     
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/producer.py", line 71, in sy
nc_if_local_memory
    worker.run(timeout=broker_socket_timeout)                           
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run            
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume        
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 315, in drain_event
s 
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 963, in
 drain_events                                                                                
    get(self._deliver, timeout=timeout)                                                      
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get   
    return self.fun(resource, callback, **kwargs)                                            
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 1001, i
n _drain_channel                                                                             
    return channel.drain_events(callback=callback, timeout=timeout)                          
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in
 drain_events                                                                                
    return self._poll(self.cycle, callback, timeout=timeout)                                 
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in
 _poll                                                                                       
    return cycle.get(callback)                                                               
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)                                            
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in
 _get_and_deliver                                                    
    callback(message, queue)                                                                 
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 983, in
 _deliver                                                             
    callback(message)                                                                        
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in
 _callback                                                           
    return callback(message)                                                                 
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_cal
lback                                                               
    return on_m(message) if on_m else self.receive(decoded, message)                      
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks]                                      
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/consumer.py", line 104, in o
n_geoserver_messages                                                                         
    geoserver_post_save_local(layer)                                                         
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 53, in wrapper
    return func(*args, **kwargs)                                                             
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 321, in ge
oserver_post_save_local                         
    resources.update(**to_update)                                                         
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 388, in upd
ate                                                                                          
    return super(MultilingualQuerySet, self).update(**kwargs)                 
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 650, in updat
e                                                                                            
    rows = query.get_compiler(self.db).execute_sql(CURSOR)                                
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 1204, 
in execute_sql    
    cursor = super(SQLUpdateCompiler, self).execute_sql(result_type)    
  File "/usr/local/lib/python2.7/site-packages/django/db/models/sql/compiler.py", line 899, i
n execute_sql                                   
    raise original_exception                                                                 
DataError: value too long for type character varying(30)
kalxas commented 5 years ago 
Santa Fe: Aiports Nearby : Metadata updated!
Santa Fe: Blocks Urban : Metadata updated!
Santa Fe: Political Boundaries : Metadata updated!
Santa Fe: Bus Line Tour : Metadata updated!
Santa Fe: Census Demography : Metadata updated!
Santa Fe: Census Division  : Metadata updated!
Santa Fe: Unsatisfied Basic Needs : Metadata updated!
santa fe: community centers : Failed to update
Exception: Layer matching query does not exist.
Santa Fe: Flood Defenses : Metadata updated!
Santa Fe: Planialimetric Fixed Points : Metadata updated!
Santa Fe: Green Areas : Metadata updated!
Santa Fe: Industrial Parks : Metadata updated!
Santa Fe: Initial Level Schools : Metadata updated!
Santa Fe: Inland Waters  : Metadata updated!
Santa Fe: Level Curves in the Field : Metadata updated!
santa fe: model digital terrain : Failed to update
Exception: Layer matching query does not exist.
Santa Fe: Meeting Points for Emergency : Metadata updated!
Santa Fe: Metrobus : Metadata updated!
Santa Fe: Neighborhoods : Metadata updated!
sante fe: ovitraps : Failed to update
Exception: Layer matching query does not exist.
Santa Fe: Police Stations : Metadata updated!
Santa Fe: Centers of Private Medicine : Metadata updated!
Santa Fe: Pumping Operational Points : Metadata updated!
Santa Fe: Railroad : Metadata updated!
Santa Fe: Network Routes : Metadata updated!
Santa Fe: Sanitary Landfills : Metadata updated!
Santa Fe:  Educational Centers (Schools) : Metadata updated!
Santa Fe: Stations Railway : Metadata updated!
Santa Fe: Streets : Metadata updated!
Santa Fe: Sub Basins  : Metadata updated!
Santa Fe: Health Surveys : Metadata updated!
Santa Fe: Survey in Vuelta del Paraguayo : Metadata updated!
Santa Fe: Survey in Jesuitas : Metadata updated!
Santa Fe: Types of Road : Metadata updated!
Sante Fe: Universities : Metadata updated!
Santa Fe: Wastewater  : Metadata updated!
Santa Fe: Watersheds  : Metadata updated!
 : Failed to update
Exception: Layer matching query does not exist.
santa fe: water : Failed to update
Exception: Layer matching query does not exist.
santa fe: dengue cases : Failed to update
Exception: Layer matching query does not exist.
santa fe: sinister roads : Failed to update
Exception: Layer matching query does not exist.
sophialinn commented 5 years ago

There may be some projection issues with a few of these layers. We'll need to download and check. The ones whose thumbnails look strange are the questionable ones (e.g. neighborhoods, bus line tours, watersheds.)

sophialinn commented 5 years ago

The following datasets do not have a defined projection. (Maybe there are others but these are the ones I found). I believe it is OK to define them as WGS84 (EPSG: 4326). Currently, their thumbnails do not display correctly, nor do they appear on the map:

snf_bus_line snf_blocks snf_neighborhood snf_sub_basins snf_wastewater snf_watersheds

kalxas commented 5 years ago

image

The SRS seems to be the problem in this case, GeoServer cannot preview those layers:

https://secondarycities.geonode.state.gov/geoserver/geonode/wms?service=WMS&version=1.1.0&request=GetMap&layers=geonode%3Asnf_bus_line&bbox=5428400.5%2C6492774.0%2C5446334.0%2C6509302.5&width=768&height=707&srs=EPSG%3A404000&format=application/openlayers

sophialinn commented 5 years ago

Instead of 'waiting for the partner', I defined the projection for these 6 datasets to match the other data from Santa Fe. (EPSG:5347) The data, with the new .prj files, are located here and should now REPLACE the data with the same names in the GeoNode:

https://drive.google.com/open?id=1vR9IdrfRimm4CJ7Z1B1DjsgL7ZDr-qbb

kalxas commented 5 years ago

Given that layers from #213 also do not show on map, I think we need to reproject all Santa Fe layers to WGS84 (as @d3netxer is suggesting in #215 )

sophialinn commented 5 years ago

OK, those six problematic layers have ALSO been reprojected to WGS84. They can be found here:

https://drive.google.com/open?id=1vR9IdrfRimm4CJ7Z1B1DjsgL7ZDr-qbb

kalxas commented 5 years ago 
Santa Fe: Blocks Urban : Metadata updated!
Santa Fe: Political Boundaries : Metadata updated!
Santa Fe: Bus Line Tour : Metadata updated!
Santa Fe: Neighborhoods : Metadata updated!
Santa Fe: Sub Basins  : Metadata updated!
Santa Fe: Wastewater  : Metadata updated!
Santa Fe: Watersheds  : Metadata updated!
kalxas commented 5 years ago

2 layers with previous errors now fixed:

Finished processing 2 layers in 88.0 seconds.

2 Created layers
0 Updated layers
0 Skipped layers
0 Failed layers
44.000000 seconds per layer
kalxas commented 5 years ago 
Santa Fe: Community Centers : Metadata updated!
Santa Fe: Model Digital Terrain : Metadata updated!