Problem uploading (layers via UI)

[stufraser1]

Uploading the following layers (manually, at /layers/upload) worked on old Labs geonode, but repeatedly failed on new geonode.

Flaggin this as a potential issue on new geonode, for testing. Both, 122MB .tif files

http://45.55.174.20/layers/hazard%3Aadm2_uu_raster_v2 http://45.55.174.20/layers/hazard%3Aadm2_fu_raster_v2_1

[stufraser1]

These files uploaded with no problem on old Labs geonode - please confirm that manual layer upload works ok for new geonode when upgrade complete

[fvanderbiest]

Probably a timeout issue. I'll take a look at it.

[fvanderbiest]

We might miss the keepalive_timeout parameter, as seen in https://github.com/GeoNode/nginx-docker/blob/master/nginx.conf#L26 (at least something equivalent for HAProxy, that we use instead of nginx)

[fvanderbiest]

Should be fixed with https://github.com/camptocamp/terraform-geonode/pull/5 Can you try again on production server @ErikKBethke or @stufraser1 ?

[ErikKBethke]

Confirming that uploads/replacing layers still results in stalling/timeout issues.

[stufraser1]

Python sftp update is working. Sometimes the UI upload works, other times it times out.

[fjacon]

@stufraser1 geonode 2.8 is deployed on integration server https://int.geonode-gfdrrlab.org/ can you test in this server ?

[stufraser1]

I just tried to upload a 66MB tif file. The uploaded started and proceeded through to the end of the progress bar on GeoNode, then gave a message: 'No store found named: ls_nasa_modnowcast_avg'. The tif did not upload.

[pvalsecc]

Notes for myself...

I'm testing with a renamed version of https://www.geonode-gfdrrlab.org/uploaded/layers/maxdepthseu_500y.tif (139MB) and I get:

Traceback (most recent call last):
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/views.py", line 211, in layer_upload
    metadata_upload_form=form.cleaned_data["metadata_upload_form"])
  File "/usr/local/lib/python2.7/site-packages/geonode/layers/utils.py", line 594, in file_upload
    defaults=defaults  File "/usr/local/lib/python2.7/site-packages/django/db/models/manager.py", line 127, in manager_method
    return getattr(self.get_queryset(), name)(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 413, in get_or_create
    return super(MultilingualQuerySet, self).get_or_create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 407, in get_or_create
    return self._create_object_from_params(lookup, params)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 439, in
 _create_object_from_params
    obj = self.create(**params)
  File "/usr/local/lib/python2.7/site-packages/modeltranslation/manager.py", line 405, in create
    return super(MultilingualQuerySet, self).create(**kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/query.py", line 348, in create 
   obj.save(force_insert=True, using=self.db)
  File "/usr/local/lib/python2.7/site-packages/polymorphic/models.py", line 82, in save
    return super(PolymorphicModel, self).save(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 734, in save
    force_update=force_update, update_fields=update_fields)
  File "/usr/local/lib/python2.7/site-packages/django/db/models/base.py", line 771, in save_base
    update_fields=update_fields, raw=raw, using=using)
  File "/usr/local/lib/python2.7/site-packages/django/dispatch/dispatcher.py", line 189, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/usr/local/lib/python2.7/site-packages/geonode/decorators.py", line 44, in wrapper
    return func(*args, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 92, in geoserver_post_save
    producer.geoserver_upload_layer(payload)
  File "<decorator-gen-3>", line 2, in geoserver_upload_layer
  File "/usr/local/lib/python2.7/site-packages/geonode/messaging/producer.py", line 71, in sync_if_local_memory
    worker.run(timeout=broker_socket_timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 170, in run
    for _ in self.consume(limit=None, **kwargs):
  File "/usr/local/lib/python2.7/site-packages/kombu/mixins.py", line 192, in consume
    conn.drain_events(timeout=safety_interval)
  File "/usr/local/lib/python2.7/site-packages/kombu/connection.py", line 301, in drain_events
    return self.transport.drain_events(self.connection, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 961, in drain_events
    get(self._deliver, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 999, in _drain_channel
    return channel.drain_events(callback=callback, timeout=timeout) 
 File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 745, in drain_events
    return self._poll(self.cycle, callback, timeout=timeout)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 402, in _poll
    return cycle.get(callback)  File "/usr/local/lib/python2.7/site-packages/kombu/utils/scheduling.py", line 56, in get
    return self.fun(resource, callback, **kwargs)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 406, in _get_and_deliver
    callback(message, queue)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 981, in _deliver
    callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/transport/virtual/base.py", line 633, in _callback
    return callback(message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 624, in _receive_callback
    return on_m(message) if on_m else self.receive(decoded, message)
  File "/usr/local/lib/python2.7/site-packages/kombu/messaging.py", line 590, in receive
    [callback(body, message) for callback in callbacks] 
 File "/usr/local/lib/python2.7/site-packages/geonode/messaging/consumer.py", line 104, in on_geoserver_messages
    geoserver_post_save_local(layer)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/signals.py", line 135, in geoserver_post_save_local
    charset=instance.charset)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/upload.py", line 141, in geoserver_upload
    workspace=workspace)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/helpers.py", line 1126, in _create_coveragestore
    store = get_store(cat, name, workspace=workspace)
  File "/usr/local/lib/python2.7/site-packages/geonode/geoserver/helpers.py", line 1211, in get_store
    raise FailedRequestError("No store found named: " + name)
FailedRequestError: No store found named: pvibig

[pvalsecc]

It should be fixed, now. I've tested on int and deployed the fix on prod as well.