can't connect to remote service

SterckxArnaud commented 3 years ago

Why can I not connect to this service? https://www.gapmaps.org/geoserver/wms? I get this error message: The service is working properly, see for instance https://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

NyakudyaA commented 3 years ago

@ArnaudIGRAC The problem is with the upstream resource. I tried to connect in QGIS and it also times out. Maybe we need to add a better error message asking users to inspect the upstream service

SterckxArnaud commented 3 years ago

thanks @NyakudyaA. But if I can access the layers in a web browser (https://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS), there must be a way to also get them in GeoNode (on in QGIS)?

NyakudyaA commented 3 years ago

@ArnaudIGRAC GeoNode basically looks at all layers available through a get capabilities request i.e https://www.gapmaps.org/geoserver/wms?REQUEST=GetCapabilities So this is what is taking time so maybe @lucernae needs to increase the timeout time in nginx so that the request goes on further than the allocated time

meomancer commented 3 years ago

hi @ArnaudIGRAC as per @NyakudyaA said that the request to get the wms has limit time. Currently the limit is 1 minute, need to increase the limit of nginx to more than that.

But i checked on my local which does not using nginx (so it is unlimited) It is taking to long for the connection from the gapmaps and it returns "Request Time Out" from gapmaps too

So there is no way to fix it from our side (we increase the time, it is Request Time Out from gapmaps) Selection_340

SterckxArnaud commented 3 years ago

Thank you for your feedback. Is there a possibility to fetch one layer instead of a broad get capabilities request? This if for instance what I did for some FAO layers http://data.fao.org/maps/wms?styles=geonetwork_DSMW_14116_style http://data.fao.org/maps/wms?styles=geonetwork_dams_africa_38077_style http://data.fao.org/maps/wms?styles=aquamaps_rivers_africa_37333_style

meomancer commented 3 years ago

hi @seabilwe could you help me registers those layer please? thank you

SterckxArnaud commented 3 years ago

Dear colleagues, any update on this?

NyakudyaA commented 3 years ago

@ArnaudIGRAC Which layers do you need for us to register.

SterckxArnaud commented 3 years ago

There are two layers (and their color scales):

http://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

http://www.gapmaps.org/geoserver/wms?REQUEST=GetLegendGraphic&LAYER=gap:arsenic2020&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

http://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:fluoride&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

http://www.gapmaps.org/geoserver/wms?REQUEST=GetLegendGraphic&LAYER=gap:fluoride&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

NyakudyaA commented 3 years ago

@meomancer This service is extremely slow to do a getcapabilties request so even if you request a single layer is still times out because the server still does a getcapabiltiies for the whole server. So upstream improvements could help this issue

When a request is done and in the URL there is Layer defined do a getcapabitlies for a single layer than the whole server i.e http://osgeo-org.1560.x6.nabble.com/GetCapabilities-Filter-by-Layer-td5082208.html
Increase the timeout of registering remote services

meomancer commented 3 years ago

hi @NyakudyaA thank you

hi @ArnaudIGRAC yes, as per i said in here https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-802498711

we need to ask the service to increase their time out and also the comments from admire

SterckxArnaud commented 3 years ago

Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html

SterckxArnaud commented 2 years ago

Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html

Dear colleagues, any answer to my question above?

NyakudyaA commented 2 years ago

Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html

Dear colleagues, any answer to my question above?

@ArnaudIGRAC I think it would be the same thing, because the GeoNode front end is just abstracting the loading and publishing of layers into GeoServer/PostGIS database.

I will try them out again both from the GeoNode frontend and GeoServer. Otherwise, we could try to do this from the backend if possible, Maybe that would work

SterckxArnaud commented 2 years ago

thank you Admire, let me know if it works. Otherwise we could also contact the colleagues managing this server to seek a solution

clau1313 commented 2 years ago

Dear all, is there any update on this?

clau1313 commented 2 years ago

hi @meomancer, is there any update on this? shall we do something from this side?

meomancer commented 2 years ago

hi @clau1313 i'm not sure how to fix it because the problem is from their server that we can't get information of the layer and register it on geonode https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-826563463

so i think we can ask solution from their side

clau1313 commented 2 years ago

hi @meomancer, thanks, I wrote yesterday to the people in charge of the GAP portal to see if they can solve the problem. I will let you know when they answer.

clau1313 commented 2 years ago

hi @meomancer, I received an answer from the GAP portal. Unfortunately they cannot identify one specific problem, and they asked me if it would be okay if they communicate directly with you. Can I do that? If so then I will give them your email address.

Please let me know. Thanks!

meomancer commented 2 years ago

hi @clau1313 i'm not sure the solution from their site maybe i need @NyakudyaA to communicate with them how to overcome our issue from their side

could you help me @NyakudyaA ?

NyakudyaA commented 2 years ago

hi @clau1313 i'm not sure the solution from their site maybe i need @NyakudyaA to communicate with them how to overcome our issue from their side

could you help me @NyakudyaA ?

@clau1313 If they are using GeoServer they can increase the timeout for their service as depicted here https://geoserver.geo-solutions.it/multidim/enterprise/resource_limits.html

If their service is mapserver,qgis server etc hey should also see how to optimise the timeout for wms services

clau1313 commented 2 years ago

hi @NyakudyaA, thanks, can I give the developers at GAP Maps your email address in case they have more questions?

NyakudyaA commented 2 years ago

hi @NyakudyaA, thanks, can I give the developers at GAP Maps your email address in case they have more questions?

Yes @clau1313

clau1313 commented 2 years ago

@NyakudyaA what is your email address? I am not sure if I have it...

NyakudyaA commented 2 years ago

@NyakudyaA what is your email address? I am not sure if I have it...

Sorry I thought you had it, admire@kartoza.com

NyakudyaA commented 2 years ago

@meomancer Can we try to increase the Nginx timeout to the maximum we can? I have been given new links but they all fail to retrieve the layer. An alternative solution might be to actually ingest the layers directly into the database but we would need to update all the links for the resources

NyakudyaA commented 2 years ago

New links provided in email are Global arsenic (2020):

http://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

http://www.gapmaps.org/geoserver/wms?REQUEST=GetLegendGraphic&LAYER=gap:arsenic2020&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

Global fluoride (2008):

https://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:fluoride&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

http://www.gapmaps.org/geoserver/wms?REQUEST=GetLegendGraphic&LAYER=gap:fluoride&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS

meomancer commented 2 years ago

hi @NyakudyaA it turns out i'm not sure how to increase it on rancher2

hi @lucernae could you help us increase timeout for uwsgi and nginx? i'm not sure where nginx.conf is (i changed on container but it is readonly)

lucernae commented 2 years ago

@meomancer for future reference, you can easily check config location in Rancher. Select the container you want to see:

In the volumes sections, find the file you want to override, then click the link to the ConfigMap definition:

In this case here, we change it in geonode.conf:

I increase it to 72000s (20 hours) for now.

lucernae commented 2 years ago

~~If I look at the log, it seems the one doing the timeout is uwsgi, so we need to also increase uwsgi timeout~~. Sorry I was not being clear. I mean: the timeout inside the uwsgi process. It could be python job, requests module, etc. I'm not sure. But probably it's in the GeoNode module? Currently, Increasing the uwsgi timeout parameter in nginx only results with this:

Current addition of timeout in uwsgi parameter in inginx:

Harakiri parameter (uwsgi.conf):

The timeout of the process is flat out 1 mins:

If I read the uwsgi module in nginx, 75 s is the hard limit. They say it can't actually be set greater than that. So I think the processing needs to be moved as a background job if the remote GetCapabilities requests takes too much time to process.

meomancer commented 2 years ago

hi @lucernae thank you very much for increasing timeout

@NyakudyaA It is still timeout from the remote service

Traceback (most recent call last):
  File "/spcgeonode/geonode/services/serviceprocessors/handler.py", line 67, in get_service_handler
    service = handler(base_url)
  File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 127, in __init__
    INDEXED if self._offers_geonode_projection() else CASCADED)
  File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 459, in _offers_geonode_projection
    if len(list(self.get_resources())) > 0:
  File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 208, in get_resources
    contents_gen = self.parsed_service.contents.values()
  File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 162, in parsed_service
    timeout=ogc_server_settings.get('TIMEOUT', 60))
  File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 108, in WebMapService
    timeout=timeout, headers=headers))
  File "/usr/local/lib/python3.7/site-packages/owslib/map/wms130.py", line 75, in __init__
    self._capabilities = reader.read(self.url, timeout=self.timeout)
  File "/usr/local/lib/python3.7/site-packages/owslib/map/common.py", line 66, in read
    timeout=timeout, headers=self.headers, auth=self.auth)
  File "/usr/local/lib/python3.7/site-packages/owslib/util.py", line 208, in openURL
    req = requests.request(method.upper(), url_base, headers=headers, **rkwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
    return session.request(method=method, url=url, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
    resp = self.send(prep, **send_kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
    r = adapter.send(request, **kwargs)
  File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
    raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.gapmaps.org', port=443): Read timed out. (read timeout=60)
[uwsgi-http key: ggis.un-igrac.org client_addr: 188.34.198.238 client_port: 37026] hr_read(): Connection reset by peer [plugins/http/http.c line 917]

lucernae commented 2 years ago

@meomancer sorry I'm still editing my comments and only finished after you posted, so I summarize again in this comment.

In summary:

From nginx and uwsgi conf side, we already tried to increase the timeout, but there's a hard limit of 75s of uwsgi_connect_timout setting. I guess, this makes sense because nginx is a UI thread for the web, it needs to finish quickly to avoid becoming zombie thread or blocking other process.
the python requests shared by @meomancer above showed that the current python modules were set using 60s read timeout. But it's very deep in the call trace (called by owslib module). I don't think we should change this. But rather change the GeoNode code to propagate the timeout (possibly in ogc_server_settings TIMEOUT parameter). To do this, @meomancer need to create a glue code in settings.py of GeoNode (or core.settings module), that accept the OGC TIMEOUT parameter from the environment variable or local_settings.py and pass it to this ogc_server_settings object.

meomancer commented 2 years ago

i think the solution is saving it manually on database will ask time for working on it

lucernae commented 2 years ago

I just did a small patch in the k8s deployment to test this. If I increase the OGC Timeout parameter, the register process does returns after around 10 mins.

I still don't think it is a good idea though, because the uwsgi thread is blocked during that time and we don't know if there will be more than one process doing this. Because if it is, then the web app is going to be blocked or less responsive. Management command or python job is a more better idea than this.

Note: Also since the edit was an ephemeral patch, it will be gone in the next GGIS docker image upgrade.

NyakudyaA commented 2 years ago

I just did a small patch in the k8s deployment to test this. If I increase the OGC Timeout parameter, the register process does returns after around 10 mins.

I still don't think it is a good idea though, because the uwsgi thread is blocked during that time and we don't know if there will be more than one process doing this. Because if it is, then the web app is going to be blocked or less responsive. Management command or python job is a more better idea than this.

Note: Also since the edit was an ephemeral patch, it will be gone in the next GGIS docker image upgrade.

Can you register remote services through management commands ? If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend

lucernae commented 2 years ago

Can you register remote services through management commands ?

I don't actually know. From what I've understand, the bottleneck is this initial GetCapabilities request where you are not yet registering any service, but only listing all services. Doing it with the management command will still result in 10 mins waiting in the terminal. Unless you page the request to the remote server. For example if you want only the first 10 results to be shown initially. (the bottleneck is that the remote services takes too long to return all layers).

If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend

Not sure I understand what you mean by this. Can you elaborate? My point is that, after testing the UI, there are two phase: "Register service" and "Import Resources". When user clicks Create, this will "Register" the WMS endpoint. No layer is saved yet, and it seems the previous GetCapabilities requests are also not indexed. I tried traversing the layers list pagination in the above page and it seems to be requesting GetCapabilities again to the server, so I had to wait another 10 mins. If we make it a management command or python Job, we could process it in the background and store the cache somewhere in the DB. So that when the UI access it, the lists comes from the cache, at lest for the initial pagination (not the filters).

NyakudyaA commented 2 years ago

Can you register remote services through management commands ?

I don't actually know. From what I've understand, the bottleneck is this initial GetCapabilities request where you are not yet registering any service, but only listing all services. Doing it with the management command will still result in 10 mins waiting in the terminal. Unless you page the request to the remote server. For example if you want only the first 10 results to be shown initially. (the bottleneck is that the remote services takes too long to return all layers).

I am sure there must be something happening within the context of GeoNode because a straight getcapabilities report doesn't go over 5minutes in my browser and even when I time it in the shell

If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend

Not sure I understand what you mean by this. Can you elaborate? My point is that, after testing the UI, there are two phase: "Register service" and "Import Resources". When user clicks Create, this will "Register" the WMS endpoint. No layer is saved yet, and it seems the previous GetCapabilities requests are also not indexed. I tried traversing the layers list pagination in the above page and it seems to be requesting GetCapabilities again to the server, so I had to wait another 10 mins. If we make it a management command or python Job, we could process it in the background and store the cache somewhere in the DB. So that when the UI access it, the lists comes from the cache, at lest for the initial pagination (not the filters).

I think we are talking the same thing. What I also mean is that when you do register the service it will do a getcapabilities and store those results in the DB in a particular table and when you select a layer to add to your geonode we also add that layer to the django base tables. So what I was suggesting is just if it doesn't take time to understand what SQL commands are fired to insert this records in the DB, we just replicate it outself otherwise if there is a better solution then the better

meomancer commented 2 years ago

Thank you very much @lucernae now it works

CC @NyakudyaA @ArnaudIGRAC

Here are the layers https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:fluoride https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:arsenic2020

Please check it

lucernae commented 2 years ago

I am sure there must be something happening within the context of GeoNode because a straight getcapabilities report doesn't go over 5minutes in my browser and even when I time it in the shell

@NyakudyaA I also did a straight get capabilities and in my case it took only slightly less than 10 mins. Perhaps my region or ISP affect this as well.

So, @meomancer what should I do with the temporary patch? Should I make it permanent?

clau1313 commented 2 years ago

Hi everyone, I hope it is not too late to wish you a happy new year! I hope that 2022 started well.

Quick question: what is the status of this issue?

meomancer commented 2 years ago

Hi @lucernae I think we can keep it as temporary as it is a bottleneck that can make uwsgi down when we are importing slower server I think we can patch it as we need in future for slower server

@clau1313 basically the original problem has been fixed Can you check it? https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-987505961

clau1313 commented 2 years ago

hi @meomancer, thanks, I can see the layers.

Question: is it possible to change the name of the layers? for instance, this one https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:arsenic2020 is now called arsenic2020, but I would like to call it "Global Arsenic (2020)". Or shall I ask the authors of the map to change it?

meomancer commented 2 years ago

renamed @clau1313 we can rename it by using edit layer of the geonode

clau1313 commented 2 years ago

Thanks @meomancer! I think we can close this issue now.

kartoza / IGRAC-GGIS

can't connect to remote service #405