Closed SterckxArnaud closed 2 years ago
@ArnaudIGRAC The problem is with the upstream resource. I tried to connect in QGIS and it also times out. Maybe we need to add a better error message asking users to inspect the upstream service
thanks @NyakudyaA. But if I can access the layers in a web browser (https://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS), there must be a way to also get them in GeoNode (on in QGIS)?
@ArnaudIGRAC GeoNode basically looks at all layers available through a get capabilities request i.e https://www.gapmaps.org/geoserver/wms?REQUEST=GetCapabilities So this is what is taking time so maybe @lucernae needs to increase the timeout time in nginx so that the request goes on further than the allocated time
hi @ArnaudIGRAC as per @NyakudyaA said that the request to get the wms has limit time. Currently the limit is 1 minute, need to increase the limit of nginx to more than that.
But i checked on my local which does not using nginx (so it is unlimited) It is taking to long for the connection from the gapmaps and it returns "Request Time Out" from gapmaps too
So there is no way to fix it from our side (we increase the time, it is Request Time Out from gapmaps)
Thank you for your feedback. Is there a possibility to fetch one layer instead of a broad get capabilities request? This if for instance what I did for some FAO layers http://data.fao.org/maps/wms?styles=geonetwork_DSMW_14116_style http://data.fao.org/maps/wms?styles=geonetwork_dams_africa_38077_style http://data.fao.org/maps/wms?styles=aquamaps_rivers_africa_37333_style
hi @seabilwe could you help me registers those layer please? thank you
Dear colleagues, any update on this?
@ArnaudIGRAC Which layers do you need for us to register.
There are two layers (and their color scales):
@meomancer This service is extremely slow to do a getcapabilties request so even if you request a single layer is still times out because the server still does a getcapabiltiies for the whole server. So upstream improvements could help this issue
Layer
defined do a getcapabitlies for a single layer than the whole server i.e http://osgeo-org.1560.x6.nabble.com/GetCapabilities-Filter-by-Layer-td5082208.htmlhi @NyakudyaA thank you
hi @ArnaudIGRAC yes, as per i said in here https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-802498711
we need to ask the service to increase their time out and also the comments from admire
Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html
Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html
Dear colleagues, any answer to my question above?
Thank you for your feedback. Would it be possible to shortcut these limitations by importing the layers straight into GeoServer? https://docs.geoserver.org/latest/en/user/data/cascaded/wms.html
Dear colleagues, any answer to my question above?
@ArnaudIGRAC I think it would be the same thing, because the GeoNode front end is just abstracting the loading and publishing of layers into GeoServer/PostGIS database.
I will try them out again both from the GeoNode frontend and GeoServer. Otherwise, we could try to do this from the backend if possible, Maybe that would work
thank you Admire, let me know if it works. Otherwise we could also contact the colleagues managing this server to seek a solution
Dear all, is there any update on this?
hi @meomancer, is there any update on this? shall we do something from this side?
hi @clau1313 i'm not sure how to fix it because the problem is from their server that we can't get information of the layer and register it on geonode https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-826563463
so i think we can ask solution from their side
hi @meomancer, thanks, I wrote yesterday to the people in charge of the GAP portal to see if they can solve the problem. I will let you know when they answer.
hi @meomancer, I received an answer from the GAP portal. Unfortunately they cannot identify one specific problem, and they asked me if it would be okay if they communicate directly with you. Can I do that? If so then I will give them your email address.
Please let me know. Thanks!
hi @clau1313 i'm not sure the solution from their site maybe i need @NyakudyaA to communicate with them how to overcome our issue from their side
could you help me @NyakudyaA ?
hi @clau1313 i'm not sure the solution from their site maybe i need @NyakudyaA to communicate with them how to overcome our issue from their side
could you help me @NyakudyaA ?
@clau1313 If they are using GeoServer they can increase the timeout for their service as depicted here https://geoserver.geo-solutions.it/multidim/enterprise/resource_limits.html
If their service is mapserver,qgis server etc hey should also see how to optimise the timeout for wms services
hi @NyakudyaA, thanks, can I give the developers at GAP Maps your email address in case they have more questions?
hi @NyakudyaA, thanks, can I give the developers at GAP Maps your email address in case they have more questions?
Yes @clau1313
@NyakudyaA what is your email address? I am not sure if I have it...
@NyakudyaA what is your email address? I am not sure if I have it...
Sorry I thought you had it, admire@kartoza.com
@meomancer Can we try to increase the Nginx timeout to the maximum we can? I have been given new links but they all fail to retrieve the layer. An alternative solution might be to actually ingest the layers directly into the database but we would need to update all the links for the resources
New links provided in email are Global arsenic (2020):
Global fluoride (2008):
hi @NyakudyaA it turns out i'm not sure how to increase it on rancher2
hi @lucernae could you help us increase timeout for uwsgi and nginx? i'm not sure where nginx.conf is (i changed on container but it is readonly)
@meomancer for future reference, you can easily check config location in Rancher. Select the container you want to see:
In the volumes sections, find the file you want to override, then click the link to the ConfigMap definition:
In this case here, we change it in geonode.conf:
I increase it to 72000s (20 hours) for now.
If I look at the log, it seems the one doing the timeout is uwsgi, so we need to also increase uwsgi timeout.
Sorry I was not being clear. I mean: the timeout inside the uwsgi process. It could be python job, requests module, etc. I'm not sure. But probably it's in the GeoNode module?
Currently, Increasing the uwsgi timeout parameter in nginx only results with this:
Current addition of timeout in uwsgi parameter in inginx:
Harakiri parameter (uwsgi.conf):
The timeout of the process is flat out 1 mins:
If I read the uwsgi module in nginx, 75 s is the hard limit. They say it can't actually be set greater than that. So I think the processing needs to be moved as a background job if the remote GetCapabilities requests takes too much time to process.
hi @lucernae thank you very much for increasing timeout
@NyakudyaA It is still timeout from the remote service
Traceback (most recent call last):
File "/spcgeonode/geonode/services/serviceprocessors/handler.py", line 67, in get_service_handler
service = handler(base_url)
File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 127, in __init__
INDEXED if self._offers_geonode_projection() else CASCADED)
File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 459, in _offers_geonode_projection
if len(list(self.get_resources())) > 0:
File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 208, in get_resources
contents_gen = self.parsed_service.contents.values()
File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 162, in parsed_service
timeout=ogc_server_settings.get('TIMEOUT', 60))
File "/spcgeonode/geonode/services/serviceprocessors/wms.py", line 108, in WebMapService
timeout=timeout, headers=headers))
File "/usr/local/lib/python3.7/site-packages/owslib/map/wms130.py", line 75, in __init__
self._capabilities = reader.read(self.url, timeout=self.timeout)
File "/usr/local/lib/python3.7/site-packages/owslib/map/common.py", line 66, in read
timeout=timeout, headers=self.headers, auth=self.auth)
File "/usr/local/lib/python3.7/site-packages/owslib/util.py", line 208, in openURL
req = requests.request(method.upper(), url_base, headers=headers, **rkwargs)
File "/usr/local/lib/python3.7/site-packages/requests/api.py", line 61, in request
return session.request(method=method, url=url, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.7/site-packages/requests/adapters.py", line 529, in send
raise ReadTimeout(e, request=request)
requests.exceptions.ReadTimeout: HTTPSConnectionPool(host='www.gapmaps.org', port=443): Read timed out. (read timeout=60)
[uwsgi-http key: ggis.un-igrac.org client_addr: 188.34.198.238 client_port: 37026] hr_read(): Connection reset by peer [plugins/http/http.c line 917]
@meomancer sorry I'm still editing my comments and only finished after you posted, so I summarize again in this comment.
In summary:
uwsgi_connect_timout
setting. I guess, this makes sense because nginx is a UI thread for the web, it needs to finish quickly to avoid becoming zombie thread or blocking other process.local_settings.py
and pass it to this ogc_server_settings
object.i think the solution is saving it manually on database will ask time for working on it
I just did a small patch in the k8s deployment to test this. If I increase the OGC Timeout parameter, the register process does returns after around 10 mins.
I still don't think it is a good idea though, because the uwsgi thread is blocked during that time and we don't know if there will be more than one process doing this. Because if it is, then the web app is going to be blocked or less responsive. Management command or python job is a more better idea than this.
Note: Also since the edit was an ephemeral patch, it will be gone in the next GGIS docker image upgrade.
I just did a small patch in the k8s deployment to test this. If I increase the OGC Timeout parameter, the register process does returns after around 10 mins.
I still don't think it is a good idea though, because the uwsgi thread is blocked during that time and we don't know if there will be more than one process doing this. Because if it is, then the web app is going to be blocked or less responsive. Management command or python job is a more better idea than this.
Note: Also since the edit was an ephemeral patch, it will be gone in the next GGIS docker image upgrade.
Can you register remote services through management commands ? If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend
Can you register remote services through management commands ?
I don't actually know. From what I've understand, the bottleneck is this initial GetCapabilities request where you are not yet registering any service, but only listing all services. Doing it with the management command will still result in 10 mins waiting in the terminal. Unless you page the request to the remote server. For example if you want only the first 10 results to be shown initially. (the bottleneck is that the remote services takes too long to return all layers).
If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend
Not sure I understand what you mean by this. Can you elaborate? My point is that, after testing the UI, there are two phase: "Register service" and "Import Resources". When user clicks Create, this will "Register" the WMS endpoint. No layer is saved yet, and it seems the previous GetCapabilities requests are also not indexed. I tried traversing the layers list pagination in the above page and it seems to be requesting GetCapabilities again to the server, so I had to wait another 10 mins. If we make it a management command or python Job, we could process it in the background and store the cache somewhere in the DB. So that when the UI access it, the lists comes from the cache, at lest for the initial pagination (not the filters).
Can you register remote services through management commands ?
I don't actually know. From what I've understand, the bottleneck is this initial GetCapabilities request where you are not yet registering any service, but only listing all services. Doing it with the management command will still result in 10 mins waiting in the terminal. Unless you page the request to the remote server. For example if you want only the first 10 results to be shown initially. (the bottleneck is that the remote services takes too long to return all layers).
I am sure there must be something happening within the context of GeoNode because a straight getcapabilities report doesn't go over 5minutes in my browser and even when I time it in the shell
If it's a once off job I would like to believe you can test what commands are being fired against the DB when a user registers a remote service and then just do the same unless getting the code that registers remote service and run it manually in the backend
Not sure I understand what you mean by this. Can you elaborate? My point is that, after testing the UI, there are two phase: "Register service" and "Import Resources". When user clicks Create, this will "Register" the WMS endpoint. No layer is saved yet, and it seems the previous GetCapabilities requests are also not indexed. I tried traversing the layers list pagination in the above page and it seems to be requesting GetCapabilities again to the server, so I had to wait another 10 mins. If we make it a management command or python Job, we could process it in the background and store the cache somewhere in the DB. So that when the UI access it, the lists comes from the cache, at lest for the initial pagination (not the filters).
I think we are talking the same thing. What I also mean is that when you do register the service it will do a getcapabilities and store those results in the DB in a particular table and when you select a layer to add to your geonode we also add that layer to the django base tables. So what I was suggesting is just if it doesn't take time to understand what SQL commands are fired to insert this records in the DB, we just replicate it outself otherwise if there is a better solution then the better
Thank you very much @lucernae now it works
CC @NyakudyaA @ArnaudIGRAC
Here are the layers https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:fluoride https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:arsenic2020
Please check it
I am sure there must be something happening within the context of GeoNode because a straight getcapabilities report doesn't go over 5minutes in my browser and even when I time it in the shell
@NyakudyaA I also did a straight get capabilities and in my case it took only slightly less than 10 mins. Perhaps my region or ISP affect this as well.
So, @meomancer what should I do with the temporary patch? Should I make it permanent?
Hi everyone, I hope it is not too late to wish you a happy new year! I hope that 2022 started well.
Quick question: what is the status of this issue?
Hi @lucernae I think we can keep it as temporary as it is a bottleneck that can make uwsgi down when we are importing slower server I think we can patch it as we need in future for slower server
@clau1313 basically the original problem has been fixed Can you check it? https://github.com/kartoza/IGRAC-GGIS/issues/405#issuecomment-987505961
hi @meomancer, thanks, I can see the layers.
Question: is it possible to change the name of the layers? for instance, this one https://ggis.un-igrac.org/layers/httpswwwgapmapsorggeoserverwms:gap:arsenic2020 is now called arsenic2020, but I would like to call it "Global Arsenic (2020)". Or shall I ask the authors of the map to change it?
renamed @clau1313 we can rename it by using edit layer of the geonode
Thanks @meomancer! I think we can close this issue now.
Why can I not connect to this service? https://www.gapmaps.org/geoserver/wms? I get this error message: The service is working properly, see for instance https://www.gapmaps.org/geoserver/wms?REQUEST=GetMap&LAYERS=gap:arsenic2020&WIDTH=2500&HEIGHT=972&BBOX=-180.0,-56.0,180.0,84.0&FORMAT=image%2Fpng&VERSION=1.1.1&SERVICE=WMS