config proxy backend and frontend

Gaetanbrl commented 3 years ago

A plugin use an external API (stoplight.io) but return GET - 403 error with these type of URL :

https://georchestra.example.org/mapstore/proxy/?url=https://jdev.stoplight.io/mocks/...

As explained by the doc, i check the georchestra.datadir/proxy.properties (same as default datadir) :

hostnameWhitelist and hostsWhitelist are comment #hostnameWhitelist = localhost,demo1.geo-solutions.it,demo.geo-solutions.it #hostsWhitelist = 127.0.0.1

This is localConfig.json

"proxyUrl": {
    "url": "proxy/?url=",
    "useCORS": [
          "https://jdev.stoplight.io",
          "http://demo.geo-solutions.it/geoserver",
          "https://demo.geo-solutions.it:443/geoserver",
          "https://demo.geo-solutions.it/geoserver",
          "https://nominatim.openstreetmap.org",
          "http://cloudsdi.geo-solutions.it",
          "https://gs-stable.geo-solutions.it/geoserver",
          "https://gs-stable.geo-solutions.it:443/geoserver"
    ],
    "autoDetectCORS": true
},

To test, i have uncomment hostnameWhitelist and added https://jdev.stoplight.io but request returns same 403 ERROR :

hostnameWhitelist = localhost,demo1.geo-solutions.it,demo.geo-solutions.it,jdev.stoplight.io

Is it the bad solution to change proxy.properties to allow this URL ?

Here, security-proxy.log :

2021-01-21 12:21:54 statistics [INFO] testadmin|2021/01/21 12:21:54|http://localhost:8280/mapstore/proxy/?url=https%3A%2F%2Fjdev.stoplight.io%2Fmocks%2Fjdev%2Ftabou_api_swagger%2F4964747%2Fcommunes|Project Steering Committee|ROLE_SUPERUSER,ROLE_MAPSTORE_ADMIN,ROLE_USER,ROLE_ADMINISTRATOR,ROLE_CKAN_SYSADMIN,ROLE_EXTRACTORAPP,ROLE_GN_ADMIN,ROLE_EMAILPROXY
2021-01-21 12:21:54 security [WARN] Downstream server returned a status code which could be an error. Statuscode: 403, reason:

pierrejego commented 3 years ago

Hello,

After @Gaetanbrl, I tried too. Our goal, is to make an extension call an outside Rest API.

Sorry for the spam, but I wrote all steps I had done, in case this could help someone in the futur :)

At start, there is no CORS issue with the url we want to call, so we just wanted to access directly from client to API without proxy.

So I try to change (based on this documentation in geOrchestra datadir ( here /etc/georchestra/mapstore/localConfig.json)

"useCORS": ["http://nominatim.openstreetmap.org", "https://nominatim.openstreetmap.org", "https://jdev.stoplight.io"] Just as it's already done for nominatim. (And is works for nominatim, request does not go throw proxy)

I restarted mapstore tomcat server, but request style go throw /mapstore/proxy/?url=.....

I tried to change in webapps directly in the webapps mapstore/localConfig.json, and restart the webapps. but same behaviour

I can see here that this conf is used only in needProxy https://github.com/geosolutions-it/MapStore2/search?q=needProxy

We developp an extension and used the same axios.get method from Mapstore. I can see here : https://github.com/geosolutions-it/MapStore2/blob/master/web/client/libs/ajax.js#L119 that the interceptor does not use the needProxy check useCORS values.

I saw a double check with autoDectectCORS, so I put it at false and restart, but still same behaviour.

I didn't manage to debug our extension once it has be build / deployed in georchestra via zip. (docs .

And by default, it works using npm start without configuration, extension doesnot try to use Proxy....

Perhaps extension needs additionnal configuration to use the defaut configuration from useCORS localConfig, I am still searching.

So at this point, I did not management the extension to call url without proxy, so I try to configure proxy to be able to call it.

Based on thoses documentation "georchestra" and documentation backendProxy I tried to add in proxy.properties a new regex.

I found reqtypeWhitelist.fdh in documentation but reqtypeWhitelist.generic in current file. I guess there was an update in code but not in the documentation code

So in the conf file I have :

hostnameWhitelist = localhost,gis.jdev.fr
reqtypeWhitelist.generic = (.*exist.*)|(.*pdf.*)|(.*map.*)|(.*wms.*)|(.*wmts.*)|(.*wfs.*)|(.*ows.*)|(.*stoplight.*)

I restart tomcat, but still same error ( 403 )

I can see the error in security-proxy.log

2021-02-11 10:51:06 statistics [INFO] pjego|2021/02/11 10:51:06|http://localhost:8280/mapstore/proxy/?url=https%3A%2F%2Fjdev.stoplight.io%2Fmocks%2Fjdev%2Ftabou_api_swagger%2F4964747%2Fetapes-pa-mock|JDev|ROLE_MV_ADMIN,ROLE_SUPERUSER,ROLE_CKAN_EDITOR,ROLE_MAPSTORE_ADMIN,ROLE_USER,ROLE_ADMINISTRATOR,ROLE_CKAN_SYSADMIN,ROLE_EXTRACTORAPP,ROLE_CKAN_ADMIN,ROLE_GN_ADMIN,ROLE_GN_EDITOR,ROLE_GN_REVIEWER
2021-02-11 10:51:06 security [WARN] Downstream server returned a status code which could be an error. Statuscode: 403, reason

So mapstore backend blocked the connection but, does not give a reason.

And no errors in mapstore.log, and no trace of request (even in debug)

I tried to change conf by other regex, or to make same changes in webapps file, but still same behaviour.

reqtypeWhitelist.generic = .*
reqtypeWhitelist.generic =( .*)
reqtypeWhitelist.generic = (.*stoplight.*)

I surely made a mistake somewhere, because now all requested going throw proxy are in error 403, while it was working before ( csw geobretagne for example)

So i am still searching, but if anybody have any hints on this two problems ( No proxy for Url and How to configure proxy)

Our Mapstore2 is base on this version https://packages.georchestra.org/bot/wars/20.1.x/ build from master the 28/01

catmorales commented 3 years ago

I don't know if it is the same problem but it seems. We can't use the geobretagne catalog from mapstore front end. @tdipisa could you please help us ?

On https://portail-test.sig.rennesmetropole.fr I added in the "initial state, catalog" section into the localConfig.json the entry :

"geobretagne": {
                "url": "https://geobretagne.fr/geonetwork/srv/fre/csw",
                "type": "csw",
                "title": "Catalogue GéoBretagne",
                "autoload": true
              },

I have an 403 error from mapstore proxy

https://portail-test.sig.rennesmetropole.fr/mapstore/proxy/?url=https%3A%2F%2Fgeobretagne.fr%2Fgeonetwork%2Fsrv%2Ffre%2Fcsw%3Fservice%3DCSW%26version%3D2.0.2%26request%3D

So i added in the proxy.properties in the hostnameWhiteList the entry "geobretagne.fr" and it's the same. (403 error)

With mapfishapp the same URL works and the request is https://portail-test.sig.rennesmetropole.fr/proxy/?url=https%3A%2F%2Fgeobretagne.fr%2Fgeonetwork%2Fsrv%2Ffre%2Fcsw

So how should i configure this catalog entry ?

catmorales commented 3 years ago

If this catalog entry don't need to pass by mapstore proxy (backend ?) , i added the entry "https://geobretagne.fr" into the localConfig.json too in the "useCORS" section.

tdipisa commented 3 years ago

@catmorales It seems that https://geobretagne.fr/ denies POST requests if they pass through the mapstore proxy for some reasons.

hostnameWhiteList is disabled by default, so you have to configure it for your mapstore proxy instance (eg. by putting geobretagne.fr) only if that property is uncommented in the proxy.properties file.

useCORS property in localConfig.json should be configured in that case only if geobretagne.fr allow CORS for your origin (portail-test.sig.rennesmetropole.fr) but it doesn't seems to be the case because the first OPTION request performed by mapstore fails with a 403 Forbidden.

Since the OPTION requests fails, mapstore tries to perform the Ajax request passing through the mapstore proxy but geobretagne.fr denies the HTTP request performed by that proxy. Therefore, I guess geobretagne.fr doesn't properly reconize HTTP requests performed by the mapstore proxy for some reasons or it is not able to identify the origin of the HTTP request.

What do you think @offtherailz?

tdipisa commented 3 years ago

@catmorales I tried from our DEV instance of MapStore and it works fine

Therefore, I suppose the problem is in your network (or in your security proxy?).

Can you anyway send your proxy.properties?

catmorales commented 3 years ago

Can you anyway send your proxy.properties?

I send it by email now

offtherailz commented 3 years ago

Hi, I tried the request from your mapfish proxy and it gives me the same issue ( I logged in before doing this request, so I should have the necessary cookies ). https://portail-test.sig.rennesmetropole.fr/proxy/?url=https%3A%2F%2Fgeobretagne.fr%2Fgeonetwork%2Fsrv%2Ffre%2Fcsw

I don't know if I missed something, but looking at the data I have now, I think that the most possible reason is that the request from geOrchstra to geobretagne server fails.

I can suggest to:

Make you sure that the host is reachable from the machines that are running geOrchstra
You can validate the HTTPS certificates of the proxy from the machines that are running geOrchstra

Doing a request with CURL to the server from one of the instances of geOrchstra parts (mapstore or mafishapp) to the server that gives problem may help to identify the issue.

You can also try to comment (starting with # all the lines) all the rules of proxy.properties, for test, so no security restriction is applied anymore, to exclude any intervention from the proxy itself.

catmorales commented 3 years ago

It is strange because it works well from mapfishapp and both are on the same server. But i'll try to do that.

MaelREBOUX commented 3 years ago

We met the same CORS errors when using the CRAIG mapstore https://ids.craig.fr/mapstore/

And we met this 404 error when querying GéoBretagne (https://geobretagne.fr/geoserver/cadastre/wms?service=wms&request=getcapabilities for example - nginx) or Région Bretagne (https://ows.region-bretagne.fr/geoserver/ows?service=wms&version=1.3.0&request=GetCapabilities apache2)

catmorales commented 3 years ago

@tdipisa, @offtherailz In fact, after checking and correcting the proxy settings, access to Geobretagne works when you do a targeted search, e.g. : "epci" If there is no search, it displays an error. Is the number of results restricted by Mapstore ?

catmorales commented 3 years ago

Accessing to Géobretagne services is now possible but we have already problems:

Parameters

proxy .properties : *hostnameWhitelist = localhost,portail-test.sig.rennesmetropole.fr,test-dmz-sigeosp.sig.rennesmetropole.fr,data.airbreizh.asso.fr,geobretagne.fr,public.sig.rennesmetropole.fr,10.40.25.36,10.40.25.37,jdev.stoplight.io,sig-geonetwork.mutu.local
localConfig.json: "proxyUrl": { "url": "proxy/?url=", "useCORS": [ "https://portail-test.sig.rennesmetropole.fr", "http://portail-test.sig.rennesmetropole.fr", "https://test-dmz-sigeosp.sig.rennesmetropole.fr" ], "autoDetectCORS": true },

Results

1- Accessing without filtering the research:

The request goes through the proxy where "geobretagne.fr" is present in whitelist and can't display the answer sended by Géobretagne Geobretagne

2- Accessing filtering the research with "EPCI":

The request goes through the proxy where "geobretagne.fr" is present in whitelist and displays the expected result

3- Issue accessing other results pages

The request goes through the proxy where "geobretagne.fr" is present in whitelist, and can't display the answer sended by Géobretagne. Filter is deleted so we have the same behaviour as 1. Clicking on the third page of results give

The behaviour is better on your demo platform.

@tdipisa, @offtherailz To help us to understand well how to configure that, could you please send us or give us an access to the localconfig.json and proxy.properties of your demo platform ? I need to close this problem. Thank you

georchestra / mapstore2-georchestra