search

geosolutions-it / http-proxy

Lean and Mean HTTP Proxy written in Java
GNU General Public License v3.0
21 stars 31 forks source link

Wrong content type application/xml instead of application/json #59

Closed offtherailz closed 2 years ago

offtherailz commented 2 years ago

In particular the problem is that the Accept header is not forwarded correctly: Sample request: 

curl --location --request POST 'https://dev-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DExecute' \
--header 'Connection: keep-alive' \
--header 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"' \
--header 'Accept: application/json, text/plain, */*' \
--header 'Content-Type: application/xml' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36' \
--header 'sec-ch-ua-platform: "Linux"' \
--header 'Origin: http://localhost:8081' \
--header 'Sec-Fetch-Site: same-origin' \
--header 'Sec-Fetch-Mode: cors' \
--header 'Sec-Fetch-Dest: empty' \
--header 'Referer: http://localhost:8081/' \
--header 'Accept-Language: en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7' \
--data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Aggregate</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>features</ows:Identifier><wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST"><wps:Body><wfs:GetFeature  outputFormat="GML2" service="WFS" version="1.0.0"><wfs:Query typeName="gs:us_states"></wfs:Query></wfs:GetFeature></wps:Body></wps:Reference></wps:Input><wps:Input><ows:Identifier>aggregationAttribute</ows:Identifier><wps:Data><wps:LiteralData>STATE_ABBR</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>function</ows:Identifier><wps:Data><wps:LiteralData>Count</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>singlePass</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:RawDataOutput mimeType="application/json"><ows:Identifier>result</ows:Identifier></wps:RawDataOutput></wps:ResponseForm></wps:Execute>'

Returns : Content-Type: application/xml but the content of the response is in JSON format

The direct request

curl --location --request POST 'https://gs-stable.geo-solutions.it/geoserver/wps?service=WPS&version=1.0.0&REQUEST=Execute' \
--header 'Connection: keep-alive' \
--header 'sec-ch-ua: " Not A;Brand";v="99", "Chromium";v="98", "Google Chrome";v="98"' \
--header 'Accept: application/json, text/plain, */*' \
--header 'Content-Type: application/xml' \
--header 'sec-ch-ua-mobile: ?0' \
--header 'User-Agent: Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.80 Safari/537.36' \
--header 'sec-ch-ua-platform: "Linux"' \
--header 'Origin: http://localhost:8081' \
--header 'Sec-Fetch-Site: same-origin' \
--header 'Sec-Fetch-Mode: cors' \
--header 'Sec-Fetch-Dest: empty' \
--header 'Referer: http://localhost:8081/' \
--header 'Accept-Language: en-US,en;q=0.9,it-IT;q=0.8,it;q=0.7' \
--data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Aggregate</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>features</ows:Identifier><wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST"><wps:Body><wfs:GetFeature  outputFormat="GML2" service="WFS" version="1.0.0"><wfs:Query typeName="gs:us_states"></wfs:Query></wfs:GetFeature></wps:Body></wps:Reference></wps:Input><wps:Input><ows:Identifier>aggregationAttribute</ows:Identifier><wps:Data><wps:LiteralData>STATE_ABBR</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>function</ows:Identifier><wps:Data><wps:LiteralData>Count</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>singlePass</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:RawDataOutput mimeType="application/json"><ows:Identifier>result</ows:Identifier></wps:RawDataOutput></wps:ResponseForm></wps:Execute>'

Returns : Content-Type: application/json the correct one.

nmco commented 2 years ago

@shadab-geo let's follow the usual workflow, and perform the initial assessment boxed to two hours maximum.

nmco commented 2 years ago

On a second thought since you are quite busy @shadab-geo, @ydeliorman can you have a look at this one?

tdipisa commented 2 years ago

@nmco @taba90 It seems to me it probably depends on latest http-proxy updates performed by @shadab-geo here and also on master (1.3-SNAPSHOT).

Performing the request to MS DEV passing through the proxy (v1.2-SNAPSHOT) we have Content-Type: application/xml in response Headers:

curl 'https://dev-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DExecute%26ms2-authkey%3Df5b9e2bf-d5c8-45ac-8a22-dd85932f560d' \
  -H 'Connection: keep-alive' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36' \
  -H 'Content-Type: application/xml' \
  -H 'Origin: https://dev-mapstore.geosolutionsgroup.com' \
  -H 'Referer: https://dev-mapstore.geosolutionsgroup.com/mapstore/' \
  -H 'Accept-Language: en,it;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6' \
  -H 'Cookie: _gcl_au=1.1.1799171789.1639741068; _ga=GA1.2.1536343127.1639741068' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Aggregate</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>features</ows:Identifier><wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST"><wps:Body><wfs:GetFeature  outputFormat="GML2" service="WFS" version="1.0.0"><wfs:Query typeName="gs:us_states"><ogc:Filter><ogc:And><ogc:Intersects><ogc:PropertyName>the_geom</ogc:PropertyName><gml:Polygon srsName="EPSG:3857"><gml:exterior><gml:LinearRing><gml:posList>-11114555.40889091 5033836.934748568 -11114555.40889091 5836119.983629778 -11858134.820049105 5836119.983629778 -11858134.820049105 5033836.934748568 -11114555.40889091 5033836.934748568</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects><ogc:Intersects><ogc:PropertyName>the_geom</ogc:PropertyName><gml:Polygon srsName="EPSG:3857"><gml:exterior><gml:LinearRing><gml:posList>-15987681.209931165 2317914.034685443 -15987681.209931165 6764714.59220386 -6595099.174248706 6764714.59220386 -6595099.174248706 2317914.034685443 -15987681.209931165 2317914.034685443</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects></ogc:And></ogc:Filter></wfs:Query></wfs:GetFeature></wps:Body></wps:Reference></wps:Input><wps:Input><ows:Identifier>aggregationAttribute</ows:Identifier><wps:Data><wps:LiteralData>LAND_KM</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>function</ows:Identifier><wps:Data><wps:LiteralData>Sum</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>singlePass</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:RawDataOutput mimeType="application/json"><ows:Identifier>result</ows:Identifier></wps:RawDataOutput></wps:ResponseForm></wps:Execute>' \
  --compressed \

While performing the request to MS QA passing through the proxy (v1.1.1) we have Content-Type: application/json in response Headers:

curl 'https://qa-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DExecute%26ms2-authkey%3Df5b9e2bf-d5c8-45ac-8a22-dd85932f560d' \
  -H 'Connection: keep-alive' \
  -H 'Accept: application/json, text/plain, */*' \
  -H 'User-Agent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36' \
  -H 'Content-Type: application/xml' \
  -H 'Origin: https://qa-mapstore.geosolutionsgroup.com' \
  -H 'Referer: https://qa-mapstore.geosolutionsgroup.com/mapstore/' \
  -H 'Accept-Language: en,it;q=0.9,fr-FR;q=0.8,fr;q=0.7,en-US;q=0.6' \
  -H 'Cookie: _gcl_au=1.1.1799171789.1639741068; _ga=GA1.2.1536343127.1639741068' \
  --data-raw '<?xml version="1.0" encoding="UTF-8"?><wps:Execute version="1.0.0" service="WPS" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns="http://www.opengis.net/wps/1.0.0" xmlns:wfs="http://www.opengis.net/wfs" xmlns:wps="http://www.opengis.net/wps/1.0.0" xmlns:ows="http://www.opengis.net/ows/1.1" xmlns:gml="http://www.opengis.net/gml" xmlns:ogc="http://www.opengis.net/ogc" xmlns:wcs="http://www.opengis.net/wcs/1.1.1" xmlns:dwn="http://geoserver.org/wps/download" xmlns:xlink="http://www.w3.org/1999/xlink" xsi:schemaLocation="http://www.opengis.net/wps/1.0.0 http://schemas.opengis.net/wps/1.0.0/wpsAll.xsd"><ows:Identifier>gs:Aggregate</ows:Identifier><wps:DataInputs><wps:Input><ows:Identifier>features</ows:Identifier><wps:Reference mimeType="text/xml" xlink:href="http://geoserver/wfs" method="POST"><wps:Body><wfs:GetFeature  outputFormat="GML2" service="WFS" version="1.0.0"><wfs:Query typeName="gs:us_states"><ogc:Filter><ogc:And><ogc:Intersects><ogc:PropertyName>the_geom</ogc:PropertyName><gml:Polygon srsName="EPSG:3857"><gml:exterior><gml:LinearRing><gml:posList>-11114555.40889091 5033836.934748568 -11114555.40889091 5836119.983629778 -11858134.820049105 5836119.983629778 -11858134.820049105 5033836.934748568 -11114555.40889091 5033836.934748568</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects><ogc:Intersects><ogc:PropertyName>the_geom</ogc:PropertyName><gml:Polygon srsName="EPSG:3857"><gml:exterior><gml:LinearRing><gml:posList>-15987681.209931165 2317914.034685443 -15987681.209931165 6764714.59220386 -6595099.174248706 6764714.59220386 -6595099.174248706 2317914.034685443 -15987681.209931165 2317914.034685443</gml:posList></gml:LinearRing></gml:exterior></gml:Polygon></ogc:Intersects></ogc:And></ogc:Filter></wfs:Query></wfs:GetFeature></wps:Body></wps:Reference></wps:Input><wps:Input><ows:Identifier>aggregationAttribute</ows:Identifier><wps:Data><wps:LiteralData>LAND_KM</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>function</ows:Identifier><wps:Data><wps:LiteralData>Sum</wps:LiteralData></wps:Data></wps:Input><wps:Input><ows:Identifier>singlePass</ows:Identifier><wps:Data><wps:LiteralData>false</wps:LiteralData></wps:Data></wps:Input></wps:DataInputs><wps:ResponseForm><wps:RawDataOutput mimeType="application/json"><ows:Identifier>result</ows:Identifier></wps:RawDataOutput></wps:ResponseForm></wps:Execute>' \
  --compressed \
offtherailz commented 2 years ago

I notify also this, giving bad request:

https://dev-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgeorchestra.geo-solutions.it%2Fgeoserver%2Fwms%3Fservice%3DWMS%26version%3D1.3.0%26request%3DGetCapabilities

While previous version works well: https://qa-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgeorchestra.geo-solutions.it%2Fgeoserver%2Fwms%3Fservice%3DWMS%26version%3D1.3.0%26request%3DGetCapabilities

tdipisa commented 2 years ago

I notify also this, giving bad request:

https://dev-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgeorchestra.geo-solutions.it%2Fgeoserver%2Fwms%3Fservice%3DWMS%26version%3D1.3.0%26request%3DGetCapabilities

While previous version works well: https://qa-mapstore.geosolutionsgroup.com/mapstore/proxy/?url=https%3A%2F%2Fgeorchestra.geo-solutions.it%2Fgeoserver%2Fwms%3Fservice%3DWMS%26version%3D1.3.0%26request%3DGetCapabilities

@offtherailz I've already opened a dedicate issue on MS for this: https://github.com/geosolutions-it/MapStore2/issues/7862

To unblock NPA DEV I've asked @offtherailz to switch back to http-proxy v1.1.1 (see here). We will test again using the NPA DEV instance directly as soon as Jankins deployed.

nmco commented 2 years ago

@ydeliorman is investigating this one.

nmco commented 2 years ago

Meeting minutes:

Next steps from us:

nmco commented 2 years ago

In GeoServer classic OGC services (WPS, WMS, WFS, ...) don' care about the requests headers, but they need to set the right response headers.

In this case, the issue is not the content which is JSON as expected, it's that the Content-type header for the response is wrong, can you check this @ydeliorman.

ydeliorman commented 2 years ago

working on this

ydeliorman commented 2 years ago

@nmco

We can deduce from the top, geoserver wps request returns an incorrect content-type header in the result and http-proxy 1.2 fixes it. I will investigate;

  1. Make a WPS request with content-type from my local and see if the response returns correctly.
  2. Make the same WPS request using http-proxy 1.1 and 1.2 snapshots.
ydeliorman commented 2 years ago

@nmco tried to do the below investigation to see if the content-type header problem is related to GeoServer or http-proxy 1.1 or 1.2 but was faced with the errors mentioned below. I will try to do fix below errors.

  1. Did a WPS request according to the doc to the local geoserver to see if the response content-type header returns correctly. Request and response content-body header are not the same.

  2. Execute the same WPS request using local http-proxy 1.1 and 1.2 http://localhost:8080/http_proxy/proxy?url=https%3A%2F%2Fgs-stable.geo-solutions.it%2Fgeoserver%2Fwps%3Fservice%3DWPS%26version%3D1.0.0%26REQUEST%3DExecute Faced with 403: Request Type is not among the ones allowed for this proxy

aaime commented 2 years ago

For the first issue, go and enable entity resolution in the global settings (it's an issue that would not happen running from the jars, but shows up when running from the sources, don't worry about it):

image

shad-git commented 2 years ago

regarding the second issue: you need to whitelist the GeoServer path. If you are running a local http-proxy using jetty server (i.e. via start class under test package). You can update the below property in the file src/test/resources/proxy.properties file as below. reqtypeWhitelist.geostore = (.*geostore.*)|(.*geoserver.*) This will allow both the paths i.e. geoserver as well as geostore.

ydeliorman commented 2 years ago

@nmco @aaime @shadab-geo thank you very much for unblocking. We have multiple scenarios in terms of response's header content-type

  1. When a WPS request is done to the GeoServer without http-proxy, the response's content-type is different from the requests' content-type. Not expected behavior.
  2. When a WPS request is done to the GeoServer using 1.2 http-proxy, the response's content-type is the same with requests' content-type. Expected behavior.
  3. When a WPS request is done to the GeoServer using 1.1 http-proxy, the response's content-type is different from the requests' content-type. Not expected behavior.

We can say that 1.2 http-proxy is running correctly in terms of having the same content-type both in request and response. WPS request from GeoServer without proxy and WPS request using 1.1 http-proxy have different content-type in request and response headers.

nmco commented 2 years ago

We clarified the issue with @ydeliorman he will get back with more feedback today.

ydeliorman commented 2 years ago

@nmco For http-proxy 1.1, the response's Content-Type is changed from <wps:RawDataOutput mimeType="application/json"> element in the body. For http-proxy 1.2, the response's Content-Type has not affected the changes from the mimeType element in the body. Response's Content-Type is changed from request header Content-Type parameter.

The reason for this change in behavior is the change in the libraries. After this line of code execution Content-type is set to the correct one in 1.1. However in 1.2 since the libraries are different it is not the same behavior.

What we can do?

  1. For 1.2 http-proxy we can set Content-Type from the request's header content-type. For 1.1 http-proxy we can set Content-Type from the body's mimeType element.
  2. Or we can use the previous libraries if we want the behaviour in 1.1
ydeliorman commented 2 years ago

@nmco

We had a look with @taba90 to the error and managed to solve it. The problem is with the old library that is being used in the 1.1 version, response and requests headers are handled in a different way. The solve the issue we need to change here httpMethodProxyRequest.getAllHeaders() to response.getAllHeaders()

@tdipisa @offtherailz In the 1.1 version when we set <wps:RawDataOutput mimeType="application/xml"> response body's Content-Type returns text/xml. It will be the same case when we do the fix in 1.2 version. Is that suitable for your case?

The plan: changing the code, some manual tests, and writing a unit test will take 4 hours.

nmco commented 2 years ago

Go ahead @ydeliorman, we need to close this.

ydeliorman commented 2 years ago

@nmco PR is here: https://github.com/geosolutions-it/http-proxy/pull/61 . Controlled with @taba90

tdipisa commented 2 years ago

Thank you @ydeliorman. @taba90 can you review the PR please and let me know?

tdipisa commented 2 years ago

@offtherailz since we have a fix on http-proxy master. It would be good to test it on NPA MS project to verify if the problem is solved.

ydeliorman commented 2 years ago

@nmco PR is merged. Do we need any backporting?

tdipisa commented 2 years ago

Please wait @ydeliorman. Let's test it before. @ydeliorman @taba90 please let me know if the MVN repo is updated with these latest changes provided on master.

tdipisa commented 2 years ago

@offtherailz the MVN repo is aligned with latest changes provided on http-proxy master branch. Can you please do a quick check by using 1.3-SNAPSHOT as a new dependency version for http-proxy?

offtherailz commented 2 years ago

I took a look at the solutions, it seems rational. I checked and it is deployed for 1.3-SNAPSHOT (build server geo-solutions.it builds master) and so present on maven.geo-solutions.it I tested the CURL request that was failing on dev-mapstore and now it works now

LGTM

tdipisa commented 2 years ago

Thank you so much @offtherailz. @ydeliorman the new updates for the http-proxy that are involving the upgrade of the apache http client are expected to be backported until 1.2.x. The release 1.2.0 should be finally amended as soon as also this issue is fixed and tested.

tdipisa commented 2 years ago

@ydeliorman I've tested this. Please see my comment here.

tdipisa commented 2 years ago

Tested in c027 client environment with both 1.3-SNAPSHOT and 1.2-SNAPSHOT It is working now.