search

interactive-instruments / ldproxy

Share geospatial data via modern Web APIs
Mozilla Public License 2.0
68 stars 10 forks source link

ProxyPass with externalURL setting does not work for subfolders #94

Closed azahnen closed 4 years ago

azahnen commented 6 years ago

ldproxy needs to know the URL it will be accessed with, so if proxying it e.g. with apache httpd or nginx, the externalURL parameter has to be set. This works as long as ldproxy is still available in the root folder, but not when it is in a subfolder like http://localhost/ldproxy/.

KathiSchleidt commented 6 years ago

So do I understand correctly that this cannot be modified within the configuration?

Also - are the 2 subpaths I set (manager and rest) all, or must I set additional ones? If I don't do this on the subpath level, I cannot run any other services on this server!

On 02.03.2018 15:18, Andreas Zahnen wrote:

ldproxy needs to know the URL it will be accessed with, so if proxying it e.g. with apache httpd or nginx, the externalURL parameter has to be set. This works as long as ldproxy is still available in the root folder, but not when it is in a subfolder like |http://localhost/ldproxy/|.

— You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub https://github.com/interactive-instruments/ldproxy/issues/94, or mute the thread https://github.com/notifications/unsubscribe-auth/ALXQKBKJTZXa89SLSxG4VjKFV2X2vWICks5taVSagaJpZM4SZ9CX.

--

Katharina Schleidt Tel: +43 (1) 89 234 26 Mobile: +43 (650) 89 234 26 Skype: Kathi Schleidt Kathi@Schleidt.org www.schleidt.org/kathi

In the twenty-first century censorship works by flooding people with irrelevant information. -- Homo Deus, Yuval Noah Harari

The people can always be brought to the bidding of the leaders. That is easy. All you have to do is tell them they are being attacked and denounce the pacifists for lack of patriotism and exposing the country to danger. It works the same way in any country. --Hermann Göring, Nazi military leader (12 Jan 1893-1946)

cportele commented 4 years ago

Code has been rewritten for the 2.0 release.

The external URL can be set in the configuration, we will document this in the documentation update for version 2.0.

justb4 commented 3 years ago

We have a similar situation: ldproxy gives 404 behind a subpath, even with externalURL set, context:

The above setup works for other deployed services like pygeoapi. There we need to set SCRIPT_NAME to the subpath, but that is a Python-Flask-thing. IMHO all these settings should not be necessary (also in pygeoapi) when the front-end proxy adds X-Forwarded-For and/or X-Script-Name HTTP headers on which the backend service should act. I think GeoServer does something like that.

How can we solve this with the current versions of Traefik and ldproxy?

justb4 commented 3 years ago

Working! Doing a similar thing in Traefik as classic Apache ProxyPass: using ReplacePath Traefik Middleware. Still need some ldproxy config+data to prove.

See our new config here. Maybe this is something for the ldproxy documentation, as Traefik is used extensively.

cportele commented 3 years ago

Thanks @justb4. We are also using Traefik in the xyz.ldproxy.net deployments, but then always without a path.

We will have a look, how we can update/improve the documentation.

I think there might still an issue that we need to address, because the css link does not work.

azahnen commented 3 years ago

@justb4 ReplacePath is not the right tool for this case, it will replace any path with the static path /rest/services. Instead you should use the ReplacePathRegex middleware:

- traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.regex=(.*)
- traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.replacement=/rest/services/$$1

For simpler cases without subpath we use the AddPrefix middleware:

- traefik.http.middlewares.ldproxy-replace-restservices-path.addprefix.prefix=/rest/services
justb4 commented 3 years ago

@azahnen thanks. I tried your suggestion within https://github.com/Geonovum/ogc-api-testbed/issues/6, and several variants, but no success.

Aim is to run ldproxy on a subpath via Traefik as /rest/services, also disabling manager: https://oapi.map5.nl/ldproxy/ (domain name will change next week) or local: http://localhost:8000/ldproxy/

When trying your suggestion, still /ldproxy is attached to /rest/services. Then I tried (current version) to use Traefik middleware that first strips the ldproxy prefix before adding the regex to render /rest/services$$1. Also /rest/services/$$1 gave double slashes, //. We are stuck here. Also leaving out the regex-replace and just strip the ldproxy prefix and call /ldproxy/rest/services did not work...

Our Docker compose file with the Traefik config. We did get services behind Traefik working for pygeoapi (using SCRIPT_NAME=pygeoapi , Python Flash-specific) and GeoServer. Maybe an HTTP header that should be set in Traefik?

azahnen commented 3 years ago

@justb4 Sorry, that was just a theoretical example and obviously it does not work. I set up the example in my environment, here is a minimal working docker-compose.yml to run ldproxy at http://localhost:8080/ldproxy:

version: "3.3"

services:
  traefik:
    image: traefik:latest
    command:
      - --providers.docker=true
      - --providers.docker.exposedByDefault=false
      - --entrypoints.http.address=:8080
    ports:
      - 8080:8080
    volumes:
      - /var/run/docker.sock:/var/run/docker.sock

  ldproxy:
    image: iide/ldproxy:master
    volumes:
      - ./data:/ldproxy/data
    labels:
      - "traefik.enable=true"

      - "traefik.http.services.ldproxy.loadbalancer.server.port=7080"

      - "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.regex=/ldproxy(.*)"
      - "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.replacement=/rest/services$$1"

      - "traefik.http.routers.ldproxy_http.rule=Host(`localhost`) && PathPrefix(`/ldproxy`)"
      - "traefik.http.routers.ldproxy_http.entrypoints=http"
      - "traefik.http.routers.ldproxy_http.middlewares=ldproxy-replace-restservices-path"
justb4 commented 3 years ago

Thanks @azahnen , your suggestion works on localhost. A WFS provider has been configured, collections can be navigated, logs look good, ignoring some layer-errors: 

DEBUG [2021-06-12 13:31:46,834]  RCE_Landschapsatlas_WFS - Creating http client for host: https://services.rce.geovoorziening.nl/landschapsatlas/wfs?service=WFS&request=GetCapabilities&version=2.0.0  
DEBUG [2021-06-12 13:31:47,195]  RCE_Landschapsatlas_WFS - HTTP GET request: https://services.rce.geovoorziening.nl/landschapsatlas/wfs?REQUEST=GetCapabilities&SERVICE=WFS&VERSION=2.0.0  
DEBUG [2021-06-12 13:31:49,888]  RCE_Landschapsatlas_WFS - warming up GeoTools Filter ...  
DEBUG [2021-06-12 13:31:51,147]  RCE_Landschapsatlas_WFS - BBOX geometry | EPSG:25833 | null  
DEBUG [2021-06-12 13:31:51,151]  RCE_Landschapsatlas_WFS - PROP geometry null  
DEBUG [2021-06-12 13:31:56,702]  RCE_Landschapsatlas_WFS - done  
ERROR [2021-06-12 13:31:57,169]  RCE_Landschapsatlas_WFS - Collection 'groenerfgoed' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,175]  RCE_Landschapsatlas_WFS - Collection 'begraafplaatsen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,176]  RCE_Landschapsatlas_WFS - Collection 'bruinkoolgroeves' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,178]  RCE_Landschapsatlas_WFS - Collection 'droogmakerijen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,178]  RCE_Landschapsatlas_WFS - Collection 'fabrieksschoorstenen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,180]  RCE_Landschapsatlas_WFS - Collection 'gemalen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,181]  RCE_Landschapsatlas_WFS - Collection 'gesloten_stations' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,183]  RCE_Landschapsatlas_WFS - Collection 'haltes_en_depots_museumspoorlijnen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,185]  RCE_Landschapsatlas_WFS - Collection 'haltes_oorlogsspoorlijnen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,187]  RCE_Landschapsatlas_WFS - Collection 'hertenkampen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,192]  RCE_Landschapsatlas_WFS - Collection 'luchtwachttorens' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,194]  RCE_Landschapsatlas_WFS - Collection 'museumspoorlijnen' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,195]  RCE_Landschapsatlas_WFS - Collection 'stations_en_haltes' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,200]  RCE_Landschapsatlas_WFS - Collection 'turfwinning' has been disabled because its feature type has no property with role ID in the provider schema.  
ERROR [2021-06-12 13:31:57,205]  RCE_Landschapsatlas_WFS - Collection 'wielen' has been disabled because its feature type has no property with role ID in the provider schema.  
INFO  [2021-06-12 13:31:57,702]  RCE_Landschapsatlas_WFS - Feature provider with id 'RCE_Landschapsatlas_WFS' started successfully.  
DEBUG [2021-06-12 13:31:57,792]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointConformance  
DEBUG [2021-06-12 13:31:57,944]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointConformance  
DEBUG [2021-06-12 13:31:57,963]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointDefinition  
DEBUG [2021-06-12 13:31:58,033]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointDefinition  
DEBUG [2021-06-12 13:31:58,045]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointLandingPage  
DEBUG [2021-06-12 13:31:58,095]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointLandingPage  
DEBUG [2021-06-12 13:31:58,100]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointCollection  
DEBUG [2021-06-12 13:31:58,181]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointCollection  
DEBUG [2021-06-12 13:31:58,198]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointFeatures  
DEBUG [2021-06-12 13:31:58,717]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointFeatures  
DEBUG [2021-06-12 13:31:58,727]  RCE_Landschapsatlas_WFS - Generating API definition for EndpointCollections  
DEBUG [2021-06-12 13:31:58,741]  RCE_Landschapsatlas_WFS - Finished generating API definition for EndpointCollections  
INFO  [2021-06-12 13:31:58,779]  RCE_Landschapsatlas_WFS - Service with id 'RCE_Landschapsatlas_WFS' started successfully.  
DEBUG [2021-06-12 13:31:58,803]                          - Listening for events for [entities, overrides]  
DEBUG [2021-06-12 13:32:21,960]                          - Processing request: GET /rest/services/ [17b57355-77c4-4ae6-bb10-32e746b2e47c] 
DEBUG [2021-06-12 13:32:22,044]                          - Sending response: 200 OK [17b57355-77c4-4ae6-bb10-32e746b2e47c] 
DEBUG [2021-06-12 13:32:38,595]  RCE_Landschapsatlas_WFS - Processing request: GET / [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:38,722]  RCE_Landschapsatlas_WFS - accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:38,802]  RCE_Landschapsatlas_WFS - content-type Optional[ApiMediaType{type=text/html, label=HTML, parameter=html, qs=1000}] [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:38,806]  RCE_Landschapsatlas_WFS - accept-language en-GB,en-US;q=0.9,en;q=0.8 [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:38,823]  RCE_Landschapsatlas_WFS - content-language Optional[en] [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:38,967]  RCE_Landschapsatlas_WFS - Sending response: 200 OK [13647036-4bea-43c8-b28a-081c619ec15c] 
DEBUG [2021-06-12 13:32:51,211]  RCE_Landschapsatlas_WFS - Processing request: GET /collections [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:32:51,231]  RCE_Landschapsatlas_WFS - accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:32:51,232]  RCE_Landschapsatlas_WFS - content-type Optional[ApiMediaType{type=text/html, label=HTML, parameter=html, qs=1000}] [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:32:51,233]  RCE_Landschapsatlas_WFS - accept-language en-GB,en-US;q=0.9,en;q=0.8 [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:32:51,233]  RCE_Landschapsatlas_WFS - content-language Optional[en] [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:32:51,500]  RCE_Landschapsatlas_WFS - Sending response: 200 OK [8c4bfe13-82e3-49de-b81f-c74c81540f7b] 
DEBUG [2021-06-12 13:33:08,884]  RCE_Landschapsatlas_WFS - Processing request: GET /collections/molens/items [be9394fd-80f4-4f25-87bd-7c205f498567]

But on the server the Traefik path middelware works, but cannot get through the WFS Collection. Logs:

DEBUG [2021-06-12 13:49:41,985]                          - CAPABILITY CapabilityFoundation  
DEBUG [2021-06-12 13:49:42,543]  RCE_Landschapsatlas_WFS - warming up GeoTools Filter ...  
DEBUG [2021-06-12 13:49:43,321]  RCE_Landschapsatlas_WFS - BBOX geometry | EPSG:25833 | null  
DEBUG [2021-06-12 13:49:43,323]  RCE_Landschapsatlas_WFS - PROP geometry null  
DEBUG [2021-06-12 13:49:49,395]  RCE_Landschapsatlas_WFS - done  
INFO  [2021-06-12 13:49:49,801]  RCE_Landschapsatlas_WFS - Feature provider with id 'RCE_Landschapsatlas_WFS' started successfully.  
INFO  [2021-06-12 13:49:49,938]  RCE_Landschapsatlas_WFS - Service with id 'RCE_Landschapsatlas_WFS' started successfully.  
DEBUG [2021-06-12 13:49:49,983]                          - Listening for events for [entities, overrides]  
DEBUG [2021-06-12 13:50:07,569]                          - Processing request: GET /rest/services [c4d5df22-c7f5-40da-9876-efaa63cedc8f] 
DEBUG [2021-06-12 13:50:07,718]                          - Sending response: 200 OK [c4d5df22-c7f5-40da-9876-efaa63cedc8f] 
DEBUG [2021-06-12 13:50:13,096]  RCE_Landschapsatlas_WFS - Processing request: GET / [0dd44114-5f6a-47ba-be99-cb98fbe57f9b] 
DEBUG [2021-06-12 13:50:13,202]  RCE_Landschapsatlas_WFS - accept text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9 [0dd44114-5f6a-47ba-be99-cb98fbe57f9b] 
DEBUG [2021-06-12 13:50:13,217]  RCE_Landschapsatlas_WFS - content-type Optional[ApiMediaType{type=text/html, label=HTML, parameter=html, qs=1000}] [0dd44114-5f6a-47ba-be99-cb98fbe57f9b] 
DEBUG [2021-06-12 13:50:13,225]  RCE_Landschapsatlas_WFS - Client error, HTTP status 404, Request path /RCE_Landschapsatlas_WFS: The requested path is not a resource in this API. [0dd44114-5f6a-47ba-be99-cb98fbe57f9b] 
DEBUG [2021-06-12 13:50:13,228]  RCE_Landschapsatlas_WFS - Sending response: 404 Not Found [0dd44114-5f6a-47ba-be99-cb98fbe57f9b]

Somehow the WFS Resource is not processed and https://oapi.map5.nl/ldproxy/RCE_Landschapsatlas_WFS gives 404 (the localhost version works). Configuration is identical, in GitHub. Could it be permission problems ? We had permission problems, container changing file-ownership, with GeoServer.

Also I tried the same convention to access the manager on localhost via /manager_ldproxy (we like to run Manager only on localhost, checking-in the result config). But when trying the login form cannot login, error: manager.404 page not found ._label. Relevant config is:


  ldproxy:
    # Seems 'latest' is another version
    image: iide/ldproxy:master

    container_name: ldproxy

    expose:
      - "7080"

#    ports:
#      - "7080:7080"

    volumes:
      - ./data:/ldproxy/data

    labels:
      # Enable Traefik routing on overlay service network
      - "traefik.enable=true"
      - "traefik.docker.network=service-network"

      # This will replace any /ldproxy/* with /rest/services/* when calling the ldproxy container
      - "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.regex=/ldproxy(.*)"
      - "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.replacement=/rest/services$$1"

      - "traefik.http.middlewares.ldproxy-replace-manager-path.replacepathregex.regex=/manager_ldproxy(.*)"
      - "traefik.http.middlewares.ldproxy-replace-manager-path.replacepathregex.replacement=/manager$$1"

      # SSL/https router
      - "traefik.http.routers.ldproxy_https.rule=Host(`${TRAEFIK_SSL_DOMAIN}`) && PathPrefix(`/ldproxy`)"
      - "traefik.http.routers.ldproxy_https.entrypoints=https"
      - "traefik.http.routers.ldproxy_https.tls=${TRAEFIK_USE_TLS}"
      - "traefik.http.routers.ldproxy_https.tls.certresolver=${TRAEFIK_SSL_CERT_RESOLVER}"
      - "traefik.http.routers.ldproxy_https.tls.options=my_default@file"
      - "traefik.http.routers.ldproxy_https.middlewares=secure-headers@file,ldproxy-replace-restservices-path"

      # local http router - REST services
      - "traefik.http.routers.ldproxy_http.rule=Host(`localhost`) && PathPrefix(`/ldproxy`)"
      - "traefik.http.routers.ldproxy_http.entrypoints=http"
      - "traefik.http.routers.ldproxy_http.middlewares=ldproxy-replace-restservices-path"

      # local http router - for ldproxy Manager
      - "traefik.http.routers.ldproxy_manager_http.rule=Host(`localhost`) && PathPrefix(`/manager_ldproxy`)"
      - "traefik.http.routers.ldproxy_manager_http.entrypoints=http"
      - "traefik.http.routers.ldproxy_manager_http.middlewares=ldproxy-replace-manager-path"
azahnen commented 3 years ago

@justb4 It looks like you figured out the 404 issue, as I can access https://apitestbed.geonovum.nl/ldproxy/RCE_Landschapsatlas_WFS.

The manager needs some rest endpoints relative to the manager path, so one more RegexReplace is needed. For example this works for me, manager runs at http://localhost:8080/ldproxy/manager/:

- "traefik.http.middlewares.ldproxy-replace-manager-path.replacepathregex.regex=/ldproxy/manager(.*)"
- "traefik.http.middlewares.ldproxy-replace-manager-path.replacepathregex.replacement=/manager$$1"

- "traefik.http.middlewares.ldproxy-replace-rest-path.replacepathregex.regex=/ldproxy/rest(.*)"
- "traefik.http.middlewares.ldproxy-replace-rest-path.replacepathregex.replacement=/rest$$1"

- "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.regex=/ldproxy(.*)"
- "traefik.http.middlewares.ldproxy-replace-restservices-path.replacepathregex.replacement=/rest/services$$1"

- "traefik.http.routers.ldproxy_http.rule=Host(`localhost`) && PathPrefix(`/ldproxy`)"
- "traefik.http.routers.ldproxy_http.entrypoints=http"
- "traefik.http.routers.ldproxy_http.middlewares=ldproxy-replace-manager-path,ldproxy-replace-rest-path,ldproxy-replace-restservices-path"
justb4 commented 3 years ago

@azahnen thanks again. Yes, somehow the endpoint https://apitestbed.geonovum.nl/ldproxy/RCE_Landschapsatlas_WFS suddenly worked. Ok, took the latest master DockerHub version, but I also suspect some "dangling" data in the Docker-mounted dirs. Or maybe a permission issue? Is there a way that the ldproxy container main user runs with the Docker host uid and guid ? We had to do a similar config with GeoServer, otherwise the Container runs with user tomcat causing permission issues for non-matching uid/guid.

We found another issue when opening the OAS Swagger page, see: https://github.com/Geonovum/ogc-api-testbed/issues/27 . Maybe it is a thing we can solve in our config, before opening an issue here.