Permit forwarded WMS requests to be cached

keithamoss commented 12 years ago

I'm going to throw this out here and see what people think of the idea.

If MapCache forwards a request to a WMS service for a single tile (256x256 / 272x272) of a layer it would be great to be able to specify that that layer be stored in a cache. I'm envisioning something like this:

   <cache name="sqlite" type="sqlite3">
      <dbname_template>/ms4w/tmp/ms_tmp/mapcache/{layer}/{grid}.db</dbname_template>
      <hitstats>false</hitstats>
   </cache>

   <service type="wms" enabled="true">
       <forwarding_rule name="first rule">
        <cache>sqlite</cache>
        <grid>g</grid>
        <append_pathinfo>true</append_pathinfo>
        <http>
          <url>http://server/mapserver?</url>
        </http>
       </forwarding_rule>
   </service>

I see the grid as either being specified manually, as here, or there being some smarts behind it by examining the SRS provided and working out the grid to use from there.

For expiry it would be great if mapcache could examine the Cache-Control or Expires headers and use that to determine when it needs to bypass the cache and request a new copy.

Thoughts and revisions more than welcome.

tbonfort commented 12 years ago

I imagine that you are trying to build a set of caches without knowing the layer names in advance right ? As for the expiry handling, please open another ticket.

keithamoss commented 12 years ago

Yup, that's correct.

I've opened https://github.com/mapserver/mapcache/issues/36 for the expiry handling side of things.

Let us know if you'd like anything more by way of explanation or detail of how we see this working.

tbonfort commented 12 years ago

mapcache already has a built in "mirroring mode" when configured with a single wms source. It will cache all LAYERS combination on getmap requests that corresponds to a configured <grid>, and will forward the rest to the original wms server.


<?xml version="1.0" encoding="UTF-8"?>

<mapcache mode="combined_mirror">
   <cache name="disk" type="disk">
      <base>/tmp</base>
      <symlink_blank/>
   </cache>

   <source name="vmap0" type="wms">
      <getmap>
         <params>
            <FORMAT>image/jpeg</FORMAT>
         </params>
      </getmap>

      <http>
         <url>http://www2.dmsolutions.ca/cgi-bin/mswms_gmap</url>
      </http>
   </source>

   <tileset name="mirror">
      <source>vmap0</source>
      <cache>disk</cache>
      <grid>WGS84</grid>
      <grid>g</grid>
      <format>JPEG</format>
      <metatile>5 5</metatile>
      <metabuffer>10</metabuffer>
      <expires>3600</expires>
   </tileset>

   <default_format>JPEG</default_format>

   <service type="wms" enabled="true">
      <full_wms>assemble</full_wms>
      <resample_mode>bilinear</resample_mode>
      <format>JPEG</format>
      <maxsize>4096</maxsize>
      <forwarding_rule name="catch all">
            <http>
               <url>http://www2.dmsolutions.ca/cgi-bin/mswms_gmap</url>
            </http>
      </forwarding_rule>
   </service>
   <service type="wmts" enabled="true"/>
   <service type="tms" enabled="true"/>
   <service type="kml" enabled="true"/>
   <service type="gmaps" enabled="true"/>
   <service type="ve" enabled="true"/>
   <service type="demo" enabled="true"/>

   <errors>report</errors>
   <lock_dir>/tmp</lock_dir>

</mapcache>

The important part is mode="combined_mirror" which can also take the value split_mirror:

In "combined_mirror" mode, a request with LAYERS=layer1,layer2 will cache a single tileset (i.e. the wms will be queried once per tile, with LAYERS=layer1,layer2).
In "split_mirror", a request with LAYERS=layer1,layer2 will result in two tilesets being cached (i.e. the wms will be queried twice, once with LAYERS=layer1, and once with LAYERS=layer2)

keithamoss commented 12 years ago

Thanks Thomas, I didn't know this feature existed. It's pretty much exactly what we're after - though the ability to mirror more than one WMS source would be great.

Manfred73 commented 8 years ago

Hi,

I am using above mapcache.xml, but when I make a request, I get an unsupported format error in the apache log file:

Request:

http://10.31.42.202:8008/mapcache/?map=/var/www/mapserver/maps/mymapfile.map&service=wms&request=GetMap&version=1.1.1&layers=wind_area&srs=EPSG:4326&BBOX=-43.6,17.5,42.8,73.6&WIDTH=400&HEIGHT=300&FORMAT=image/png&TIME=2016-01-01T18:00:00.000Z

Error:

[Wed Feb 10 12:24:50.149724 2016] [:error] [pid 15:tid 140199099021056] [client 10.31.42.169:59779] wms request for tileset wind_area returned an unsupported format:\n\nMapServer Message\n\n\nmsCGILoadMap(): Web application error. CGI variable "map" is not set.\n

When I do a request for GetCapabilities, this seems to work OK and it doesn't complain about the CGI variable 'map' is not set: http://10.31.42.202:8008/mapcache/?map=/var/www/mapserver/maps/mymapfile.map&service=wms&request=GetCapabilities

Any idea why the getMap request fails with this error?

Regards, Manfred

sgrossberndt commented 7 years ago

The definition of MAP must be part of the source->getmap->params for the mapserver to query, not part of the mapcache request. This question is not related to the issue, please create an own instead.

sgrossberndt commented 7 years ago

The mapcache mirror modes are still not part of the documentation.

MapServer / mapcache

Permit forwarded WMS requests to be cached #34