Add support for purging layers

[khumps]

I am running to weird oddities around trying to purge our cache in S3. I am trying to run it using the command: docker run -e TILE_CACHE_S3_BUCKET -e TILE_CACHE_S3_KEY -e TILE_CACHE_S3_SECRET -e PG_HOST -e PG_ROOT_PASS -e PG_ROOT_PASS -v ${PWD}/tegola-config-dev.toml:/config.toml gospatial/tegola:latest --config /config.toml cache purge --map gw --max-zoom 3 Relevant config file sections:

[cache]
type = "s3"
max_zoom = 7
bucket = "${TILE_CACHE_S3_BUCKET}"
basepath = "dev/"
region = "us-west-2"
aws_access_key_id = "${TILE_CACHE_S3_KEY}"
aws_secret_access_key = "${TILE_CACHE_S3_SECRET}"

[[maps]]
name = "gw"

I get output from the program (abbreviated but it goes all the way from zoom 0-3 as requested)

2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/6/3)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/6/4)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/6/5)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/6/6)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/6/7)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/0)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/1)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/2)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/3)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/4)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/5)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/6)
2021-02-19 16:06:34 [INFO] worker.go:103: purging map (gw) tile (3/7/7)
2021-02-19 16:06:34 [INFO] cache.go:204: waiting for workers to finish up
2021-02-19 16:06:34 [INFO] cache.go:214: all workers are done
2021-02-19 16:06:34 [INFO] provider.go:256: cleaning up providers
2021/02/19 16:06:34 postgis.go:682: cleaning up postgis providers

However, not a single file was removed from the S3 bucket. Is there something gravely wrong with how I am running this or is this a bug? I have verified that the IAM credentials I am passing do in fact have full R/W access to the bucket I am using.

[ARolek]

That all looks correct to me. Can you double check the region?

[khumps]

Just checked in the AWS console and the region is correct, us-west-2

This is what the bucket looks like after the purge happens

[ARolek]

@khumps I'm going to have to dig into this some more as I'm fairly certain this functionality is confirmed working (but maybe something changed). During the init process for the s3 cache provider we issue a purge command for a test file we upload. This makes sure we have permissions. The actual Purge command is quite simple:

https://github.com/go-spatial/tegola/blob/7f0e7e66089ead2823a65c50baa995b5c0d9a487/cache/s3/s3.go#L330

I wonder if the issue is with the actual worker routine. I believe @gdey refactored that some time ago.

[khumps]

@ARolek, after some more testing. I am realizing that the issue is due to the fact that our cache data was coming from calls being made to individual layers of the map. dev is the basepath, gw is the map name, conduit is a layer then inside of that you have Z,X,Y. However, the purge command only knows how to delete tiles rendered for the entire map (stored at basepath/mapname/Z/X/Y). This was confirmed by pulling the repo and adding some command line output to the Purge command

Is it possible to add command-line parameters that allow purging of individual layers as well?

[ARolek]

@khumps thanks for the follow-up and investigative work. This does make sense though as I don't think we ever implemented layer-based purging. I would need to dig into the code but I'm fairly sure that layer-based purging could be added. Would you want to take a stab at doing the implementation? We can help with the code review and navigating the codebase.

[khumps]

@ARolek I could definitely take a shot at it. I am pretty tied up right now but I can take a look when I have some spare cycles.

[ARolek]

@khumps I'm in a similar situation, but we will figure out how to get this done. I'm going to adjust the title of this issue to reflect what you discovered.