feat(catalog): Etag cache implementation into catalog - [merged] #124

Closed benoitdm-oslandia closed 1 year ago

In GitLab by @nrevelant on Oct 14, 2022, 16:24

_Merges feat/catalogcache -> develop

support for PGFS_CACHE environment variable
etag cache implemented with active and passive mode
etag exchanges between clients and pg_featureserv are now Base64 encoded
tests implemented for etag management

close #51

In GitLab by @nrevelant on Oct 24, 2022, 17:21

Problème reproduit : `fatal error: concurrent map writes

goroutine 248 [running]: github.com/CrunchyData/pg_featureserv/internal/data.CacheNaive.AddWeakEtag({0xa60940?}, {0xc001bf8968?, 0xb155c4?}, {0xa60940?, 0xc001d91950}) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/data/cache_naive.go:54 +0x46 github.com/CrunchyData/pg_featureserv/internal/data.scanFeature({0xc10420?, 0xc000487278?}, 0x0, {0xc000489c70, 0x5, 0x5}, {0xc0c5d0, 0xc000517b60}) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/data/catalog_db.go:683 +0x1f8 github.com/CrunchyData/pg_featureserv/internal/data.scanFeatures({0xc0cd10, 0xc001d91410}, {0xc10420, 0xc000487278}, 0x1?, {0xc000489c70, 0x5, 0x5}, {0xc0c5d0, 0xc000517b60}) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/data/catalog_db.go:651 +0xea github.com/CrunchyData/pg_featureserv/internal/data.readFeaturesWithArgs({0xc0cd10, 0xc001d91410}, 0xc0000eb538?, {0xc000462320, 0x99}, {0xc001f7b110, 0x1, 0x1}, 0xa?, {0xc000489c70, ...}, ...) `

In GitLab by @nrevelant on Oct 24, 2022, 17:21

En utilisant un mutex sur les écritures dans la map du cache :

`DEBU[0004] 127.0.0.1:50064 GET /collections/public.mock_a/items/1 fatal error: concurrent map read and map write

goroutine 2338 [running]: github.com/CrunchyData/pg_featureserv/internal/data.CacheNaive.ContainsWeakEtag({0x1?}, {0xc00020b831?, 0x0?}) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/data/cache_naive.go:52 +0xc8 github.com/CrunchyData/pg_featureserv/internal/data.(*catalogDB).CheckStrongEtags(0x1077380, {0xc000477ce0?, 0x1, 0x1?}) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/data/catalog_db.go:528 +0x76 github.com/CrunchyData/pg_featureserv/internal/service.handleItem({0xc0b000, 0xc00034f1d0}, 0xc001bf7e00) /home/NRevelant/code/oslandia/geoplateforme-ign-pg-featureserv/internal/service/handler.go:637 +0x2a5 `

In GitLab by @nrevelant on Oct 24, 2022, 17:28

added 1 commit

2f544058 - a mutex is now used to fix concurrent accesses to the cache

Compare with previous version

In GitLab by @nrevelant on Oct 24, 2022, 17:42

Un mutex est maintenant utilisé à la fois pour les accès en lecture et en écriture sur la map.

In GitLab by @lowzonenose on Oct 25, 2022, 09:41

Article intéressant sur les performances du cache :

In GitLab by @azarz on Oct 27, 2022, 09:34

Bug résolu du coup @nrevelant @lowzonenose ?

In GitLab by @nrevelant on Oct 28, 2022, 17:32

added 3 commits

1b61d9a1 - route /etags/decodestrong/{etag} added
81cd1e40 - IsCacheEnabled() function added to the etag cache
05c57ba4 - last-modified response header + openapi page update

Compare with previous version

In GitLab by @nrevelant on Oct 30, 2022, 09:28

added 1 commit

42fc0f4f - go 1.13 fix

Compare with previous version

Bien vu ! Est ce que ca résoud le pbm ?

In GitLab by @nrevelant on Nov 2, 2022, 15:21

De ce que j'ai testé en local, ça semble fonctionner :

_$GOPATH/bin/go-wrk -c 10 -H Accept:json -H Content-Type:application/json -H If-None-Match:"Im1vY2tfYS00MzI2LWpzb24tMTcyMzIyMzAxOCI=" http://localhost:9000/collections/public.mock_a/items/1_

received status code 304 from map[Content-Encoding:[gzip] Date:[Wed, 02 Nov 2022 14:16:03 GMT] Vary:[Accept-Encoding]] content  &{GET http://localhost:9000/collections/public.mock_a/items/1?crs=4326 HTTP/1.1 1 1 map[Accept:[json] Content-Type:[application/json] If-None-Match:[Im1vY2tfYS00MzI2LWpzb24tMTcyMzIyMzAxOCI=] User-Agent:[go-wrk]] <nil> <nil> 0 [] false localhost:9000 map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc00001a088}
received status code 304 from map[Content-Encoding:[gzip] Date:[Wed, 02 Nov 2022 14:16:03 GMT] Vary:[Accept-Encoding]] content  &{GET http://localhost:9000/collections/public.mock_a/items/1?crs=4326 HTTP/1.1 1 1 map[Accept:[json] Content-Type:[application/json] If-None-Match:[Im1vY2tfYS00MzI2LWpzb24tMTcyMzIyMzAxOCI=] User-Agent:[go-wrk]] <nil> <nil> 0 [] false localhost:9000 map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc00001a088}
received status code 304 from map[Content-Encoding:[gzip] Date:[Wed, 02 Nov 2022 14:16:03 GMT] Vary:[Accept-Encoding]] content  &{GET http://localhost:9000/collections/public.mock_a/items/1?crs=4326 HTTP/1.1 1 1 map[Accept:[json] Content-Type:[application/json] If-None-Match:[Im1vY2tfYS00MzI2LWpzb24tMTcyMzIyMzAxOCI=] User-Agent:[go-wrk]] <nil> <nil> 0 [] false localhost:9000 map[] map[] <nil> map[]   <nil> <nil> <nil> 0xc00001a088}

Ok nous en parlerons demain (jeudi 03) !

resolved all threads

In GitLab by @azarz on Nov 2, 2022, 17:15

(demain matin ! :wink: )

    // The date format here conforms to HTTP Date, always in GMT time zone like "Wed, 21 Oct 2015 07:28:00 GMT"
    // -> https://www.rfc-editor.org/rfc/rfc7231#section-7.1.1.1

    location, _ := time.LoadLocation("GMT")
    current_Time := time.Now().In(location)

        return current_time.Format(time.RFC1123)

In GitLab by @nrevelant on Nov 3, 2022, 10:48

Commented on internal/api/net.go line 149

changed this line in version 14 of the diff

In GitLab by @nrevelant on Nov 3, 2022, 10:48

added 1 commit