A better way to handle remote content-type, fix etag

[n0vad3v]

• This PR will have a better way to determine content-type when using remote mode
  • Instead of relying on content-type header returned from remote, it will download the remote content and check for MIME type, this will solve problem in issue https://github.com/webp-sh/webp_server_go/issues/215 and https://github.com/webp-sh/webp_server_go/issues/213, however, risks at when requesting something big, it will waste some traffic to fetch it. (Consider remote has /path/to/image.jpg and /path/to/10GB.bin and requests are made to the latter one).
  • Maybe we can add more checks here to mitigate some risks.
• Added back etag value used for caching remote original image, which is being removed in https://github.com/webp-sh/webp_server_go/pull/194
  • Now when fetching remote image, contents in remote-raw will be Sha1Path(reqURIwithQuery) + "-etag-" + Sha1Path(etagValue), example: /home/webp_server/remote-raw/378e740ca56144b7587f3af9debeee544842879a-etag-123e740ca56333b7587f3af9debeee5448428123.
  • As on Linux the maximum length for a file name is 255 bytes. The maximum combined length of both the file name and path name is 4096 bytes. This shouldn't give us any problems.

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[codecov[bot]]

Codecov Report

Merging #216 (f3e9976) into master (9efead2) will decrease coverage by 3.24%. The diff coverage is 77.81%.

@@            Coverage Diff             @@
##           master     #216      +/-   ##
==========================================
- Coverage   83.57%   80.34%   -3.24%     
==========================================
  Files           6        6              
  Lines         542      636      +94     
==========================================
+ Hits          453      511      +58     
- Misses         61       85      +24     
- Partials       28       40      +12     

Impacted Files	Coverage Δ
encoder.go	67.03% <68.88%> (-10.11%)	:arrow_down:
helper.go	82.35% <74.60%> (-6.42%)	:arrow_down:
router.go	93.16% <95.00%> (+3.47%)	:arrow_up:
config.go	100.00% <100.00%> (ø)
prefetch.go	100.00% <100.00%> (ø)
webp-server.go	79.20% <100.00%> (+2.03%)	:arrow_up:

:mega: We’re building smart automated test selection to slash your CI/CD build times. Learn more

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[bugfest]

HI @n0vad3v, I've being thinking about how to improve the cache; here you're using go-cache for the in-memory locking but I was wondering if we could take the opportunity to use it thru the gocache wrapper instead: https://github.com/eko/gocache#memory-using-go-cache

That way we could extend the server to use other caching services in the future more easily.

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[n0vad3v]

@bugfest

I was wondering if we could take the opportunity to use it thru the gocache wrapper instead

Agree with you, however I'm keep using this in this PR right now as we only have this one item to cache and trying to fix the problem in issue: https://github.com/webp-sh/webp_server_go/issues/213

I've also edited https://github.com/webp-sh/webp_server_go/issues/217 to add this potential feature.

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)

[github-actions[bot]]

ghcr.io/webp-sh/webp_server_go (debian 11.7)
============================================
Total: 0 (HIGH: 0, CRITICAL: 0)