search

curl / curl-www

The curl.se website
https://curl.se/
MIT License
101 stars 88 forks source link

Enable web site log analysis #346

Open dfandrich opened 2 months ago

dfandrich commented 2 months ago

Right now, the statistics that Fastly gives on web site accesses is extremely broad. It would be useful at times to have more details on which URLs are being hit most, and which ones are causing errors. Logs on the origin server don't provide many details because of the caching. This would be very useful in cases like #344 for example, and it might have spotted #272 for us, as well as just giving general estimates on site usage.

Fastly has just announced their own analysis system that is in beta: https://www.fastly.com/blog/introducing-log-manager-and-insights-now-in-beta It might be worth asking them if we could try it out. With curl's traffic levels, even running it for an hour would likely provide some useful insights.

Alternately, running a simple access log analysis tool like https://www.awstats.org/ on the origin server would be better than nothing.

If this is done, privacy of the reports needs to be considered, which probably means most reports can't be publicly visible.

vszakats commented 1 month ago

Assuming it doesn't involve scripts / cookies / tracking pixels on the client-side — IOW it's based on server logs —, this seems useful to get a picture of what's consumed from the website.