search

OpenTechFund / bypass-otf_proxy

1 stars 1 forks source link

Analyze cloudfront logs #11

Closed maxpearl closed 3 years ago

maxpearl commented 4 years ago

Cloudfront logs are in this format: 

#Fields: date time x-edge-location sc-bytes c-ip cs-method cs(Host) cs-uri-stem sc-status cs(Referer) cs(User-Agent) cs-uri-query cs(Cookie) x-edge-result-type x-edge-request-id x-host-header cs-protocol cs-bytes time-taken x-forwarded-for ssl-protocol ssl-cipher x-edge-response-result-type cs-protocol-version fle-status fle-encrypted-fields c-port time-to-first-byte x-edge-detailed-result-type sc-content-type sc-content-len sc-range-start sc-range-end
2020-05-08  12:29:19    ATL50-C1    31661   2a03:2880:21ff:9::face:b00c GET d3g0g1bncts1h.cloudfront.net    /sites/g/files/cauiqe166/files/css/css_47_ZmHwNqtxmax9LzRluJ6oSu-E9uP_Oy49Kp0O3R-Y.css  200 -   Mozilla/5.0%20(X11;%20Linux%20x86_64)%20AppleWebKit/537.36%20(KHTML,%20like%20Gecko)%20Chrome/79.0.3945.88%20Safari/537.36- -   Miss    ZyFcKkoUXPyDgKjB4N0zg1Be3GP_8fpZpc3VPQnxcy-vX7RsK1WlPw==    d3g0g1bncts1h.cloudfront.nethttps   265 0.096   -   TLSv1.2 ECDHE-RSA-AES128-GCM-SHA256 Miss    HTTP/2.0    -   -   590600.096  Miss    text/css    31124   -   -
maxpearl commented 3 years ago

Analysis of cloudfront files is implemented in https://github.com/OpenTechFund/bypass-otf_proxy/commit/31d56d5cbe0d6894785b12faa299aafac98c1a69. Files are gotten by age, if they are within the range, they are aggregated and analyzed, and a report is made in the database.