search

openeduhub / metalookup

Provide metadata about domains w.r.t accessibility, licencing, adds, etc.
GNU General Public License v3.0
5 stars 0 forks source link

Clean up logging and use some technique to track logs for individual concurrent requests #122

Open MRuecklCC opened 2 years ago

MRuecklCC commented 2 years ago

Use an approach as described here to improve logging: https://medium.com/gradiant-talks/identifying-fastapi-requests-in-logs-bac3284a6aa

Also we currently seem to have different logging settings going on at the same time:

extractor_1   | 2022-06-22 12:09:52,109 INFO metalookup.core.metadata_manager Extracted LicenceExtractor in  1.70s.
extractor_1   | 2022-06-22 12:09:52,155 INFO metalookup.features.adblock_based Found 1 links that should be blocked according to ad-block rules in  0.55s
extractor_1   | 2022-06-22 12:09:52,155 INFO metalookup.core.metadata_manager Extracted FanboyAnnoyance in  2.27s.
extractor_1   | 2022-06-22 12:09:52,328 INFO metalookup.features.adblock_based Found 0 links that should be blocked according to ad-block rules in  0.26s
extractor_1   | 2022-06-22 12:09:52,328 INFO metalookup.core.metadata_manager Extracted EasylistAdult in  2.44s.
extractor_1   | 2022-06-22 12:09:52,588 INFO metalookup.core.metadata_manager Extracted Accessibility in 12.92s.
extractor_1   | INFO:     127.0.0.1:54056 - "POST /extract?extra=true HTTP/1.1" 200 OK
extractor_1   | 2022-06-22 12:09:52,592 INFO root Warming up https://www.bpb.de/geschichte/zeitgeschichte/geschichte-im-fluss/142042/der-geopferte-rhein with task 5
extractor_1   | 2022-06-22 12:09:52,600 INFO metalookup.app.api Received request for https://www.bpb.de/geschichte/zeitgeschichte/geschichte-im-fluss/142042/der-geopferte-rhein
extractor_1   | 2022-06-22 12:09:54,239 INFO metalookup.features.adblock_based Found 0 links that should be blocked according to ad-block rules in   2.6s
extractor_1   | 2022-06-22 12:09:54,239 INFO metalookup.core.metadata_manager Extracted Advertisement in  4.35s.
extractor_1   | 2022-06-22 12:09:57,986 ERROR metalookup.core.metadata_manager Failed to extract accessibility

The subprocess for cache warmup probably uses yet another logging configuration.

MRuecklCC commented 2 years ago

Alternative howto: https://github.com/snok/asgi-correlation-id