mgdigital
commented
10 months ago - Clean up logging and reduce verbosity of debug level
- Add optional logging to rotated files
- Add prometheus metrics endpoint
- Add pprof profiling endpoint
- Add asynqmon monitoring web UI
- Add example configurations for Grafana, Prometheus, Loki, Pyroscope
- Dial down the concurrency/buffering in parts of the crawler pipeline which was a bit too high by default
- Some refactoring in places to decouple things and avoid munging implementations with the prometheus collector and logging