pratacosmin
commented
4 days ago To fix this bug we filter the traces by start and end time in order to reduce the number and also keep track of old traces that were process already in order to avoid reprocessing. Maybe a better solution is to use a queue this will also allow for multiple instances of pythia to process traces in parallel
RIght now Pythia re-processes all Jaeger traces by default once its up and has no memory of previously processed traces.
This leads to duplication of work and metrics becoming erroneus.
Need a watermarking solution to keep track of last processed timestamp so if the Pythia docker crashes and is brought back up it doesnt reprocess old traces