[Ideas] User vs machine generated traces

[keegancsmith]

Observation: A lot of traces we have in Sourcegraph originate from machine actions (bots), rather than a user action. Machine traces are less valuable than what happens from a direct user action. (ie a pageload/XHR request trace is usually more valuable than a trace of a worker asking for a job). Machine traces are also higher volume. We get into situations of appdash service degradation or traces expiring soon. Dashboards / lists of traces can be overpopulated with traces originating from machines.

What can we do so we get a better experience when investigating or discovering traces from user actions?

[keegancsmith]

Random ideas:

• On the client side send to different collectors
• Extend client API to mark sampling rate of trace. if a trace isn't sampled, none of its childern are.
• Extend client API to mark priority of trace. child spans inherit priority. GC and discovery can use priority
• Disable collection of machine traces
• Ignore and just make appdash more scalable. We have influxdb for storage, not sure what we have for discovery.
• Appdash uses heuristics to keep exemplar traces for each type.