search

mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.
https://mediacloud.org
Apache License 2.0
2 stars 5 forks source link

Indexer stack prefixes can collide in stats #305

Closed pgulley closed 5 months ago

pgulley commented 5 months ago

Issue raised by @thepsalmist in slack- Stats namespaces in grafana/statsd for story-indexer are determined by the stack-type prefix of a deployment ("csv-", "rss-", etc) - if more than one of a given stack type is deployed at a time (in the context of historical re-ingest for example) the stats will be collected in the same statsd location, making observation of both stacks difficult. Would it be simple to introduce a unique infix (ie rss-1-*, per @kilemensi) to prevent this in the future?

philbudne commented 5 months ago

Would it be simple to introduce a unique infix (ie rss-1-*, per @kilemensi) to prevent this in the future?

IMO: No. Since stacks of the same name/type would be, by definition, deployed on different swarm clusters, there is no automagic way to determine if there are others of the same name/type active, nor a way to deterministically enumerate them (other than to include the name of the current cluster leader).

At some point I did consider adding a command line option to deploy.sh to allow a manual override for some additional parameter (I forget if it was stats reporting name, or archive file name), but it struck me as "too many knobs" and adding clutter for situations that are (or should be) to say the least "out of the ordinary" and "to be avoided"

I've stated that I think I could improve tqfetcher's rates, which would have the long-term benefit/defense of dealing better of being better able to catch up from downtime, as well as speed up the 2022 rss ingest, but that didn't seem to get any traction.

And while I'm not a fan of running THREE tqfetchers (production and two rss stacks), if that's determined to be acceptable in this instance, my suggestion is to make a copy of deploy.sh (editing it in place will trigger it's test that the repo be "clean" -- checked in and pushed), and crock it for the hopefully RARE occasions when more than one stack of the same type/name is run.

rahulbot commented 5 months ago

Right now, because of early-2022 data messiness we're seeing, I'm leaning more towards careful than fast.

pgulley commented 5 months ago

Ok, heard! Just planning to avoid this circumstance sounds a lot more straightforward than trying to address this. Thanks for the feedback!