mediacloud / story-indexer

The core pipeline used to ingest online news stories in the Media Cloud archive.
https://mediacloud.org
Apache License 2.0
2 stars 5 forks source link

where to capture system performance and needs documentation? #206

Closed rahulbot closed 7 months ago

rahulbot commented 10 months ago

We are starting to learn more about (1) the compute/memory/storage needs and (2) system throughput and performance. For instance, @philbudne shared that:

This kind of information seems useful to capture in a collaborative space (rather than my personal notes app), but I'm not sure where. Perhaps a wiki page on this GH repo? Better ideas?

philbudne commented 10 months ago

I don't have any brilliant thoughts on how to capture the data.

I've certainly seen that parsers running on ramos can do more than the same number on ifill, so it depends on the hardware. And the software revision (unless one monitors for performance regressions)!

rahulbot commented 7 months ago

Don't have a great solution, and numbers will likely change with hardware we have in house. Closing as archival information.