Closed parkzhou0527 closed 1 year ago
@kaleocheng
actually there are two issues:
for the first one in a long-term we'd better to introduce a db storage for pipelines( e.g. redis/postgres), but it's another topic, before that we can apply a quick fix on the current file based storage:
X
items sorted by created or modified timeX
for pipelines cli. let's take Park's suggestion name --log-limits
for the second one I will create another ticket to determine.
for a quick mitigation, see #134.
current run folder names are just uuid, lacking metadata, causing hard to sort without further looking into the status JSON files.
a further mitigation could be writing an additional metadata status file for each pipeline, including history runs, with their creation date etc. this would only happen when create new runs. may still easier than totally switching to DBMS.
Background
We have a pipeline installation running for almost 3 years (in one of our customers); so build history accumulated for quite a large number of 1000+;
And our developers find that it's very very slow to open the pipeline homepage, and the chrome memory was consumed too much.
After a quick debug, we find it's was caused by the pipeline history, the homepage tries to load all the history at once.
So, we cleanup the history to less than 200; everything runs so quickly again.
Proposal
Pipeline should support a global flag --log-limits (which might have it's own default value, say 200); this global flag was respected by the pipeline.
We do not want to too much complicated policy to control the history log reservation. For our daily development cycle, the latest 5 build log (I means most of the times) is enough to let us debug errors.