Spike: SS Dashboard should display when output files are out of sync

[MattToast]

Situation

Consider the following experiment:

from smartsim import Experiment

exp = Experiment("my-exp", launcher="slurm")
rs = exp.create_run_settings("echo", ["hello", "world"])
rs.set_nodes(1)
rs.set_tasks(1)
model = exp.create_model("say-things-model", run_settings=rs)

exp.generate(model)
exp.start(model, block=True)

model.run_settings.exe_args = ["spam", "eggs"]
exp.generate(model, overwrite=True)
exp.start(model, block=True)

As we can see, a user is trying to run a Model, mutating it and then overwriting it. This is valid SmartSim behavior as a user might save off some information from a run into an Orchestrator and then use that information to modify the input parameters/files/run settings/etc. to a model by mutating it in place

Unfortunately, the dashboard is currently unaware of the fact that models are mutable and that there output logs may change out from underneath it, and it will display the logs as though they were produced by the model. This may throw off users who are using the dashboard for debugging purposes.

Feature Request

The dashboard should have some way of indicating that the files being displayed are "out of sync" and may not be representative of the what was run given the displayed run settings/batch settings/exec args. This could potentially be done by checking the touch time of the file and the stop time of the model (at time of writing, this can be collected by reading in its respective  stop.json) and displaying a banner if the former is later than the latter.