Fashion MNIST update

[adam-peaston-SC]

Description

Overhaul of the Fashion MNIST example to more thoroughly demonstrate and document important elements necessary for interruptible training.

Model details

Overhaul of fashion MNIST example

Steps to reproduce any training

What settings did you use, batch size, lr, number of nodes, etc

Training results

A gist of logs, a link to tensorboard etc if relevant.

Things done

General:

• [ ] Linting (ruff, black, isort)

If adding a demo model training script:

• [ ] Atomic saving - saving is done atomically to avoid corruption
• [ ] Checkpointing - model can successfully save a checkpoint and resume.
• [ ] Completed a full run