There are common neutral network architectures that we should test separately from an end-to-end model.
Testing these architectures can make diagnosing future problems much easier since we would be able to independently verify that certain network features work correctly on a stand-alone basis.
We already have verify tests for model features that we modify in some way in compiler passes.
Here are some features that we should test (feel free to add more)
Bert Transformer
Long short term memory (LSTM)
Encoder
Decoder
Autoencoders
We can also take snippets of models and create verify tests for them.
May want to take a look at ORT tests that also use snippets