Weight Stripping

[hgaspar]

Enable creating engines (currently, MXR files, eventually perhaps dynamic objects) without embedding the weights in the engine.

Use cases: (1) Support compilation for various batch sizes without duplicating the weights. (2) Support multiple execution configurations with different quantization options (including mixed precision), without necessarily having to embed the weights in all the created engines. (3) multi-GPU execution may benefit from this also, especially when it comes to creating multiple multiGPU execution configurations (partitions, execution schedules)

Technical considerations: How do we treat literals? Perhaps we need to have the MXR files contain the steps required to recreate the literals from the weights' file, and that may require a new type ( finalized lliterals vs future literal or meta-literal) 

[eddieliao]

Looking to work on this as an extension of weight streaming; do we have a specific format already for a weights file or is that something that needs to be decided?

[eddieliao]

List of items for basic proof of concept:

• [x] Remove weights from .mxr files
• [x] Save weights to separate file(s)
• [x] Load in weights during runtime

[eddieliao]

Replaced current literals with a fetch_literals dummy instruction that contains no data. Greatly reduces .mxr size, although still need to investigate why read of the model fails.

[eddieliao]

Added the ability to write and save weights in the strip_weights pass. Need to figure out how to pass output location to the pass (remove hard-coded location).

[simberg-amd]

Fixed issue with writing weights and added test that successfully reads weights from file and adds weight back to MXR file

[simberg-amd]

• Might look into taking the extra pass out of target.cpp and moving things into write_literals. This way instead of adding the literal first and then removing it, just add the dummy instruction in there and save the weights to the file if want to strip_weights. This would make it so no time is added during compilation and don't have to do MIGRAPHX_COPY_LITERALS{}

• Also could put in a check to see if weight file is already available for model compiling, and then don't write weights again for new compiled MXR file.

[simberg-amd]

Finished above, going to look into different quantization options.