search

neuralmagic / sparseml

Libraries for applying sparsification recipes to neural networks with a few lines of code, enabling faster and smaller models
Apache License 2.0
2.08k stars 148 forks source link

Update SparseGPT updates to respect base model's sparsity #2222

Open abhinavnmagic opened 7 months ago

abhinavnmagic commented 7 months ago

Motivation

If we run SparseGPT on a base model at some sparsity, the sparsity mask after SparseGPT could be very different from the initial one. In other words, SparseGPT doesn't preserve the initial sparsities of the base model. This PR fixes that. If preserve_sparsity_mask=True in the recipe, the sparsity mask of the base model being pruned is preserve. This can be useful for multiple scenarios: one-shot from 50->70% sparsity, going from unstructured to structured sparsity or vice versa, etc.

Testing

The implementation was tested by applying 2:4 sparsity on a 70% sparse model. The resulting model was checked if it preserves the 70% sparsity masks (ie. weights which were pruned out in the base model remain pruned), and if it has 2:4 sparsity pattern.

Recipe

test_stage:
  obcq_modifiers:
    SparseGPTModifier:
      sparsity: 0.5
      block_size: 128
      sequential_update: false
      quantize: false
      percdamp: 0.01
      mask_structure: "2:4"
      targets: ["re:model.layers.\\d*$"]
      preserve_sparsity_mask: true