"'Sequential' object has no attribute 'named_modules'" error with Keras Sequential model

[vvolhejn]

Hi, I'm trying to learn how to use the library. I tried to follow the guide for Keras and wrote the following code:

import tensorflow as tf

resolution = 14
channels = 16
expansion = 6

model = tf.keras.Sequential(
        [
            tf.keras.Input(shape=(resolution, resolution, channels)),
            # Expand with a pointwise 1x1 convolution.
            tf.keras.layers.Conv2D(
                expansion * channels,
                kernel_size=1,
                padding="same",
                use_bias=False,
                activation="relu6",
            ),
            # Depthwise 3x3 convolution.
            tf.keras.layers.DepthwiseConv2D(
                # no channels argument since this is a depthwise conv
                kernel_size=3,
                padding="same",
                use_bias=False,
                activation="relu6",
            ),
            # Project with a pointwise 1x1 convolution.
            tf.keras.layers.Conv2D(
                channels,
                kernel_size=1,
                padding="same",
                use_bias=False,
                activation=None,
            ),
        ]
)

optimizer = None
num_train_batches = 10

from sparseml.pytorch.optim import ScheduledModifierManager
manager = ScheduledModifierManager.from_yaml("./sparseml-recipe.yaml")

model, optimizer, callbacks = manager.modify(
    model, optimizer, steps_per_epoch=num_train_batches
)

which crashes with

AttributeError                            Traceback (most recent call last)
Input In [24], in <cell line: 42>()
     39 from sparseml.pytorch.optim import ScheduledModifierManager
     40 manager = ScheduledModifierManager.from_yaml("./sparseml-recipe.yaml")
---> 42 model, optimizer, callbacks = manager.modify(
     43     model, optimizer, steps_per_epoch=num_train_batches
     44 )

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/optim/manager.py:498, in ScheduledModifierManager.modify(self, module, optimizer, steps_per_epoch, wrap_optim, epoch, allow_parallel_module, **kwargs)
    495         module = module.module  # unwrap parallel module
    497 if not self.initialized:
--> 498     self.initialize(module, epoch, **kwargs)
    500 if wrap_optim is None:
    501     wrap_optim = optimizer

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/optim/manager.py:430, in ScheduledModifierManager.initialize(self, module, epoch, loggers, **kwargs)
    426 if mod.initialized:
    427     # check in case modifier was initialized from apply_structure
    428     continue
--> 430 mod.initialize(module, epoch, loggers, **kwargs)

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/sparsification/pruning/modifier_pruning_base.py:294, in BasePruningModifier.initialize(self, module, epoch, loggers, **kwargs)
    283 """
    284 Grab the params and apply if epoch in range to control pruning for.
    285 
   (...)
    291     for individual modifiers.
    292 """
    293 super().initialize(module, epoch, loggers, **kwargs)
--> 294 named_layers_and_params = self._create_named_layers_and_params(module)
    295 layers = [nlp.layer for nlp in named_layers_and_params]
    296 param_names = [nlp.param_name for nlp in named_layers_and_params]

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/sparsification/pruning/modifier_pruning_base.py:774, in BaseGradualPruningModifier._create_named_layers_and_params(self, module)
    772 def _create_named_layers_and_params(self, module: Module) -> List[NamedLayerParam]:
    773     if isinstance(self._final_sparsity_orig, float):
--> 774         return super()._create_named_layers_and_params(module)
    776     # update NamedLayerParam values to account for final sparsities dict
    778     final_sparsities = []

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/sparsification/pruning/modifier_pruning_base.py:529, in BasePruningModifier._create_named_layers_and_params(self, module)
    526 else:
    527     param_names = self._params
--> 529 return get_named_layers_and_params_by_regex(
    530     module,
    531     param_names,
    532     params_strict=True,
    533 )

File ~/venv3.8/lib/python3.8/site-packages/sparseml/pytorch/utils/helpers.py:861, in get_named_layers_and_params_by_regex(module, param_names, params_strict)
    859 named_layers_and_params = []
    860 found_param_names = []
--> 861 for layer_name, layer in module.named_modules():
    862     for param_name, param in layer.named_parameters():
    863         if "." in param_name:  # skip parameters of nested layers

AttributeError: 'Sequential' object has no attribute 'named_modules'

I know I probably shouldn't be calling the function with optimizer=None but this is unrelated, since passing a PyTorch Sequential model works (and then crashes later because optimizer=None).

The recipe is a slightly modified version of the example from the docs:

version: 0.1.0
modifiers:
    - !EpochRangeModifier
        start_epoch: 0.0
        end_epoch: 2.0

    - !LearningRateModifier
        start_epoch: 0
        end_epoch: -1.0
        update_frequency: -1.0
        init_lr: 0.005
        lr_class: MultiStepLR
        lr_kwargs: {'milestones': [43, 60], 'gamma': 0.1}

    - !GMPruningModifier
        start_epoch: 0
        end_epoch: 2.0
        update_frequency: 1.0
        init_sparsity: 0.05
        final_sparsity: 0.85
        mask_type: unstructured
#        params: ['sections.0.0.conv1.weight', 'sections.0.0.conv2.weight', 'sections.0.0.conv3.weight']
        params:  __ALL__

Environment Include all relevant environment information:

1. OS [e.g. Ubuntu 18.04]: Debian GNU/Linux 11 (bullseye)
2. Python version [e.g. 3.7]: 3.8.12
3. DeepSparse version or commit hash [e.g. 0.1.0, f7245c8]: 0.12.2
4. ML framework version(s) [e.g. torch 1.7.1]: TensorFlow 2.8.0
5. Other Python package versions [e.g. SparseML, Sparsify, numpy, ONNX]: tf2onnx 1.11.1
6. CPU info - output of deepsparse/src/deepsparse/arch.bin or output of cpu_architecture(): {'vendor': 'GenuineIntel', 'isa': 'avx512', 'vnni': False, 'num_sockets': 1, 'available_sockets': 1, 'cores_per_socket': 1, 'available_cores_per_socket': 1, 'threads_per_core': 2, 'available_threads_per_core': 2, 'L1_instruction_cache_size': 32768, 'L1_data_cache_size': 32768, 'L2_cache_size': 1048576, 'L3_cache_size': 40370176}

[bfineran]

Hi @vvolhejn this error is occurring because you are using a keras model (tf.keras.Sequential) with our pytorch manager (from sparseml.pytorch.optim import ScheduledModifierManager).

We do have some support for sparsification with keras (see example notebook here), however the best supported path would be to port the model to pytorch and use the pytorch SchedueldModifierManager (see example notebook here).

Hope this helps, let me know if you have any more questions!