Check and modify the weights of a layer of an onnx model at runtime

[IzanCatalan]

Describe the issue

Hello, I want to check and modify the weights of a convolutional layer (or any layer) of an onnx model at runtime. This is to do it inside 'OnnxRuntime.InferenceSession.run()'. Is that possible? Did anyone do something similar?

Could be done by adding a layer before the layer I want to check and modify its weights? This new layer should do this procedure before the convolutional layer, but it would need access to the weights of the next layer.

Thanks.

Izan.

To reproduce

-

Urgency

No response

Platform

Linux

OS Version

Ubuntu 18.04

ONNX Runtime Installation

Built from Source

ONNX Runtime Version or Commit ID

onnxruntime-gpu 1.12.0

ONNX Runtime API

Python

Architecture

X64

Execution Provider

CUDA

Execution Provider Library Version

Cuda 10.2 and 11.2

[tianleiwu]

You can add a graph input for the initializer (weights), then you can dynamically set the weights using graph inputs.

[brian-at-pieces]

You can add a graph input for the initializer (weights), then you can dynamically set the weights using graph inputs.

@tianleiwu where would you write the code to dynamically set the weights? In a new node? Thanks

[tianleiwu]

@brian-pieces, here is an example: https://stackoverflow.com/questions/74139718/onnx-runtime-adding-multiple-initializers-in-python

[brian-at-pieces]

@tianleiwu wow that's exactly that I was looking for! Thanks!

[brian-at-pieces]

@tianleiwu sorry to bother again but do you have an example of this with the C++ api? I got this working easily in Python but I'm struggling with segfaults trying to do this in C++

[brian-at-pieces]

@tianleiwu sorry to bother again but do you have an example of this with the C++ api? I got this working easily in Python but I'm struggling with segfaults trying to do this in C++

Dumb issue on my side. Got it working. For anyone else intersted in a C++ impl here's what I got to work:

  // Read each initializer h5 file and store the data in a vector.
  for (const auto &entry: std::filesystem::directory_iterator(task_path)) {
      // Filter out the .DS_Store file on macOS.
      if (entry.path().filename() == ".DS_Store") {
          continue;
      }

      // Read initializer matrix from h5 file
      const std::filesystem::path &initializer_path = entry.path();
      std::string initializer_name = initializer_path.filename().string();
      H5::H5File file(initializer_path.string(), H5F_ACC_RDONLY);
      H5::DataSet dataset = file.openDataSet("data");

      // Get the dimensions of the dataset.
      H5::DataSpace dataspace = dataset.getSpace();
      hsize_t dims_out[2]; // 2D matrix
      dataspace.getSimpleExtentDims(dims_out, nullptr);

      // Create a vector to store the initializer data.
      std::vector<float> initializer_data(dims_out[0] * dims_out[1]);
      dataset.read(initializer_data.data(), H5::PredType::NATIVE_FLOAT);

      // Create an OrtValue to store the initializer data.
      std::vector<int64_t> dims{static_cast<int64_t>(dims_out[0]), static_cast<int64_t>(dims_out[1])};
      Ort::Value initializer_value = Ort::Value::CreateTensor<float>(
              memoryInfo,
              initializer_data.data(),
              initializer_data.size(),
              dims.data(),
              dims.size());

      // Add initializer to session options
      session_options.AddInitializer(initializer_name.c_str(), initializer_value);
  }

Note that this differs from the Python example b/c I'm using hdf5 files rather than numpy.