Syntax Problems and TR decomposition alone without NN Layers

[mhelal]

Hi

I am trying to implement your MNIST TR classifier:

# Import Necessary Pytorch Modules
import torch
import torch.nn as nn
from torch import Tensor
from tednet.tnn import tensor_ring as tr

# A Simple MNIST Classifier based on Tensor Ring.
class TRClassifier (nn.Module) :
    def init (self):
        super (TRClassifier, self).init()

        # Define a Tensor Ring Convolutional Layer
        self.trcnn = tr.TRConv2D ([1] , [4, 5] , [ 6, 6, 6, 6], 3)
        # Define a Tensor Ring Fully−Connected Layer
        self.trfc = tr.TRLinear ([20, 26, 26],[10], [6, 6, 6, 6])

    def forward (self, inputs: Tensor ) -> Tensor :
        # Call TRConv2D to process inputs
        out = self.trcnn (inputs)
        out = torch.relu (out)
        out = out.view (inputs.size (0), -1)

        # Call TRLinear to classify the features
        out = self.trfc (out)
        return out

First, the syntax for passing inputs and outputs is not clear in any of your tutorials. I tried the following without the output yet:

TRCls = TRClassifier( X_train)
TRCls

and I am getting the error: TypeError: init() takes 1 positional argument but 2 were given

Second, I would like to do a TR decomposition of a ndarray, such as the TT decomposition syntax elsewhere:

from tensorly.contrib.decomposition import tensor_train_cross
factors = tensor_train_cross(T1, rank)
reconstruction_t = np.round(tt_to_tensor(factors), decimals=10)
reconstruction_t
TT_RMSE = math.sqrt(np.square(np.subtract(T1,reconstruction_t)).mean() )
print ("TT   for rank " + str(rank) + " RMSE = ", TT_RMSE)

or

import t3f
a_tt = t3f.to_tt_tensor(T1, max_tt_rank=2)
reconstruction_t = t3f.full(a_tt)
T3f_TT_RMSE = math.sqrt(np.square(np.subtract(T1,reconstruction_t)).mean() )
print ("T3F TT   for rank " + str(rank) + " RMSE = ", T3f_TT_RMSE)

I could not find in your APIs something similar to this. Can you please advise, if you have a TR decomposition or any other Python package provided one?

Third, it is not clear to me from the tutorials, how a linear layer takes an input shape, and an output shape, and then a specific rank for what? If you provide an example with some explanations, this will be clearer.

rank=[2, 2]
model = tr.TRLinear(T1.shape, [2, 2], ranks=rank)
tn_type = "tr"
model.tn_info["type"] = tn_type

thank you,

Manal

[MorinW]

Hi Manal,

Thanks for your interest in our package.

To answer your first question, 'TRClassifier' is a Python class, and your 'TRClassifier' has no parameters declared.

You may use:

TRCls = TRClassifier()
Output = TRCls(X_train)

The neural network you defined is called "TRCls," which is distinct from "TRClassifier()."

Regarding your second inquiry, our package does not contain the decomposition implementation. Our package only supports the TR formats for neural network design. However, as far as we know, there is not a python package that supports the tensor ring decomposition algorithm (TR-svd) well. You can try the Matlab package at https://qibinzhao.github.io. You might also find the https://github.com/KhrulkovV/tt-pytorch useful.

For your third question, this is the term used in neural networks. Input shape refers to the input data's feature shape, and output shape means the shape of the feature map after processing.

If you have other questions, feel free to contact us.

Best regards,

Morin

[Johnzer1]

Since this package does not include tensor decomposition algorithms, how can I perform tensor decomposition on layers of neural networks?

[MorinW]

Since this package does not include tensor decomposition algorithms, how can I perform tensor decomposition on layers of neural networks?

You can utilize TensorLy for tensor decomposition of the parameter weights and subsequently reload the corresponding tensors into our specified format~(like TT-Layer). Our package exclusively implements these formats, and all our experiments are conducted with random initialization.