Create notebook with linear algebra basics

[AndreWeiner]

The idea is to create a notebook that briefly introduces important concepts from linear algebra that are essential to understand dimensionality reduction algorithms like proper orthogonal decomposition (POD) or dynamic mode decomposition (DMD). Potential topics are (list will be updated):

• [x] tensors (n-dimensional arrays)
• [x] dot product
• [x] cross product
• [x] outer product
• [x] matrix multiplication
• [x] identity matrix
• [x] orthogonal matrix
• [x] orthonormal matrix
• [x] Hermitian matrix
• [x] Unitary matrix
• [x] Eigenvalue decomposition
• [x] Singular value decomposition

Each description should contain the following three elements:

1. mathematical notation
2. implementation using PyTorch
3. visualization (using Matplotlib)

[MehdiMak96]

I have two questions: -does the notation have to be detailed? -do you have any links or videos that could help me understand implementation using Pytorch? .

[AndreWeiner]

1) the notation should be detailed 2) your main PyTorch resources are the PyTorch Python API and the PyTorch tutorial collection

[MehdiMak96]

I'm getting the following error message when I try to add ((for ax in axarr:) to the code :

"name 'axarr' is not defined"

even though I did define it:"fig, axarr = plt.subplots(1, 3, figsize=(3*MATRIX_WIDTH, height))"

[AndreWeiner]

Hi Mehdi, not sure where the problem is. Two things to check:

• spelling
• the line fig, axarr = plt.subplots(...) must come before for ax in axarr:

If that doesn't work, post the code here. Best, Andre

[MehdiMak96]

i tried to but it in different spots but it didn't work ,

def plot_matrix_multiplication_as_heatmap(A: pt.Tensor,B: pt.Tensor,C: pt.Tensor, annotate: bool=True): n_rows, n_cols = A.size() n_rows, n_cols = B.size() height = MATRIX_WIDTH n_rows / n_cols fig, axarr = plt.subplots(1, 3, figsize=(3MATRIX_WIDTH, height)) axarr[0].imshow(A) axarr[1].imshow(B) axarr[2].imshow(C) if annotate: for row in range(n_rows): for col in range(n_cols): ax.text(row, col, "{:2.2f}".format(A[col, row]), ha="center", va="center", color="w", fontsize=ANNO_SIZE) for ax in axarr: ax.set_xticks(pt.arange(n_cols+1) - 0.5, minor=True) ax.set_yticks(pt.arange(n_rows+1) - 0.5, minor=True) ax.grid(which="minor", color="w", linestyle='-', linewidth=1) ax.tick_params(which="minor", bottom=False, left=False) ax.set_xticklabels([]) ax.set_yticklabels([])

[AndreWeiner]

Hi Mehdi, please, paste the code using the code environment:

def my_function: pass

Without indentation, it is hard to spot where the error might be.

[MehdiMak96]

def plot_matrix_multiplication_as_heatmap(A: pt.Tensor,B: pt.Tensor,C: pt.Tensor, annotate: bool=True):
    n_rows, n_cols = A.size()
    n_rows, n_cols = B.size()
    height = MATRIX_WIDTH * n_rows / n_cols
    fig, axarr = plt.subplots(1, 3, figsize=(3*MATRIX_WIDTH, height))
    axarr[0].imshow(A)
    axarr[1].imshow(B)
    axarr[2].imshow(C)
    if annotate:
         for row in range(n_rows):
            for col in range(n_cols):
                ax.text(row, col, "{:2.2f}".format(A[col, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE)
for ax in axarr:
    ax.set_xticks(pt.arange(n_cols+1) - 0.5, minor=True)
    ax.set_yticks(pt.arange(n_rows+1) - 0.5, minor=True)
    ax.grid(which="minor", color="w", linestyle='-', linewidth=1)
    ax.tick_params(which="minor", bottom=False, left=False)
    ax.set_xticklabels([])
    ax.set_yticklabels([])
    ax.set_xlabel("{:d} columns".format(n_cols), fontsize=LABEL_SIZE)
    ax.set_ylabel("{:d} rows".format(n_rows), fontsize=LABEL_SIZE)

[AndreWeiner]

Hi Mehdi, your for-loop is not part of the plot_matrix_mutli... function. To fix the problem, indent the for-loop such that it is at the same level as if_annotate: Best, Andre

[MehdiMak96]

I already tried this and for some reason I'm getting the values of the first matrix annotated and viewed in the other two matrices.

def plot_matrix_multiplication_as_heatmap(A: pt.Tensor, B: pt.Tensor, C: pt.Tensor, annotate: bool=True):
    n_rows, n_cols = A.size()
    n_rows, n_cols = B.size()
    n_rows, n_cols = C.size()
    height = MATRIX_WIDTH * n_rows / n_cols
    fig, axarr = plt.subplots(1, 3, figsize=(3*MATRIX_WIDTH, height))
    axarr[0].imshow(A)
    axarr[1].imshow(B)
    axarr[2].imshow(C)
    if annotate:
      for ax in axarr:
         for row in range(n_rows):
            for col in range(n_cols):
                ax.text(row, col, "{:2.2f}".format(A[col, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE)
    ax.set_xticks(pt.arange(n_cols+1) - 0.5, minor=True)
    ax.set_yticks(pt.arange(n_rows+1) - 0.5, minor=True)
    ax.grid(which="minor", color="w", linestyle='-', linewidth=1)
    ax.tick_params(which="minor", bottom=False, left=False)
    ax.set_xticklabels([])
    ax.set_yticklabels([])
    ax.set_xlabel("{:d} columns".format(n_cols), fontsize=LABEL_SIZE)
    ax.set_ylabel("{:d} rows".format(n_rows), fontsize=LABEL_SIZE)

[AndreWeiner]

That's exactly what you have implemented

ax.text(row, col, "{:2.2f}".format(A[col, row]), ha="center", va="center", color="w", fontsize=ANNO_SIZE)

You're setting the values of A in all subplots. Try

for ax, mat in zip(axarr, [A, B, C]):
  ...
  ax.text(row, col, "{:2.2f}".format(mat[col, row]), ha="center", va="center", color="w", fontsize=ANNO_SIZE)

[MehdiMak96]

it works thanks a lot

Best,Mehdi

[MehdiMak96]

Hi Andre, I'm having a trouble adding annotation for vector matrix multiplication

A = pt.tensor(
[
    [0, 1, 2],
    [3, 4, 5],
    [6, 7, 8],
])
B = a = pt.tensor([[0, 1, 2]]).view(3, 1)
C = pt.mm(A, B)
vis.plot_matrix_vector_multiplication_as_heatmap(A, B, C, annotate=True)
C

def plot_matrix_vector_multiplication_as_heatmap(A: pt.Tensor, B: pt.Tensor, C: pt.Tensor, annotate: bool=True):
    n_rows, n_cols = A.size()
    n_rows, n_cols = B.size()
    n_rows, n_cols = C.size()
    height = MATRIX_WIDTH * n_rows / n_cols
    fig, axarr = plt.subplots(1, 3, figsize=(3*MATRIX_WIDTH, height))
    axarr[0].imshow(A)
    axarr[1].imshow(B)
    axarr[2].imshow(C)
    if annotate:
      for ax, mat in zip(axarr, [A, B, C]):
         for row in range(n_rows):
            for col in range(n_cols):
                ax.text(row, col, "{:2.2f}".format(mat[col, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE) 

i'm getting this error : index 1 is out of bounds for dimension 1 with size 1

can you please help me plotting the annotation.

Best;Mehdi

[AndreWeiner]

Hi Mehdi, in your code,

    n_rows, n_cols = A.size()
    n_rows, n_cols = B.size()
    n_rows, n_cols = C.size()

n_rows and n_cols is overwritten in each line and finally corresponds to the dimensions of C. You should determine the number of columns and rows in the annotation loop instead:

def plot_matrix_vector_multiplication_as_heatmap(A: pt.Tensor, B: pt.Tensor, C: pt.Tensor, annotate: bool=True):
    height = MATRIX_WIDTH * n_rows / n_cols
    fig, axarr = plt.subplots(1, 3, figsize=(3*MATRIX_WIDTH, height))
    axarr[0].imshow(A)
    axarr[1].imshow(B)
    axarr[2].imshow(C)
    if annotate:
      for ax, mat in zip(axarr, [A, B, C]):
         n_rows, n_cols = mat.size()
         for row in range(n_rows):
            for col in range(n_cols):
                ax.text(row, col, "{:2.2f}".format(mat[col, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE) 

Best, Andre

[MehdiMak96]

Hi Andre, but when I write this I get the following error : local variable 'n_rows' referenced before assignment and it's clear cause n_cols and n_rows were written in this line height = MATRIX_WIDTH * n_rows / n_cols any way I don't think this is what causing the error : index 1 is out of bounds for dimension 1 with size 1 I had the same problem with vis.plot_vector_multiplication_as_heatmap(a, b, c, annotate=True) but I solved it when I changed the format in the code

 if annotate:
        for row in range(n_rows):
            for col in range(n_cols):
                axarr[0].text(row ,col , "{:2.2f}".format(a[col, 0]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE)
                axarr[1].text(row ,col , "{:2.2f}".format(b[0, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE)
                axarr[2].text(row ,col , "{:2.2f}".format(c[col, row]),
                        ha="center", va="center", color="w", fontsize=ANNO_SIZE)

and it worked but it didn't work with this one. Best, Mehdi

[AndreWeiner]

Hi Mehdi, you have to come up with another formula to set the figure's width and height. You could give each subplot a different size depending on the size of A, B, and C. I recommend using gridspec. The old formula n_rows/n_cols constant* is too simple for plotting multiple matrices, so it should be deleted.

In the annotation portion of the code, you need to define n_rows and n_cols before using them:

if annotate:
      for ax, mat in zip(axarr, [A, B, C]):
         n_rows, n_cols = mat.size()
...

Best, Andre

[MehdiMak96]

Hi Andre ,

I'm trying to use assert function with (islcose)

A = pt.tensor(
[ 
    [1, 3, 5],
    [2, 4, 6],
])
A_transpose = pt.transpose(A, 0, 1)
assert pt.isclose(A_transpose, pt.tensor([[1, 2],[3, 4],[5, 6]]).view(3, 2))
vis.plot_matrix_transpose_as_heatmap(A, A_transpose, annotate=True)
A_transpose,pt.tensor([[1, 3, 5],[2, 4, 6],]).view(3, 2)

i'm getting this error : Boolean value of Tensor with more than one value is ambiguous

I can't understand the error but I tried  torch.equal instead of isclose and it worked is that okay, or does it make a difference to what we want to achieve?

[AndreWeiner]

Hi Mehdi,

sensible usage of assert in your scenario would be

A = pt.tensor(
[ 
    [1, 3, 5],
    [2, 4, 6]
])
AT = pt.tensor(
[ 
    [1, 2],
    [3, 4],
    [5, 6]
])
assert (A.T - AT).sum().item() == 0
# another option
assert (pt.transpose(A, 0, 1) - AT).sum().item() == 0

Suggestion: dissect the statement from the inside to the outside and try to understand what each step is doing.

Best, Andre