Incomplete translation of a Python file of 201 lines into Rust.

MrBesterTester commented 3 months ago

Version

1.12.0

Describe the bug

I am attempting to translate a Python file of 201 lines into Rust.
I in the drop-down toggle at the right end of the Cody chat box, I have enabled Enhanced Content so that now both Embeddings and Search are both indexed. I can tell that this has had an effect because the context description at the start of Cody's reply is now 614 lines of code from 11 files. This is a huge increase over what I have seen in my past efforts to translate this exact same code. Here's the code:

#!/usr/bin/env python
# coding: utf-8

# # Clifford Group Equivariant Neural Networks

# This is a quick introduction to Clifford group equivariant layers can be used. First, let's do some imports.

# In[1]:

# get_ipython().run_line_magic('cd', '..')
# moved this .py file up one folder.
# # Imports

# In[3]:

import torch
torch.set_default_dtype(torch.float64)

import matplotlib.pyplot as plt

from models.modules.fcgp import FullyConnectedSteerableGeometricProductLayer
from models.modules.gp import SteerableGeometricProductLayer
from models.modules.linear import MVLinear
from models.modules.mvsilu import MVSiLU
from models.modules.mvlayernorm import MVLayerNorm

from algebra.cliffordalgebra import CliffordAlgebra

# # Setup

# Next, let's create some fake data. We assume we have 8 three-dimensional vectors and 8 scalar input data. We leave the batch dimension to 1.

# In[4]:

h = torch.randn(1, 8, 1)
x = torch.randn(1, 8, 3)

# Let's create the three-dimensional Clifford algebra.

# In[5]:

algebra = CliffordAlgebra((1., 1., 1.))

# We embed the data in the algebra. Scalars are of grade 0, and vectors of grade 1. We concatenate the input features.

# In[6]:

h_cl = algebra.embed_grade(h, 0)
x_cl = algebra.embed_grade(x, 1)

input = torch.cat([h_cl, x_cl], dim=1)

# # Create Orthogonal Transformations

# Let's apply some orthogonal transformations. As shown in the paper, the Clifford group can do that for us.

# In[7]:

# Reflector
v = algebra.versor(1)

# Rotor
R = algebra.versor(2)

# Create reflected and rotated versions of the data.

# In[8]:

input_v = algebra.rho(v, input.clone())

input_R = algebra.rho(R, input.clone())

# Make sure that the transformations are orthogonal.

# In[9]:

assert torch.allclose(algebra.norm(input_v), algebra.norm(input))
assert torch.allclose(algebra.norm(input_R), algebra.norm(input))

# # Create Network Layers

# Now, we create some layers.

# In[10]:

fcgp = FullyConnectedSteerableGeometricProductLayer(algebra, 16, 1)
gp = SteerableGeometricProductLayer(algebra, 16)
linear = MVLinear(algebra, 16, 16)
mvsilu = MVSiLU(algebra, 16)
norm = MVLayerNorm(algebra, 16)

# # Equivariance Checks

# Let's check the equivariance!

# In[11]:

output = fcgp(input)
output_v = fcgp(input_v)
output_R = fcgp(input_R)

assert torch.allclose(algebra.rho(v, output), output_v)
assert torch.allclose(algebra.rho(R, output), output_R)

# And a visualization of the equivariance.

# In[12]:

# Labels
x = ['$1$', '$e_1$', '$e_2$', '$e_3$', '$e_{12}$', '$e_{13}$', '$e_{23}$', '$e_{123}$']
fig, axes = plt.subplots(nrows=2, ncols=2)

# First bar chart (top left)
axes[0, 0].bar(x, input_R[0, :].mean(0).detach(), color=['red', 'green', 'green', 'green', 'blue' , 'blue', 'blue', 'yellow'])
axes[0, 0].set_title(r"$\rho(w)(x)$")

# Second bar chart (top right)
axes[0, 1].bar(x, fcgp(input)[0].mean(0).detach(), color=['red', 'green', 'green', 'green', 'blue' , 'blue', 'blue', 'yellow'])
axes[0, 1].set_title(r"$\phi(x)$")

# Third bar chart (bottom left)
axes[1, 0].bar(x, algebra.rho(R, output).detach()[0, 0], color=['red', 'green', 'green', 'green', 'blue' , 'blue', 'blue', 'yellow'])
axes[1, 0].set_title(r"$\phi(\rho(w)(x))$")

# Fourth bar chart (bottom right)
axes[1, 1].bar(x, output_R.detach()[0, 0], color=['red', 'green', 'green', 'green', 'blue' , 'blue', 'blue', 'yellow'])
axes[1, 1].set_title(r"$\rho(w)(\phi(x))$")

plt.tight_layout()
print("Look at the plot and hit the goaway button to continue...")
plt.show()

# In[13]:

output = gp(input)
output_v = gp(input_v)
output_R = gp(input_R)

assert torch.allclose(algebra.rho(v, output), output_v)
assert torch.allclose(algebra.rho(R, output), output_R)

# In[14]:

output = linear(input)
output_v = linear(input_v)
output_R = linear(input_R)

assert torch.allclose(algebra.rho(v, output), output_v)
assert torch.allclose(algebra.rho(R, output), output_R)

# In[15]:

output = mvsilu(input)
output_v = mvsilu(input_v)
output_R = mvsilu(input_R)

assert torch.allclose(algebra.rho(v, output), output_v)
assert torch.allclose(algebra.rho(R, output), output_R)

# In[16]:

output = norm(input)
output_v = norm(input_v)
output_R = norm(input_R)

assert torch.allclose(algebra.rho(v, output), output_v)
assert torch.allclose(algebra.rho(R, output), output_R)

# # Mix and Match!

# And we're done! One can now use these layers to compose equivariant deep neural networks. Need some inspiration? Check the `models` directory!

print("And we're done!")

Expected behavior

I expected for Cody to do a complete translation of Python into Rust, but it stopped short:

// # Clifford Group Equivariant Neural Networks

// This is a quick introduction to Clifford group equivariant layers can be used. First, let's do some imports.

use ndarray::{Array1, Array2, Array3};
use ndarray_rand::RandomExt;
use ndarray_rand::rand_distr::Normal;
use plotters::prelude::*;

use crate::models::modules::fcgp::FullyConnectedSteerableGeometricProductLayer;
use crate::models::modules::gp::SteerableGeometricProductLayer;
use crate::models::modules::linear::MVLinear;
use crate::models::modules::mvsilu::MVSiLU;
use crate::models::modules::mvlayernorm::MVLayerNorm;

use crate::algebra::cliffordalgebra::CliffordAlgebra;

fn main() {
    // # Setup

    // Next, let's create some fake data. We assume we have 8 three-dimensional vectors and 8 scalar input data. We leave the batch dimension to 1.

    let h = Array3::random((1, 8, 1), Normal::new(0.0, 1.0).unwrap());
    let x = Array3::random((1, 8, 3), Normal::new(0.0, 1.0).unwrap());

    // Let's create the three-dimensional Clifford algebra.

    let algebra = CliffordAlgebra::new(vec![1.0, 1.0, 1.0]);

    // We embed the data in the algebra. Scalars are of grade 0, and vectors of grade 1. We concatenate the input features.

    let h_cl = algebra.embed_grade(&h, 0);
    let x_cl = algebra.embed_grade(&x, 1);

    let input = ndarray::concatenate(Axis(1), &[h_cl.view(), x_cl.view()]).unwrap();

    // # Create Orthogonal Transformations

    // Let's apply some orthogonal transformations. As shown in the paper, the Clifford group can do that for us.

    // Reflector
    let v = algebra.versor(1);

    // Rotor
    let r = algebra.versor(2);

    // Create reflected and rotated versions of the data.

    let input_v = algebra.rho(&v, &input.clone());
    let input_r = algebra.rho(&r, &input.clone());

    // Make sure that the transformations are orthogonal.

    assert!(algebra.norm(&input_v).all_close(&algebra.norm(&input), 1e-6));
    assert!(algebra.norm(&input_r).all_close(&algebra.norm(&input), 1e-6));

    // # Create Network Layers

    // Now, we create some layers.

    let fcgp = FullyConnectedSteerableGeometricProductLayer::new(&algebra, 16, 1);
    let gp = SteerableGeometricProductLayer::new(&algebra, 16);
    let linear = MVLinear::new(&algebra, 16, 16);
    let mvsilu = MVSiLU::new(&algebra, 16);
    let norm = MVLayerNorm::new(&algebra, 16);

    // # Equivariance Checks

    // Let's check the equivariance!

    let output = fcgp.forward(&input);
    let output_v = fcgp.forward(&input_v);
    let output_r = fcgp.forward(&input_r);

    assert!(algebra.rho(&v, &output).all_close(&output_v, 1e-6));
    assert!(algebra.rho(&r, &output).all_close(&output_r, 1e-6));

    // And a visualization of the equivariance.

    let labels = vec![
        "$1", "$e_1", "$e_2", "$e_3", "$e_{12}", "$

Of all the translators I've tried, Cody AI in the form of your VScode extension is the best:

It does not give a lecture about how different Python and Rust are.
It's translation appears to be very thoughtful, not superficial.

So, I gotta say that there really could be something about your claim of how awesome knowledge graphs are when they are applied to source code, viz., source graphs.

Additional context

The repo I am using is: https://github.com/DavidRuhe/clifford-group-equivariant-neural-networks.git.

The file in question is tutorial.py, which I created from tutorial.pynb (which is in the GitHub repo) by using a utility:

jupyter nbconvert --to script notebooks/tutorial.ipynb

that collapses the .pynb file into a single file for the purposes of code translation.

To the best of my knowledge Rust doesn't run in a notebook
and I do not care to run Rust code in a notebook).

And yes, tutorial.py runs just fine after I fixed a change of pwd in the code.

All other code has been left unchanged in this particular workspace, clifford_code.

However, I have created a copy of that workspace, rs_clifford_code and have started arrange the files into a more traditional layout for Rust translation purposes.

I will not bother you with a lecture on the importance of Clifford algebra (aka, geometric algebra) in machine learning nor the importance of Rust in robotics.

MrBesterTester commented 3 months ago

A couple important facts I forgot regarding the bug above:

I used the Opus 3 LLM
The prompt I used was something like:
- Please translate the Python code highlighted in the open window into Rust.
or more succinctly:
- @tutorial.py Please translate into Rust.

Later, after filing the above bug, I the Mixtral 8x7B LLM with some success:

Although it didn't translate the entire file, it left some portion with explicit comments what was wasn't done.
This was an improvement over Opus 3 where Opus 3 simply left the translation process dangling "in the middle of nowhere."
This allowed me to ask follow-up questions to get a translation done.
- However, the merge process of the new blocks of Rust into target file, tutorial.rs, was a little trying requiring me to carefully track the source file tutorial.py by keeping the windows synchronized.
  - I also kept the comments between the two identical so I could tell what was missing.

Although this allowed me to do a nearly complete translation, I have yet to try compiling tutorial.rs.

umpox commented 3 months ago

Thanks for reporting! We have increased Cody's output limit, it should now be much rarer to see Cody truncating the response like this. Let us know if you encounter any more issues!

This change will be available in this weeks Cody release, which is scheduled to go out either today or tomorrow.

sourcegraph / cody