Getting loads/stores to work with all mutable types

[ameliatastic]

Adds the ability to define account classes that store arbitrary data types, including:

• Lists of things (List[T])
• Arrays of anything (including nested arrays - Array[Array[T, M], N])
• User-defined classes (including classes that store any other types)

This is going to be a big change to the code gen, so it necessarily won't work with any of the current tests. I'll make sure to create a new test for all of the more complex features and use them in a real program so that we can be reasonably sure that this won't break existing code.

Closes #33 Closes #59 Closes #64

[ameliatastic]

The following snippet compiles right now, and generates (hopefully correct) code to load/store the account type and relevant nested types. Feast your eyes:

from util import data_structures

# ...

class Data:
    x: str
    y: List[i32]
    z: data_structures.IntList

class MyAccount(Account):
    a: List[i32]
    b: List[List[i32]]
    c: Array[Array[i32, 2], 2]
    d: Data
    e: List[Data]
    f: data_structures.IntList
    g: str
    h: u64

@instruction
def use_my_account(my_account: MyAccount):
    pass

So that covers storing:

• lists
• lists of lists
• arrays of arrays
• user classes (which themselves have strings, lists, and other classes)
• lists of user classes
• user classes that are imported from other files (!)
• strings
• u64 my beloved

This is the sort of comprehensive test I hope to include and actually test with in a local validator setup. Haven't tried doing much in the way of using the contents of the account yet.

[mcintyre94]

This is great, will make things much more flexible! It might fix #61 as well?

[ameliatastic]

This is great, will make things much more flexible! It might fix #61 as well?

That issue is closed, did you mean #64?

[mcintyre94]

That issue is closed, did you mean #64?

Oops yep that's the one! If this fixes the IDL types that'd be awesome!

Actually it won't close that because we'll still need changes to get serialization on events with these fields, but it might fix the IDL part

[ameliatastic]

Update: I think it's working. I THINK IT'S WORKING.

Here's the program I wrote:

class Deep:
    num: i32

    def __init__(self, num: i32):
        self.num = num

class Nested:
    deep: Deep

    def __init__(self, num: i32):
        self.deep = Deep(num)

    def reset(self):
        self.deep = Deep(0)

class Data(Account):
    array_2d: Array[Array[i32, 2], 2]
    int_list: List[i32]
    int_list_2d: List[List[i32]]
    string: str
    nested: Nested
    nested_list: List[Nested]

@instruction
def init(signer: Signer, data: Empty[Data]):
    init_data = data.init(
        payer=signer,
        seeds=[signer],
        padding=1024
    )

    init_data.int_list = [1, 2]
    init_data.int_list_2d = [[3, 4], [5, 6]]
    init_data.string = 'Hello'
    init_data.nested = Nested(7)
    init_data.nested_list = [Nested(8), Nested(9)]

@instruction
def test_stored_mutables(signer: Signer, data: Data):
    # Modify everything in the Data account 

    # [[0, 0], [0, 0]] -> [[1, 0], [0, 0]]
    data.array_2d[0][0] = 1
    # [1, 2] -> [1, 2, 0]
    data.int_list.append(0)
    # [[3, 4], [5, 6]] -> [[3, 0], [5, 6]]
    data.int_list_2d[0][-1] = 0
    # "Hello" -> "Hello World"
    data.string = data.string + ' World'
    # N(D(7)) -> N(D(0))
    data.nested.reset()
    # [N(D(8)), N(D(9))] -> [N(D(0)), N(D(9)), N(D(10))]
    data.nested_list[0].reset()
    data.nested_list.append(Nested(10))

I then wrote a test harness to call init then test_stored_mutables. Here's the before and after of the data account:

{
  array2D: [ [ 0, 0 ], [ 0, 0 ] ],
  intList: [ 1, 2 ],
  intList2D: [ [ 3, 4 ], [ 5, 6 ] ],
  string: 'Hello',
  nested: { deep: { num: 7 } },
  nestedList: [ { deep: [Object] }, { deep: [Object] } ]
}

{
  array2D: [ [ 1, 0 ], [ 0, 0 ] ],
  intList: [ 1, 2, 0 ],
  intList2D: [ [ 3, 0 ], [ 5, 6 ] ],
  string: 'Hello World',
  nested: { deep: { num: 0 } },
  nestedList: [ { deep: [Object] }, { deep: [Object] }, { deep: [Object] } ]
}

Every change seems to be applied correctly, and of course there weren't any program-stopping bugs (which is important since multi-level indexing needed some changes to work). Can't see the contents of the nestedList (and I don't want to just post the stringified output bc it's a mess and this is already a huge comment) here but I verified it myself.

Can you think of any other types of modifications/storage features that might need testing? @mcintyre94

[ameliatastic]

With the last commit, here's what the generated IDL looks like for an event with a single field nums: List[i32]. Is this right?

  "events": [
    {
      "name": "MyEvent",
      "fields": [
        {
          "name": "nums",
          "type": {
            "vec": "i32"
          },
          "index": false
        }
      ]
    }
  ]

[mcintyre94]

Nice, that looks like a really good set of tests! The only thing I can think of that's missing would be pulling one of the classes into a separate imported file like you did in the earlier comment just to make sure everything still works across the import

Event IDL looks good!

I tested the equivalent in Anchor:

#[event]
pub struct MyEvent {
    nums: Vec<i32>
}

IDL output from Anchor:

  "events": [
    {
      "name": "MyEvent",
      "fields": [
        {
          "name": "nums",
          "type": {
            "vec": "i32"
          },
          "index": false
        }
      ]
    }
  ]

[ameliatastic]

Alright, new test added. I hand-checked all the tests (after some refactoring) and found that thankfully the only code that was changed for each test (except for event.py) was unused, so the update should be safe!