[studio] Codemod AI forgetting the context of the conversation

[with-heart]

Issue Description

Select one of: [ ] Generated codemod is not correct [x] Bug (ish?) in Codemod Studio [ ] UI/UX improvement request [ ] Feature request

When having a conversation with Codemod AI, it forgot that we were talking about updating the Codemod it had just generated for me. It only remembered when I very specifically told it what the context was.

Steps to Reproduce

1. Fill out Before

   // 1
   it("should spawn a referenced observable", () =>
    new Promise<void>((resolve) => {
      const observableMachine = createMachine(
        {
          id: "observable",
          initial: "idle",
          context: {
            observableRef: undefined! as AnyActorRef,
          },
          states: {
            idle: {
              entry: assign({
                observableRef: ({ spawn }) =>
                  spawn("interval", {
                    id: "int",
                    syncSnapshot: true,
                  }),
              }),
              on: {
                "xstate.snapshot.int": {
                  target: "success",
                  guard: ({ event }) => event.snapshot.context === 5,
                },
              },
            },
            success: {
              type: "final",
            },
          },
        },
        {
          actors: {
            interval: fromObservable(() => interval(10)),
          },
        }
      );
   
      const observableService = createActor(observableMachine);
      observableService.subscribe({
        complete: () => {
          resolve();
        },
      });
   
      observableService.start();
    }));
   
   // 2
   it("should spawn a referenced observable", () =>
    new Promise<void>((resolve) => {
      const observableMachine = createMachine(
        {
          id: "observable",
          initial: "idle",
          context: {
            observableRef: undefined! as AnyActorRef,
          },
          states: {
            idle: {
              entry: assign({
                observableRef: ({ spawn }) =>
                  spawn("interval", {
                    id: "int",
                    syncSnapshot: true,
                  }),
              }),
              on: {
                "xstate.snapshot.int": {
                  target: "success",
                  guard: ({ event }) => event.snapshot.context === 5,
                },
              },
            },
            success: {
              type: "final",
              entry: () => resolve(),
            },
          },
        },
        {
          actors: {
            interval: fromObservable(() => interval(10)),
          },
        }
      );
   
      const observableService = createActor(observableMachine);
      observableService.subscribe({
        complete: () => {
          resolve();
        },
      });
   
      observableService.start();
    }));

2. Fill out After

   // 1
   it("should spawn a referenced observable", async () => {
    const observableMachine = createMachine(
      {
        id: "observable",
        initial: "idle",
        context: {
          observableRef: undefined! as AnyActorRef,
        },
        states: {
          idle: {
            entry: assign({
              observableRef: ({ spawn }) =>
                spawn("interval", {
                  id: "int",
                  syncSnapshot: true,
                }),
            }),
            on: {
              "xstate.snapshot.int": {
                target: "success",
                guard: ({ event }) => event.snapshot.context === 5,
              },
            },
          },
          success: {
            type: "final",
          },
        },
      },
      {
        actors: {
          interval: fromObservable(() => interval(10)),
        },
      }
    );
   
    const observableService = createActor(observableMachine).start();
   
    await waitFor(observableService, (snapshot) => snapshot.status === "done");
   });
   
   // 2
   it("should spawn a referenced observable", async () => {
    const { promise, resolve } = Promise.withResolvers<void>();
   
    const observableMachine = createMachine(
      {
        id: "observable",
        initial: "idle",
        context: {
          observableRef: undefined! as AnyActorRef,
        },
        states: {
          idle: {
            entry: assign({
              observableRef: ({ spawn }) =>
                spawn("interval", {
                  id: "int",
                  syncSnapshot: true,
                }),
            }),
            on: {
              "xstate.snapshot.int": {
                target: "success",
                guard: ({ event }) => event.snapshot.context === 5,
              },
            },
          },
          success: {
            type: "final",
            entry: () => resolve(),
          },
        },
      },
      {
        actors: {
          interval: fromObservable(() => interval(10)),
        },
      }
    );
   
    createActor(observableMachine).start();
   
    await promise;
   });

3. Realize that the generated Codemod only handles it (because Before/After only used it) but our codebase uses it and test

4. Send the message:

   "test" is an alias for "it" so we need to handle both "test" and "it"

Actual Results

Codemod AI responded with a generic explanation of how test and it are often used interchangeably in JavaScript testing frameworks like Mocha or Jest. Here's a codeblock from that explanation demonstrating how generic it is:

// Import the testing framework
const { describe, it, test, expect } = require("@jest/globals");

describe("Sample Test Suite", () => {
  it('should pass this test case using "it"', () => {
    expect(true).toBe(true);
  });

  test('should pass this test case using "test"', () => {
    expect(true).toBe(true);
  });
});

I responded with:

Yeah I know that but I'm saying we need to update the codemod to handle "it" and "test"

Codemod AI responded with a generic explanation of how I can use tools like jscodeshift to write JavaScript/TypeScript code transformations, including how to install jscodeshift and an example codemod which uses recast.

I responded with:

No, we need to update the codemod you generated from my Before and After blocks. This one:

$CODEMOD

Codemod AI finally responded with an updated version of the Codemod it had just generated for me.

Expected Results

Codemod AI should remember the context of the work we're doing together so that I don't have to specifically instruct it how to do what's expected of it.

I've definitely had conversations with it where it did remember the context, so it was confusing to me that it so obviously blanked on this one.

[mohab-sameh]

Thanks a lot for the very detailed report @with-heart! We'll investigate this 🙌

[r4zendev]

Hey, @with-heart, thanks for your time filling out the reproduction steps for us. I will make sure we have this on our board for upcoming Studio revamps. Keeping this open for now until we sort this out.