[BUG]: `Transform Dataset` operator behaves anomalously - cannot generated code or access input dataframes

liunelson commented 1 month ago

Scenario: Transform Dataset
Step: 3.4
Link to the project: https://app.staging.terarium.ai/projects/34e7b955-f408-4e8b-be27-9429480ad6aa/workflow/954cf33c-f207-449d-b069-5337b8fc24ba?operator=0e564ce4-afc6-460c-8feb-57c1934cccfe

Describe the issue

The Transform Dataset operator has a few bugs:

After just two code cells (printing what d2 is and asking the LLM for the join code), the session seems to have lost access to the input dataset d1.
Starting from a new Transform Dataset operator, now the generated LLM code won't run fully - the status is either stuck at Busy or Offline
Minor issue: unknown code output shown in the operator every time - not informative, nor should it be shown to users?
```
{'modules': {}, 'variables': {}, 'functions': {}}
```

blanchco commented 1 month ago

Asked Jataware about this, beaker seems to just hang with this command which may cause downstream issues https://unchartedsoftware.slack.com/archives/C049E6PA16V/p1728054249803609

blanchco commented 1 month ago

It seems this might be an issue with Jupyter itself, we run out of memory. It might be worth looking into a solution where if beaker starts to hang like this we can add a timeout and start a new session informing the user of the issue

liunelson commented 1 month ago

What's the plan for this issue?

DARPA-ASKEM / terarium

[BUG]: `Transform Dataset` operator behaves anomalously - cannot generated code or access input dataframes #5056