LIDA rename columns

[trojrobert]

LIDA columns replace special characters with _

[victordibia]

Can you explain the exact issue? E.g., are you saying that replacing special characters in column names (i.e. standardizing column names) with is a problem? If so, what is the use case, and how does this impact the developer workflow or solution?

[trojrobert]

I was building an app where I wanted to do more with the column names. For instance, I wanted to show number of nan and manually and more details about the data

[victordibia]

Thanks for the clarification. It seems to me that your app should keep track of the dataset and all other processing needed without relying on LIDA's representation of the data by the Manager (.data). You can also keep track of both name lists and use that information as needed. Note that the column index will not change, but the title may.

In general, the renaming is done (and required) for two high level reasons:

• Data with special characters in column names can lead to serialization errors (e.g., data not being saved or loaded correctly in )
• Having clear column names as single variables (e.g., removing space etc) makes it easier for LLMs to reference those columsn and ground the generated code using those column names. Without this, the LLM is more likely to hallucinate incorrect names.

What do you think?

[trojrobert]

I understand why the column renaming is importance. Serialization error can be a big problem. Maybe we can add a note in the code or in the summary that let the user have an idea of some things that changed and few information about the summary