Open FabienArcellier opened 3 months ago
Hey, this is really cool stuff.
1) The wrapped Dataframe makes sense. I like wf.EditableDataframe
. I think it's ok to add some complexity for dataframe editing. I don't see why we'd require wrapping it for display.
2) To store, we'd keep a copy of the original dataframe and manipulate it? Support Pandas, Polars, etc, serialize to Apache Arrow?
3)
"df": df # raw dataframe, user has to reimplement everything (straight forward and user friendly for simple binding)
What do you mean here? Do you want to allow two-way binding?
This ticket specifies several proposals for rewrithing
DataFrame
component. I was inspired by Mikita's work on the AI part for these proposals.User interface
[OK] - P1. Adding a record to a dataframe editor is done using a button that appears on hover at the bottom of the dataframe editor. The click triggers the
wf-dataframe-add
event.illustration of interaction from obsidian![Peek 2024-06-12 16-37](https://github.com/FabienArcellier/streamsync/assets/159559/590a5d25-0a08-4b85-9eec-13585343b9b0)
[OK] - P2. A quick action menu on the left appears on hover and allows you to delete a record / add your own actions. This menu implements delete a record as the default action. The actions would be accessible in the
wf-record-action
event.illustration of interaction from monday.com![Peek 2024-06-12 16-27](https://github.com/FabienArcellier/streamsync/assets/159559/71414eaf-d4af-4496-8f74-e17b344f028c)
Here is an example of content for the "quick actions" field with 3 items in the menu.
[OK] - P3. A user can select multiple rows of a dataframe in the dataframe editor. Selecting a record triggers the
wf-dfeditor-select
event.[OK] - P4. moving records is not supported. This would be a plus but I don't know how to implement it. The ideas I have would only be compatible with a specific dataframe format. This would also prevent sorting on the columns.
[LATER] - P5. implement a data repeater working as a repeater. This part is for later (to describe). This feature would allow to customize rendering.
[DISCUSSION] - P6. the event and fields from frontend
single
multiple selection
Backend
[OK] - P7. a dataframe is re-encoded in writer. Like the AI component, it allows you to have pre-existing operations to process the payloads of the editor triggered in the UI and to have your own serializer. We can use the same object for the current dataframe reader (to be challenged). The frontend component must also be compatible with a simple dataframe but in this case, the exchange protocol will be heavier. We return the dataframe at each change (see the section on the protocol).
[OK] - P8. changes in the dataframe editor trigger events. Changes in the dataframe editor trigger events that will be processed easily because the
writer.Dataframe
object has methods to directly process payloads.Protocol
[LATER] - P9. a double protocol at the stream level optimizes the transfer of the dataframe. This model allows you to update a dataframe efficiently during front/back exchanges by transferring mutations. This optimization improves the responsiveness of the UI. This pattern is possible thanks to the creation of
wf.EditableDataframe
.[KO] - P10: encode the record position into
__index_level_0__
: pyarrow drop the support of index in the dataframe by default. We need asurogate key
to manage update. There is a way to handle dataframe index that encode the index into__index_level_0__
. I propose to use this attribute as row identifier.I write code that test this possibility with 4 datastructure (pandas, polars, list of record, list of array) :