[VEDA2 Refactor] Exploration - Data layer scaffolding

[sandrahoang686]

Description: As part of the refactor work for veda-ui v2, we should decouple data sources from components by creating a data layer as part of the new core ui logic library which can be consumed at the instance level to be then fed into new feature components. We should define how this data layer structure would look like for example should it be a library or should we introduce the Context API Provider / Consumer model to also effectively manage global state?

Acceptance Criteria:

• [ ] Define data layer structure and pattern to use, data layer should be able to scale with new data sources
• [ ] Research on whether or not its worth bringing in the context api provider/consumer model here, currently veda-ui has "virtual modules" (import { datasets, DatasetLayer } from 'veda';) is it worth having global state replace these?
• [ ] Document Data Layer Architecture here
• [ ] Create the scaffold PR to veda-ui repo. Ideally, this data layer should live at the root level with the core-ui library, but this is to be determined and up for thoughts, for example...

  
  repo level

• veda-ui
  • ...current veda ui library....
• data-layer

Resources: https://miro.com/app/board/uXjVN6lkBnc=/ Virtual modules from VEDA Architect document

[hanbyul-here]

The current architect document has a section about virtual modules: Virtual modules from VEDA Architect document

[sandrahoang686]

Also worth mentioning, we currently use jotai (state management library) across the exploration components. We should decide whether or not we should keep using it or is it worth implementing the context api ourselves. Personally, from limited use already, I do like it as it looks like we dont have to implement a reducer to update state but whoever takes this ticket on, should explore more.

[sandrahoang686]

Spoke with @hanbyul-here here and we agreed to keep managing state with Jotai