Data catalog API - Githubissues

vhawk19 commented 2 months ago

Currently we dont have a good data catalog
This would allow us to better support remote datasources
What does this mean?
- Essentially everything dataset be it input/output is relation
- Can we tag these and make it easily accessible over an api?
- Preview the contents etc.

malavikasmenon commented 2 months ago

wanted to understand this better. By data catalog do you mean have some datasets/tables already available for people to play with?

vhawk19 commented 2 months ago

My bad, should have phrased it better.
But in essence what I meant is (1) A better API to access the underlying relations (2) surface this better on the UI.
For instance, when we upload a file using duckdb wasm iit creates a table, but if we create a table from a remote data source it creates a table and we are not tracking it (i.e. what is there on the left panel is currently is coupled with uploading files and not tables, whereas we should be tracking the tables)
Another question worth asking is do we store the results also as a temporary table, not sure how this would look like in terms of structure seems interesting nonetheless.

malavikasmenon commented 2 months ago

got it yes agreed on UI and API front

vhawk19 / ambaan