search

datopian / datahub

🌀 Rapidly build rich data portals using a modern frontend framework
https://datahub.io/opensource
MIT License
2.19k stars 325 forks source link

Show nice data table view for CSV file with markdown transclusion pattern e.g. `![](link-to-file.csv)` #1233

Open rufuspollock opened 2 months ago

rufuspollock commented 2 months ago

Would love a short-hand for have a data table using the classic markdown transclusion ...

Job story / Feature description

In markdown have this ...

![](path-to-my.csv)

And turns into a nice data table:

Image

Other features

Tasks

### Tasks
- [ ] Shape it ...

Shaping

I think we can reuse a lot of code we use for pdf transclusion etc in our remark plugin ...

rufuspollock commented 2 months ago

I think this is a super cool feature 😉

gradedSystem commented 1 month ago

(UPDATED)

Situation

Markdown documents need to be enhanced by automatically converting CSV file references (e.g., ![](path-to-my.csv)) that appear at the start of a line into fully transcluded content within the page.

Problem

The challenge is to display the referenced CSV data as well-formatted, readable tables directly within the markdown documents, ensuring the data is both accessible and visually appealing.

Solution

Appetite

Rabbit-holes

No-goes

Ignoring edge cases such as: missing headers or special characters that could lead to incorrect table generation

Appendix

cc @olayway

rufuspollock commented 1 month ago

@gradedSystem you need to do way more shaping here. You really need to show you've understood the problem and that you have investigations solutions in detail. For example, i'd imagine you'd be looking at the existing code we have for the remark plugin that handles image transclusions etc. I imagine you'd be saying which table tool we'd be using for the rendering etc.

olayway commented 1 month ago

@gradedSystem you need to do way more shaping here. You really need to show you've understood the problem and that you have investigations solutions in detail. For example, i'd imagine you'd be looking at the existing code we have for the remark plugin that handles image transclusions etc. I imagine you'd be saying which table tool we'd be using for the rendering etc.

+1

gradedSystem commented 1 month ago

@rufuspollock @olayway I have updated the shape-up could you please review it?

olayway commented 1 month ago

Situation Markdown documents need to be enhanced by automatically converting CSV file references (e.g., ) that appear at the start of a line into fully transcluded content within the page.

I'd say the situation is that, currently, in order to embed a CSV preview in your markdown file, you need to include this:

<FlatUiTable data={{ url: "data.csv" }}/>

And I'd say it's not that much of a problem, but rather we have an opportunity to make this even simpler, i.e. to use a well known (debatable I think) transclusion syntax, i.e. ![](data.csv) or ![[data.csv]].

olayway commented 1 month ago

I'm not sure I understand your points in rabbit holes and no-goes sections. Can you elaborate?

olayway commented 1 month ago

@gradedSystem As for the solution:

Currently our remark-wiki-link plugin has support for Obsidian-style - ![[]] - image and pdf embeds. It would be weird to translate ![](file.csv) to csv preview table and not have a corresponding support for Obsidian-style ![[file.csv]] in the remark-wiki-link plugin. So I think we have two options here:

  1. Add support for Obsidian-style ![[data.csv]] transclusion only.
  2. Add support for Obsidian-style ![[data.csv]] transclusion and also create a new remark plugin for common-mark format transclusions ![](data.csv).

I'd say let's start with extending our existing plugin as it's easiest to implement. Also, creating a new plugin requires a bit more thought in my opinion, e.g. do we plan to support other file formats as well, do we keep the two plugins in sync feature-wise, is it even needed as most people are used to ![]() being reserved for images anyway, etc., does anyone really needs this or is it ok if we just have ![[]] support?

gradedSystem commented 1 month ago

I'm not sure I understand your points in rabbit holes and no-goes sections. Can you elaborate?

  1. Meaning that there could be other formats, but now I have read it and it makes no sense since we are working with only csv support ( I will remove from the first point)
  2. Regarding the second point I was just curious that how much large files our website could handle, but now that I thought about, it also doesn't make sense since Github itself has the limitation to the file size, so in this case it is also not applicable

Currently, I have no idea what the rabbit holes could be.

olayway commented 1 month ago

@gradedSystem

Now I think about it, is the following an improvement worth the effort?

From this: <FlatUiTable data={{ url: "data.csv" }}/> to this: ![](data.csv) (or ![[data.csv]]).

For someone who uses this feature frequently and doesn't care about any table configuration options - yes, this may be a slight improvement. But probably more often than not, you'll also want to e.g. sort the table by some column or filter out some values, instead of displaying a raw file as is (see https://github.com/datopian/datahub/issues/1270). At least let's go with the easiest option - extend existing remark-wiki-link plugin instead of creating a new one for common-mark style embeds.

cc: @rufuspollock

olayway commented 1 month ago

Meaning that there could be other formats, but now I have read it and it makes no sense since we are working with only csv support ( I will remove from the first point)

I think our FlatUiTable doesn't support other tabular formats atm.

rufuspollock commented 2 weeks ago

From this: <FlatUiTable data={{ url: "data.csv" }}/> to this: (or ![[data.csv]]).

Yes, this is an improvement. One i have to lookup in the docs, one i know automatically.

And it should be cheap to do ...