Open kevboh opened 3 years ago
Taking an array of arrays and writing it to a CSV sounds fairly doable for simple cases.
I guess the next question should be: why doesn't there seem to exist a plugin to export the outcome of the preview mode into a separate file?
I think it would be possible to write a plugin to do exactly that by exporting the preview mode HTML directly; seems like an interesting idea.
I think it would be possible to write a plugin to do exactly that by exporting the preview mode HTML directly; seems like an interesting idea.
Oh, I‘d love to do that but I never looked into how to write an Obsidian plugin. Interesting challenge for the weekend! But I totally don’t mind if you do it, too, @blacksmithgu! 😊
Plugin "basically" works. Don't know how to write the CSV string I have into a file inside the vault, inside a subfolder. But will find out eventually ...
Plugin "basically" works. Don't know how to write the CSV string I have into a file inside the vault, inside a subfolder. But will find out eventually ...
Okay, last post regarding this (don't want to hijack this issue for my plugin development experience): it saves CSV files now. So it works – for me. Not in a state to be published, though. Have to fiddle with many more details, like settings, edge cases etc. And currently it'll only work correct if there's just one table (doesn't have to be one from Dataview) in the reading pane. Have to work on the case when there is more than one table ...
As a plugin developer, it will be cool to have an API in external interface Dataview with following method:
With such logic, plugins like Github Publisher or Digital Garden can "burn" views, when upload a note.
— Huge +1 for this idea. It seems such an API would require relatively little code/effort added to the plugin itself. — I would also want a second method call that returned whatever internal structure you have prior to rendering, this might be a preferable starting point for building various export capabilities. — Other folks could then extend the plugin to cover this range of ‘export' oriented use-cases.
@stefan wolfrum, I agree this thread is not the proper place to talk about your plugin work, but I do encourage you to keep posting about it as a newbie. (Maybe on the Obsidian discord plugin dev channel? Not sure.) In desperation, I have also written a plugin to extend obsidian behaviors, despite not being a JS programmer, and having very little in the way of docs for Obsidian internals. (I have NO idea how complex plugins are written… I am in awe… mine was done by morphing an existing plugin and grepping source code for dozens of existing plugins… never programmed this way before! Anyway, please keep posting somewhere..
On May 30, 2022, at 10:32 AM, Ivan Chernov @.***> wrote:
As a plugin developer, it will be cool to have an API in external interface Dataview with following method:
method accepts code as string renders content returns as an HTMLElement With such logic, plugins like Github Publisher or Digital Garden can "burn" views, when upload a note.
— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1141370314, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL2237Z3XKICSXIXZ63NLVMT3S5ANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.
I've add some copy'n'paste functionality for an external plugin API. The implementation is not near good, but I can change based on feedback / review.
@blacksmithgu I'm getting a little lost in this issue now, was wondering if I missed an update to the main feature you discussed in February?
Alright, so I'm finally around to looking at this again. I have two modes which I am planning on supporting directly right away:
- Preview mode: All dataviews will support both "Export to Markdown" and "Freeze as Markdown", which (1) copy markdown for you to paste, and (2) directly replace the query block with markdown. This requires manual action to freeze but at least enables the core functionality.
- Bulk Freezing: Dataview will provide a simple UI for replacing ALL queries in your vault with their static results in one atomic operation - best for if you are migrating away from Obsidian or Dataview but want to keep all of your tables and indices. This will also support the ability to keep the existing Dataview block and just append the result as Markdown after the block.
These two scenarios should unblock the most important use-cases: data archiving, basic view support in publish and GitHub, migrating away from Dataview, and even being able to see important Dataview-based links in the graph. I can also consider implementing "Bulk Freezing" as a simple automatic job that runs at user-defined intervals or on important operations.
Thank you!!
Just wanted to let you guys know that I submitted my „table to csv export“ plugin to be reviewed by the Obsidian team. It’s here, in case you want to test manually in advance.
Only applying the freezer to a specific dataview source (i.e., anything you can put in
FROM
) seems like a good feature.
I think you can make every file inside a specific folder(modified in the plugins configuration) be freezed or trasnformed in markdown. So for exemple you can copy a "bookreview" folder and place inside a "publish" folder them is goin to be converted.
Is there any ETA for any of the two proposed features? For me the most useful one would be automatic bulk freezing (I export my obsidian notes to my personal site). Is there any big blocking issue or chunks of work that might benefit from contributors?
Thanks!
Alright, so I'm finally around to looking at this again. I have two modes which I am planning on supporting directly right away:
- Preview mode: All dataviews will support both "Export to Markdown" and "Freeze as Markdown", which (1) copy markdown for you to paste, and (2) directly replace the query block with markdown. This requires manual action to freeze but at least enables the core functionality.
- Bulk Freezing: Dataview will provide a simple UI for replacing ALL queries in your vault with their static results in one atomic operation - best for if you are migrating away from Obsidian or Dataview but want to keep all of your tables and indices. This will also support the ability to keep the existing Dataview block and just append the result as Markdown after the block.
These two scenarios should unblock the most important use-cases: data archiving, basic view support in publish and GitHub, migrating away from Dataview, and even being able to see important Dataview-based links in the graph. I can also consider implementing "Bulk Freezing" as a simple automatic job that runs at user-defined intervals or on important operations.
Is there any ETA for any of the two proposed features? For me the most useful one would be automatic bulk freezing (I export my obsidian notes to my personal site). Is there any big blocking issue or chunks of work that might benefit from contributors?
I've recently added the ability to render arbitrary Dataview queries to markdown in 0.5.39; support for DataviewJS queries will take a bit longer and involves a breaking change the render API (unfortunately). The exact feature work to get this across the finish line:
Obsidian plugin experience is probably useful for implementing both of these, but they do not require huge Dataview code prior knowledge. Would be happy to accept contributions which implement any of this.
Is there any ETA for any of the two proposed features? For me the most useful one would be automatic bulk freezing (I export my obsidian notes to my personal site). Is there any big blocking issue or chunks of work that might benefit from contributors?
I've recently added the ability to render arbitrary Dataview queries to markdown in 0.5.39; support for DataviewJS queries will take a bit longer and involves a breaking change the render API (unfortunately). The exact feature work to get this across the finish line:
- Track codeblock locations + types in the index. Doing this correctly is a bit hard since the Obsidian metadata cache will not track codeblocks inside of admonitions/callouts/quotes/lists correctly, but it will work fine for regular codeblocks.
- Implement a UI which implements the bulk freezing by tracking progress & listing all views to be exported.
- Figure out how to tell if a codeblock has previously been exported before (by looking for HTML comments), and replace instead of append if so.
- For "per-codeblock" exports, find a way to figure out which codeblock to replace and implement the logic that swaps the codeblock for the rendered markdown.
Obsidian plugin experience is probably useful for implementing both of these, but they do not require huge Dataview code prior knowledge. Would be happy to accept contributions which implement any of this.
@blacksmithgu Thanks for the detailed info and for the update. I will play around a bit with the code and let you know if I start making significant progress with it.
hello ! I tried to play with the dv.markdownTable I got it to work, but I can't find how to get the generated markdown code ? I would like the link to appear in the outgoing link, etc. Have I misunderstood something ?
Is the output being written to the file or is it part of a DVJS codeblock? You'll need the former for links to show up.
I used specifically:
let pages = dv.pages('"Periodic Notes 📅" or "Projets 🖇️"')
const table = dv.markdownTable(["Nom", "File"],
pages.filter(p => p.zettel)
.map(p => [p.zettel, "[[" + p.file.name + "#Zettel 🍃]]"])
)
dv.paragraph(table);
So, I get a table displayed markdown style, but there is no way to retrieve the markdown code itself, at least not that I understood. Also, when I say "outgoing links", the links in the markdown table does work fine. I'm speaking about "outgoing links" in the right pane, or in the "graph analysis" plugin, or just the stock graph view, for instance. Since the links are not hard coded in the note, I suppose they are not detected, hence the question "can i burn out the table in hard markdown code ?"
I hope this clarify my problem. The documentation says about dv.markdownXXX "Functions which render to plain Markdown strings which you can then render or manipulate as desired." But I can't find how to get those. Thank you !
@avillani can you look at dv.queryMarkdown()
? It looks super useful in your case.
@avillani can you look at dv.queryMarkdown()? It looks super useful in your case.
@AB1908 , I'm not @avillani , but are you suggesting dv.queryMarkdown() can be used to write the markdown strings to file? How is this done? Another plugin? I don't see anything in the Dataview documentation about writing outputs to file. Apologies if I am overlooking something obvious.
You can use templater
Would yo have an exemple ? I'm not proficient in javascript, I'm reading the obsidian doc, but I cannot reproduce the examples. Concerning templater, I suppose I have to use tp.file.cursor_append, but I cannot figure out how arguments are passed from the query to the templater function.
edit:
I finally managed to get something like:
<%*
const dv = this.app.plugins.plugins["dataview"].api ;
const te = await dv.queryMarkdown("LIST FROM #foo")
await tp.file.cursor_append(te.value);
%>
<%*
const dv = this.app.plugins.plugins["dataview"].api
const te = await dv.queryMarkdown('TABLE tags as Tags, file.folder as Dossier FROM -\"Zettelkasten 🗃️\" AND -\"Periodic Notes 📅\" WHERE contains(modified,"tp.date.now(\"dddd Do MMMM YYYY\")") ')
tp.file.cursor_append(te.value);
%>
But sometimes, the output appears and disappear right away. Pressing undo shortcut does show me that it has been written because it show up.
Shouldn't need to do .cursor_append
. Try using tR += te.value
instead.
Thank you for your help For future reference, I got to something like:
<%*
const dv = this.app.plugins.plugins["dataview"].api ;
const te = await dv.queryMarkdown(`TABLE parent as Parent, siblings as Siblings, tags as Tags FROM "Zettelkasten 🗃️" WHERE contains(modified, "`+ tp.date.now('Do MMMM YYYY') + `") or contains(file.name, "`+ tp.date.now('YYYYMMDD') + `")` )
tR += te.value
%>
fantastic !
You can also directly use the dataview markdown functions.. Here is an example pulling a list of ingredients from the yaml frontmatter and turning each one into a link in a list.
<%*
const dv = app.plugins.plugins["dataview"].api;
const ingr = tp.frontmatter.ingredients;
let sortedIngr = ingr.sort();
var arr = [];
for (i=0; i < sortedIngr.length; i++) {
var name = sortedIngr[i];
var link = '[[' + name + ']]'
arr.push(link);
}
let dvList = dv.markdownList(arr);
tR += dvList;
%>
Currently planning on just using HTML comment syntax for now, since it doesn't look too bad:
<!-- Start Dataview Export: ... --> - Value - Value - Value <!-- End Dataview Export: ... -->
@blacksmithgu 'm just catching up on this thread so apologies if I have mistakenly missed something ;-)
Just as a side note: would it be possible to replace the 'HTML' comment syntax by its Markdown equivalent (%) ? The rational behind that is that I'm using Obsidian as a source feed to Sphinx. It would therefore be nice if the lines commented out could directly be managed by the Sphinx Myst-Parser extension. Thanks for your contribution and support anyway :-)
I've just found this plugin called "file cooker", it can copy dataview result links to clipboard and seems okay for me. 😊
Any updates on this?
I use dataview to automatically link to related notes for my weekly meeting. However, over time I edit files and the metadata falls out of line with the search which means notes from weeks or months ago are different from at the time. I know it's a niche issue but would love for optional static results.
Sadly not yet. Given that we now have methods that output markdown, it is much cleaner through templater than it previously was.
That's a shame. One of my use cases is that I scripted reading my vault non-interactively, and passing them to other (non-Obsidian) actions. It works well for simple logic.
Dataview is wonderful to query my notes in advanced ways, and it's not worth re-implementing it outside of Obsidian.
Being able to non-interactively extract data from my vault via Dataview via scripts, will unlock a lot of power.
I was hoping to use "burned out" views as something I can then query and pass to other tools.
Does this look like it helps? You could have a note whose contents are replaced by a templater script.
Indeed, I also have a similar use case for non-interactively generating derived data files that are easily expressed using DataView.
Alas, I think the only solution is to create your own plug in which triggers the internal APIs for data view and then dump the internal data structures that come back into a file. I don't think you can use templater in a non-interactive way.
On Oct 8, 2022, at 12:36 PM, Ioannis Stoilis @.***> wrote:
That's a shame. One of my use cases is that I scripted reading my vault non-interactively, and passing them to other (non-Obsidian) actions. It works well for simple logic.
Dataview is wonderful to query my notes in advanced ways, and it's not worth re-implementing it outside of Obsidian.
Being able to non-interactively extract data from my vault via Dataview via scripts, will unlock a lot of power.
I was hoping to use "burned out" views as something I can then query and pass to other tools.
— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1272384404, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL226PLMOPAINPCORYDBTWCHEKPANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.
You are correct.
I appreciate the confirmation. So much has been said about templater and data view capabilities I was not certain.
I am not a javascript programmer, so I have found it easier to write export functions in python. But this has the disadvantage of deriving directly from raw text, so I need to replicate functionality that already exists within dataview as I do this.
One day, a JS programmer is going to need to do export, and they will create a plugin or fork of dataview for this purpose. I think dataview is the most powerful query language over the obsidian data vault, thus having non-JS access to this API will happen one day.
Thanks for a great plugin!
—dan
On Oct 8, 2022, at 10:40 PM, AB1908 @.***> wrote:
You are correct.
— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1272461467, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL222GETCGHHQ3D32R52TWCJLGFANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.
+1 for this feature. Currently I am putting my cursor over a field name, selecting the output, copying and pasting elsewhere in the note to keep a text copy. Entirely manual.
Just wanted to voice that templater works fine by me for generating fixed text / table results. For complicated queries it might be difficult if you don't have any programming experience, but I had little to no javascript experience, just plenty of python experience, and found it easy to pick up by looking at examples online.
Just to reiterate my initial use case: I want to be able to, either manually or (ideally) via some sort of timer, convert a dataview query into its markdown, making it static.
A lot of the suggestions here are around converting dataview into markdown, which is great, but it is the entire idea of being able to go from dataview to its markdown at a time I decide and ideally without me intervening or running a script that is the feature request, for me.
I agree that being able to do this without another plugin/script would be nice, but templater does accomplish the function of converting a dataview query into markdown, making it static at a time that you decide. It is manual rather than with a timer and it does require the prep of having a template or script ready to do that conversion, but in use it is just hitting Alt-E and then the clicking which conversion that I want.
Jack, this is an embarrassingly newbie kind of question, but are there API docs for any of this stuff? I am used to finding a list of classes/methods with a phrase or couple sentences explaining their functioning. I actually hacked a plugin myself by editing the source of another plugin…. kind of blindly, by googling for relevant snippets of source code. It was the weirdest programming experience of my life. I have never felt so out in the wilderness! But maybe I just missed the correct place for docs…. (not just for templater, but for the obsidian app itself)
it was a kind of programming by analogy and cut-paste mashup!
thanks!
On Oct 24, 2022, at 9:57 AM, Jack Reid @.***> wrote:
I agree that being able to do this without another plugin/script would be nice, but templater does accomplish the function of converting a dataview query into markdown, making it static at a time that you decide. It is manual rather than with a timer and it does require the prep of having a template or script ready to do that conversion, but in use it is just hitting Alt-E and then the clicking which conversion that I want.
— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1289326534, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL222WMWR4KGLP6EDGNVLWE25YZANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.
I don't know exactly what your use case is, but here are some good references.
Dataviewjs docs: https://blacksmithgu.github.io/obsidian-dataview/api/code-reference/ Dataviewjs examples: https://forum.obsidian.md/t/dataviewjs-snippet-showcase/17847
Templater docs (including examples): https://silentvoid13.github.io/Templater/introduction.html
Obsidian plugin API docs: https://github.com/obsidianmd/obsidian-api Example plugin: https://github.com/obsidianmd/obsidian-sample-plugin In general, I recommend taking a look at the "Advanced Topics": https://help.obsidian.md/Obsidian/Index
Hope that helps!
Here's my flow for some inspiration.
Thanks Jack!
On Oct 25, 2022, at 7:17 AM, Jack Reid @.***> wrote:
I don't know exactly what your use case is, but here are some good references.
Dataviewjs docs: https://blacksmithgu.github.io/obsidian-dataview/api/code-reference/ https://blacksmithgu.github.io/obsidian-dataview/api/code-reference/ Dataviewjs examples: https://forum.obsidian.md/t/dataviewjs-snippet-showcase/17847 https://forum.obsidian.md/t/dataviewjs-snippet-showcase/17847 Templater docs (including examples): https://silentvoid13.github.io/Templater/introduction.html https://silentvoid13.github.io/Templater/introduction.html Obsidian plugin API docs: https://github.com/obsidianmd/obsidian-api https://github.com/obsidianmd/obsidian-api Example plugin: https://github.com/obsidianmd/obsidian-sample-plugin https://github.com/obsidianmd/obsidian-sample-plugin In general, I recommend taking a look at the "Advanced Topics": https://help.obsidian.md/Obsidian/Index https://help.obsidian.md/Obsidian/Index Hope that helps!
— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1290631553, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL222F7PQWYMHFQQV4ZKLWE7TWTANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.
Thanks @jackreid and @avillani. your posts helped me to figure out how to “burn out” the dv results into my daily notes with the templater.
Awesome!!!
A small nugget for those heading down a similar rabbithole, and using Templater & Dataview to achieve this:
This line:
const dv = app.plugins.plugins["dataview"].api;
Grants access to the Plugin API of dataview, which is a subset of the full Dataview API, lacking rendering functions like dv.header(..)
and dv.paragraph(...)
. That's fine - we can do all our rendering using Templater, but I got stuck for a while 'cos those were the functions I was using to test things!
Here's the plugin api code, so you can see what you have access to: https://github.com/blacksmithgu/obsidian-dataview/blob/829b01364f1cfd3ba3f66bb02e46b490292da06a/src/api/plugin-api.ts
Many thanks to those above for paving the way.
Hello, I am trying to use Templater to query Dataview and burn out out a task list into Markdown. I am 99% of the way there, but am falling at the final hurdle! I am very bad at JavaScipt, so I wonder if anyone can help?
My goal is to retrieve a task list that contains all the tasks that mention the current file.
This code works, showing all the tasks from pages tagged #journal:
<%*
const dv = app.plugins.plugins["dataview"].api;
const te = await dv.queryMarkdown("TASK FROM #journal");
tR += te.value;
%>
But this code (intended to filter those tasks by mentions of the current page) throws up an error, even though the same query works when using Dataview normally:
<%*
const dv = app.plugins.plugins["dataview"].api;
const te = await dv.queryMarkdown("TASK FROM #journal WHERE contains(text, this.file.name)");
tR += te.value;
%>
Any ideas / help greatly appreciated! :)
@simoncamp I wonder if there are context issues with this
.
Try `TASK FROM #journal WHERE contains(text, ${tp.file.title})`
@AB1908 Thanks for your reply. That's actually one step closer: before I couldn't get any WHERE contains()
to work, but with the ` characters instead of " surrounding it, I can now get it to match a specific string (any idea why this made a difference?).
But unfortunately with the ${tp.file.title}
variable it just renders nothing - although templater is no longer throwing an error.
I wonder what the right variable is? I've tried <% tp.file.title %>
too but that doesn't work either.
@AB1908 Fixed! That was the right variable, but it needed to be surrounded by " as it is supposed to be a text string. Final code is:
<%*
const dv = app.plugins.plugins["dataview"].api;
const te = await dv.queryMarkdown(`TASK FROM #journal WHERE contains(text, "${tp.file.title}")`);
tR += te.value;
%>
Thanks SO MUCH for your help!
Ah whoops, missed that since I didn't test that myself.
Alright, so I'm finally around to looking at this again. I have two modes which I am planning on supporting directly right away:
- Preview mode: All dataviews will support both "Export to Markdown" and "Freeze as Markdown", which (1) copy markdown for you to paste, and (2) directly replace the query block with markdown. This requires manual action to freeze but at least enables the core functionality.
- Bulk Freezing: Dataview will provide a simple UI for replacing ALL queries in your vault with their static results in one atomic operation - best for if you are migrating away from Obsidian or Dataview but want to keep all of your tables and indices. This will also support the ability to keep the existing Dataview block and just append the result as Markdown after the block.
These two scenarios should unblock the most important use-cases: data archiving, basic view support in publish and GitHub, migrating away from Dataview, and even being able to see important Dataview-based links in the graph. I can also consider implementing "Bulk Freezing" as a simple automatic job that runs at user-defined intervals or on important operations.
Hoping this feature will arrive, it would be very nice
I've started using dataviews in my daily notes via the daily note template. One dataview I'm using lists all the notes I edit that day:
That ends up inserted in my daily note as a dataview that matches any files edited today. It's very useful. However, if I edit a file after that day, the list stops showing it, even though I edited a file on that day. Because I'd love to have a history of the files I've worked on for any given day, I need some way to "burn out" a dataview—effectively replace its code fence with its markdown result.
I can see this being a button that appears on hovering the code fence when in preview mode, or even a command in the command palette that operates on the dataview under the cursor. Even better would be a way to bake this into the dataview so that it happens automatically after a certain threshold, e.g.
but I have no idea if such a thing is technically feasible, as I haven't dug too deeply into Obsidian's internals. Either way, a way to go from a dataview query to its hardcoded result would be very useful to freeze documents in time.