"Burning out" dataviews

kevboh commented 3 years ago

I've started using dataviews in my daily notes via the daily note template. One dataview I'm using lists all the notes I edit that day:

list from "some-folder"
where file.mtime.day = date({{date:YYYY-MM-DD}}).day and file.mtime.month = date({{date:YYYY-MM-DD}}).month and file.mtime.year = date({{date:YYYY-MM-DD}}).year

That ends up inserted in my daily note as a dataview that matches any files edited today. It's very useful. However, if I edit a file after that day, the list stops showing it, even though I edited a file on that day. Because I'd love to have a history of the files I've worked on for any given day, I need some way to "burn out" a dataview—effectively replace its code fence with its markdown result.

I can see this being a button that appears on hovering the code fence when in preview mode, or even a command in the command palette that operates on the dataview under the cursor. Even better would be a way to bake this into the dataview so that it happens automatically after a certain threshold, e.g.

list from "some-folder"
where file.mtime.day = date({{date:YYYY-MM-DD}}).day and file.mtime.month = date({{date:YYYY-MM-DD}}).month and file.mtime.year = date({{date:YYYY-MM-DD}}).year
burnout after {{date:YYYY-MM-DD}}

but I have no idea if such a thing is technically feasible, as I haven't dug too deeply into Obsidian's internals. Either way, a way to go from a dataview query to its hardcoded result would be very useful to freeze documents in time.

blacksmithgu commented 3 years ago

It's doable, and I've had a few people ask about it for another application (Obsidian Publish). Will add support for it in an upcoming release (likely by having it replace or append the fully rendered HTML for tables, and Markdown for everything else).

kevboh commented 3 years ago

Sounds neat. Let me know how I can help, happy to dig into the code.

THeK3nger commented 3 years ago

I was writing a separate feature request but I think this fits what I'd like. I think the possibility to "permanently expand dataview's queries" in the markdown source it is useful in many scenarios.

Not all dataview queries needs to be dynamically updated. Sometimes I just want to generate a list/table of notes on the fly. Like in kevboh use case.
It will make dataview more in agreement with the philosophy of "everything is just markdown" and "future-proof". Users should not worry that in 10 years dataview will be not working/maintained anymore. In any moment I could "make permanent" any query.
Better integrations with third party apps. For instance, I may use dataview to populate some tables (or whatever new amazing view we will have in the future), and then complete the editing of my work on Ulysses (or other editors). In this case, the ability to copy the "markdown version" of my query would be a must.

So, in short, I fully agree with the need of a feature like this.

(btw, just for completeness, this feature request may be related to #99)

kevboh commented 3 years ago

Yeah, I would love for this to exist—keep finding uses for it. @blacksmithgu if you have an idea of how it could trigger, I'd be happy to try and PR it myself—been meaning to dig into the codebase anyway…

blacksmithgu commented 3 years ago

This has been on my backlog for a while but I have never managed to get around to it yet - @kevboh if you find the time to take a look at it I'd be happy to help scope it and review it.

jpalharini commented 3 years ago

I'd love to have this feature and, if possible, be able to use it "automatically" when my notes are committed to Github.

ghost commented 3 years ago

As cool as this plugin is, it's not very helpful for me unless I can use it to publish results (unless there is a way to include rendering from dataview as part of the build step in a js based framework like nextjs/gatsby/sveltekit?)

pcuellar commented 3 years ago

@blacksmithgu @kevboh I'm about to release a large vault with several dataview queries that I'd love to include. Curious to hear if there's any progress on this feature. Thanks!

dylan-k commented 2 years ago

I would also like to voice support for dataview to have a "static output" option. This could be particularly useful for use within templates, where the dataview query could be run so that the note results with the output of the query within it. For some use cases (archival), it's in keeping with the spirit of markdown to have note contents that will be readable in any context, not just Obsidian/Dataview, for years to come. In any case, thank you for the incredibly useful tool you're providing for the community. It's extremely helpful.

blacksmithgu commented 2 years ago

I think you can combine Templater + DataviewJS to do queries and dynamically generate Markdown tables + lists for this, but first class support is definitely a good idea.

The ways I'm planning on this interaction being possible are via:

A context menu in preview mode for views that allows you to replace a view with it's output in either (1) fancy dataview output via HTML or (2) as a plain dataview table / list.
A special view property which Dataview understands to be a plain Markdown output; the query block will remain, but outputs will be given in plain Markdown in a specially marked region directly above or below the codeblock.

jimbrig commented 2 years ago

Any updates or workaround for this issue?

This is my current workflow:

Lists

Copy the display mode version of the list.
Copy into editor as links in URI format.
Use the wikilinks to mdlinks converter plugin to manually adjust links to be markdown links.
Find and replace all app://obsidian.md/ and manually change any remaining issues to get normal wikilinks.

Tables

Open Dev-tools and navigate to the HTML Object and copy its contents.

Utilize this online tool: Convert HTML Table to Markdown Table - Table Convert Online to then convert the HTML to a Markdown/HTML table.

Copy Back into Vault as Markdown or HTML (but losing all wikilink functionality).

It's not ideal but for my use case this functionality is crucial (i.e. need the vault to render properly both on GitHub for navigation as well as show the tables and lists on Obsidian Publish).

Can we revert back to the avirut/obsidian-query2table Plugin for referencing the desired functionality?

evantravers commented 2 years ago

I’d love to bump this idea. I do a lot of generating summaries in quarterly / weekly notes… burning in the note to markdown gives the file more permanence and portability.

Lawmoviefan commented 2 years ago

Bump

mdbraber commented 2 years ago

Thanks for the workaround @jimbrig. Any ideas on a more permanent solution @blacksmithgu ?

maxhq commented 2 years ago

In case anyone is interested, this is a working Templater template that uses DataView:

<%*
const dv = this.DataviewAPI
const escapePipe = s => new String(s).replace(/\|/, '\\|') // required for links in Markdown table
/*
  You may want to collect your utilities in a central Javascript module below
  the Obsidian vault, e.g. in a file lib/utils.js.
  The file can be included via
    const { escapePipe } = require(app.vault.adapter.basePath + "/lib/utils.js")
*/
%>

| Project | Tasks |
| ------- | ----- |
<%
// Note that dv.table() cannot be used as it creates HTML but we want Markdown.
dv.pages("!#template")
  .where(p => p.doctype == 'project') // a custom attribute, specified like "doctype:: project"
  .map(p => {
    let project = escapePipe(dv.fileLink(p.file.path))
    let tasklist = p.file.tasks.map(t => `${t.text} ${escapePipe(t.link)}`).join('<br>')
    return `|${project}|${tasklist}|`
  })
  .join("\n")
%>

reorx commented 2 years ago

Here's my approach of getting files created during two dates, it can be used to replace https://github.com/trydalch/obsidian-daily-activity, which does not have the ability to specify a certain date.

Just like the previous reply, it's an example for anyone who's interested in a work-around before the "Buring out" feature is out.

<%*
/* config */
const IGNORE_DIR_REGEXS = [
  /^40 Daily Notes/,
]

/* utils */
const linkTailRegex = /\.md\|[^\]]+/
const removeLinkTail = s => new String(s).replace(linkTailRegex, '')

/* main logic */
const dateFormat = 'yyyy-MM-dd'
const dv = this.DataviewAPI
let startDate = await tp.system.prompt('Start date', dv.date('today').toFormat(dateFormat))
if (!startDate) {
  return
}
startDate = dv.date(startDate)
let endDate = await tp.system.prompt('End date (optional)')
console.log('endDate', endDate)
if (endDate) {
  endDate = dv.date(endDate)
} else {
  endDate = startDate.plus(dv.duration('1 day'))
}
console.log('dv', dv, startDate, endDate)

tR += `Files created at ${startDate.toFormat(dateFormat)}~${endDate.toFormat(dateFormat)}:\n`
const out = dv.pages()
  .where(p => {
    if (IGNORE_DIR_REGEXS.filter(v => v.test(p.file.path)).length > 0) {
      return false
    }
    return p.file.ctime >= startDate && p.file.ctime < endDate
  })
  .map(p => {
    return `- ${removeLinkTail(p.file.link)}`
  })
  .join("\n")
tR += out
%>

I used some methods that are not documented in DataView, including date.toFormat, date.plus, dv.duration, there's much more to find in console.log(dv, date)

The result is like:

add1989 commented 2 years ago

+1

I have read so much about DataView and I really really want to start using it, but as I want my notes to be software-agnostic this is the one feature I am waiting for.

P.S. I'm an IT professional and I do realise how cliché that sounds, but in this case it is genuinely true 😄

blacksmithgu commented 2 years ago

Alright, so I'm finally around to looking at this again. I have two modes which I am planning on supporting directly right away:

Preview mode: All dataviews will support both "Export to Markdown" and "Freeze as Markdown", which (1) copy markdown for you to paste, and (2) directly replace the query block with markdown. This requires manual action to freeze but at least enables the core functionality.
Bulk Freezing: Dataview will provide a simple UI for replacing ALL queries in your vault with their static results in one atomic operation - best for if you are migrating away from Obsidian or Dataview but want to keep all of your tables and indices. This will also support the ability to keep the existing Dataview block and just append the result as Markdown after the block.

These two scenarios should unblock the most important use-cases: data archiving, basic view support in publish and GitHub, migrating away from Dataview, and even being able to see important Dataview-based links in the graph. I can also consider implementing "Bulk Freezing" as a simple automatic job that runs at user-defined intervals or on important operations.

ceciltech commented 2 years ago

I was just searching for this an hour ago and was disappointed not to find anything. I am glad so glad I tried re-searching and finding this posted about 1/2 an hour after my first search. Your plugin is what makes obsidian what it is for so many people and this was the only shortcoming I had found with it. So glad to see you picked up this feature request again!

kevboh commented 2 years ago

@blacksmithgu that sounds ideal. Im interested in the interval-based action, specifically if there's a way to narrow it to specific files, folders, or tags. But what you describe here is exciting!

blacksmithgu commented 2 years ago

Only applying the freezer to a specific dataview source (i.e., anything you can put in FROM) seems like a good feature.

mdbraber commented 2 years ago

That sounds excellent @blacksmithgu - especially having the ability to run a simple automatic job on specific intervals would be helpful because some dataviews would be time dependent and it would be great to be able to 'burn them' at specific times.

kevboh commented 2 years ago

Only applying the freezer to a specific dataview source (i.e., anything you can put in FROM) seems like a good feature.

Oh, interesting—I'm talking about the opposite, only applying the freezer to specific destination notes. My use case is in the original ticket; I want to be able to freeze a list of files I've edited in my entire vault (source "/") in a single specific note.

ooker777 commented 2 years ago

Just a genuine question: what would make this feature hard to implement? You generate the results, and you write it down. The first part is hard, but you have done it. Why would the second part be hard as well?

blacksmithgu commented 2 years ago

Oh, interesting—I'm talking about the opposite, only applying the freezer to specific destination notes. My use case is in the original ticket; I want to be able to freeze a list of files I've edited in my entire vault (source "/") in a single specific note.

That is what I mean by "source"; you would spefiy which files you want to freeze via a dataview source query. So if I want to freeze all of the contents of my daily notes, I could do it by the source "daily".

Just a genuine question: what would make this feature hard to implement? You generate the results, and you write it down. The first part is hard, but you have done it. Why would the second part be hard as well?

A simple implementation which just verbatim replaced your query codeblocks with their results would not be that hard, but it is also an inflexible solution. A much better and more general solution is to output the results above/below the codeblock - but then when I run the freezer again, how do I figure out what is plain Markdown that you wrote (or modified), and what is Markdown that Dataview generated? There needs to be some way of indicating "this was autogenerated by Dataview by this specific query and can be safely overwritten with new results".

ooker777 commented 2 years ago

Why not just using some comments to let users know that this portion will be overwritten in the next run?

anvimea commented 2 years ago

Very interested in this feature, would genuinely be transformative for several of my workflows.

oblinger commented 2 years ago

BlacksmithGU, I am watching for this new feature with excitement!

I cannot tell, will I be able to somehow trigger obsidian from outside in order to generate a markdown output from a Dataview computation?

My use case is the generation of markdown (and then ultimately CSV) from Obsidian data for auto upload. I need to keep a public summary computed from my vault current and online. I am hoping I in one way or another I will be able to compute this data from Dataview.

Awesome plugin!!

AB1908 commented 2 years ago

@oblinger you can already do this with some of the scripts mentioned in this issue. It's not particularly difficult but won't be very portable as you'd have to use node modules to write to a CSV, to my understanding.

Olyve commented 2 years ago

Why not just using some comments to let users know that this portion will be overwritten in the next run?

I have seen something similar to this before with content that is generated and will be replaced. Surrounding the generated code in comments immediately before and then immediately after. Really looking forward to seeing something like this implemented! It would certainly simplify some workflows for me!

blacksmithgu commented 2 years ago

Currently planning on just using HTML comment syntax for now, since it doesn't look too bad:

<!-- Start Dataview Export: ... -->

- Value
- Value
- Value

<!-- End Dataview Export: ... -->

ghost commented 2 years ago

Currently planning on just using HTML comment syntax for now, since it doesn't look too bad:


- Value
- Value
- Value

That would be an awesome solution!

ashrama commented 2 years ago

Currently planning on just using HTML comment syntax for now, since it doesn't look too bad:


- Value
- Value
- Value

That works for me :)

niel-lambrechts commented 2 years ago

Following this with interest. It would be wonderful to use dataview to generate a markdown extract from my various projects that can be used by the Kanban plugin, i.e. a daily generated Eisenhower matrix would be very useful!

AB1908 commented 2 years ago

As a reminder for those who need this urgently, you can still accomplish this with a templater script that hooks into DV's API.

ooker777 commented 2 years ago

@AB1908 if it doesn't take much of your time, can you make an example?

oblinger commented 2 years ago

+1

I have written python scripts to do this, just because it was not obvious how to do it using data views

On May 1, 2022, at 11:29 PM, Ooker @.***> wrote:

@AB1908 https://github.com/AB1908 if it doesn't take much of your time, can you make an example?

— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1114539529, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAIL223RGC4XJT6TEJKC4TTVH5Y5TANCNFSM4ZM6A22Q. You are receiving this because you were mentioned.

maxhq commented 2 years ago

@ooker777 @oblinger Please have a look above: https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1033178339 https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1034932229

monyarm commented 2 years ago

I don't understand how templater is a good workaround, i mean the template will only run when a note is created with it? If i ever had to recreate the table, i'd need to reinvoke the template. @oblinger mentioned having written a python script, could he perhaps share it?

AB1908 commented 2 years ago

Templater lets you invoke scripts multiple times. However, I don't see how Python is any better unless you're running it in the background as a cronjob or something.

monyarm commented 2 years ago

Cause I could run the script as part of generating static html.

On Thu, May 5, 2022, 11:22 PM AB1908 @.***> wrote:

Templater lets you invoke scripts multiple times. However, I don't see how Python is any better unless you're running it in the background as a cronjob or something.

— Reply to this email directly, view it on GitHub https://github.com/blacksmithgu/obsidian-dataview/issues/42#issuecomment-1119011475, or unsubscribe https://github.com/notifications/unsubscribe-auth/AAXHJKVMSNBZ47NIVKDFM7DVIQUZZANCNFSM4ZM6A22Q . You are receiving this because you commented.Message ID: @.***>

oblinger commented 2 years ago

@ooker777 I see the code above, but it simply says that it "works" without saying what it does, or when it executes. I think these are newbie confusions on my part. I don't yet understand the interaction of templater and data views. when I look at the code it seems you would want that code to rerun anytime the data in the generated table would change, but my understanding of templater is that it only runs once when the template is created... so I just don't know what is going on. (I DESPERATELY wish we had a page of a dozen or hundred examples of simple and complex templates and data views that were worked out completely, so a newbie could understand the effect of each. (including when or if they update.)

@monyarm and @ab1908 Indeed just as @ab1908 suggests, my python scripts run prior to every git commit (which I regularly run on my vault). My python scripts use egrep over my whole vault in order to find the patterns I want, then I use python to process the output of that grep to generate markdown tables which then become static pages in my vault. (these scripts are specific to the computation I am doing, but I can share them if it is valuable to see an egrep-python mashup.)

HERE ARE TWO USE CASES:

For example I use the H3 tag '### mYY-MM-DD' to indicate the notes from a meeting. I wedge these H3 sections into all sorts of documents, thus it is important that I have a SINGLE chronological list of all meeting in order to quickly get back to the notes from one that occurred 3 months ago. I don't want to wait for the dataview table to generate when I look at the table-of-contents page, since I do that SOOO regualarly, thus I use my python scripts pre-generate this table on each git commit. Someday it would be cool to be able to have a data view that does the same thing. But this would require a very intelligent dataview that knew when to recompute, or the ability to trigger a recompute when I wanted.

USE CASE NUMBER TWO

I unify ALL named entities into one massive namespace. and use CMD-O to "go to" each of them (kind of like spotlight on the Mac)

This include folders on my machine not in the vault, various applications on my machine, websites requiring login, etc.

Though some tortuously gross hacking, I have configure my obsidian to trigger actions upon opening special 'command' pages in my vault. Thus if I hit my "GO" key followed by f-o-o I will go to my foo markdown page, and if I type GO c-n-n my web browser will open up on that website.

If I had a way to trigger a shell script upon obsidian page load, that would still be hacky, but far less hacky. I have the sense that somehow I could co-opt obsidian's dynamic page compute functions to trigger some shell script action, but I don't know how...

AB1908 commented 2 years ago

Obs has an API for reading changes IIRC but IMO would be better if we had an API from DV that said index changed. That said, it's not strictly necessary.

blacksmithgu commented 2 years ago

Dataview does publish index changes - app.metadataCache.on("dataview:metadata-change", (type: string, file: TFile) => { ... }).

AB1908 commented 2 years ago

Oh nice

metawops commented 2 years ago

Forgive me – I haven't read all the comments in detail but ... I think my problem is related to what's being discussed here: the very first thought after I created my first table with Dataview was: "Now how can I export this into a CSV file for further visualisation outside of Obsidian?" (e.g., Power BI or sim.) Copy/pasting the preview is not an option here. Am I missing something? It's not possible right now, right? Is it already on a feature requests list? Should I write a feature request? Thanks so much and sorry again if this is off-topic here.

blacksmithgu commented 2 years ago

It's not directly possible outside of copy/paste, but this issue does indeed concern CSV export (as well as Markdown export).

metawops commented 2 years ago

Thanks for the confirmation, @blacksmithgu. Is a CSV export feature on the to do list?

Meanwhile, I found a nice workaround: open the Developer Tools in Obsidian, select the table, copy its HTML to the clipboard, paste it on https://tableconvert.com/html-to-csv and download the result to a csv file (or copy it to the clipboard). Maybe not a good solution for confidential data but for it's fine for now. ✅ 😉

ooker777 commented 2 years ago

open the Developer Tools in Obsidian, select the table

What command do you use to get the table on the devtool?

metawops commented 2 years ago

open the Developer Tools in Obsidian, select the table

What command do you use to get the table on the devtool?

Here on the Mac it looks like this and is the very first tool icon in the upper left corner of the Web Developer Tools:

Click it so that it becomes blue. Now hover with the mouse over the table and be precise to catch the whole table. Watch the window that appears during hovering that tells you which element you're over:

Now click and on the right, in the Developer Tools, the HTML of the Dataview <table> is selected:

Right-click and select "Copy element" and you're all set for the next step on tableconvert.com/html-to-csv:

Don't know how this all looks on Windows but it might be rather similar.

blacksmithgu / obsidian-dataview

"Burning out" dataviews #42

Lists

Tables