[IDEA] Extend add filter operator for date math

[stobot]

Idea would be extend the filter operator "add" via suffix like {{{ [[20211122]add:days[1]] }}} and do days, weeks, months, years. Probably might make sense to do hours and minutes too, though I don't have a need for this.

I tried to implement this as a separate operator adddays, addweeks, addmonths, addyears based on the addsuffix operator and posted my code here: https://talk.tiddlywiki.org/t/new-tiddler-with-due-date-set-for-next-week/1629/12

Thanks for your consideration

[AnthonyMuscio]

Also subtract see my reply here

also add:month add:quarter add:halfyear add:seconds add:hour add:day

and presumably matching subtract(s)

Meaning if the parameter is date related it treats the input as a date. Presumably it uses the “prefix” so it is valid for both YYYY0MM0DD and the full Tiddlydate YYYY0MM0DD0hh0mm0ss0XXX or parts there of.

[pmario]

TLDR; User-date-fields need a way to be stored as "real date objects" ... IMO we don't need to implement date calculations until we haven't solved this problem.

---

Adding "date calculations" isn't as simple as it may look like. ... Human calendar systems have some problematic edge cases. ... eg:

• leap years
• sommer time / winter time switching
  • rules are different on a regional basis
• time zones
• week numbers
  • different depending on where you are
• ...

These are the "prototype" functions we have at our disposal https://developer.mozilla.org/en-US/docs/Web/JavaScript/Reference/Global_Objects/Date

Sadly, there are no calculations included :/

---

Some of them would probably be nice if we could expose them to users eg: .toLocaleDateString ... Which imo would make it convenient for users to read dates...

On the other hand if we have "output" like this, we will also need to have "input" in the same way. ....

---

An other problem is, that TW at the moment only knows 2 fields that are "real dates". created and modified ... Even if they are displayed in a format like: YYYY0MM0DD0hh0mm0ss0XXX they are not stored that way. They are stored as a "real UTC date object"

User fields on the other hand are stored as strings in the DateFormat, which makes things extremely complicated.

• Users usually store dates in a localized format, without specifying the time-zone they are in

eg:

• The time of writing this text it's: 2021.11.23 12:05 for me.
• Which results in: created: 20211123110521480
• A tiddler at TW-com shows: 23rd November 2021 at 12:05pm
  • The TW core does the "time-zone correction", based on the browser settings
• 1 hour later TW-com would show: 23rd November 2021 at 01:05pm
  • based on the TW-template settings $:/language/Tiddler/DateFormat
  • I'm not used to it, so it is harder to read for me
    • I'd like to change it, but I don't want to change the UI language, which this output format is based on
    • So we would need a user-setting for date-formats

(just brainstorming)

---

Back to the date-formats. ... User-date-fields need a way to be stored as "real date objects" ... IMO we don't need to implement date calculations until we didn't solve this problem.

---

I did discuss this problem at: How could system-fields look like with 5.2.x #5805 .... The ideas there are probably a bit overkill, but I still would like to have it ; )

I think the idea posted at: https://github.com/Jermolene/TiddlyWiki5/discussions/5805#discussioncomment-903226 .. would still be worth considering.

#/_/mime/date: <info for a date-picker to specify>        <- jCal format https://datatracker.ietf.org/doc/html/rfc7265#section-3.6.5
date: application/calendar+json | -=-=-=-=-
["dtstart", {}, "date-time", "2012-10-17T12:00:00"],
["dtstamp", {}, "date-time", "2012-10-17T12:00:00Z"],
["dtend",
    { "tzid": "Europe/Berlin" },
    "date-time",
    "2011-10-17T13:00:00"
]

[Jermolene]

Hi @pmario

We do plenty of date/time processing in filters at the moment in the days operator, eachday, sameday. It's fairly simple but effective: we only parse TW format dates, and do our arithmetic with milliseconds-since-epoch values. We should continue to consider integrating something like moment.js but I don't think that it's worth it for the core.

Back to the date-formats. ... User-date-fields need a way to be stored as "real date objects" ... IMO we don't need to implement date calculations until we didn't solve this problem.

I disagree. Filter functions can parse string formatted dates as required without any difficulties. The special treatment of modified and created to make them read as JS date objects was a mistake for reasons that we've covered elsewhere, and so I wouldn't want to extend the problem to further fields.

[pmario]

We should continue to consider integrating something like moment.js but I don't think that it's worth it for the core.

moment.js imo is too big. ... There are smaller ones and most of them have a lot of code in it, that we already have in TW. We just solve the problems in different ways: eg: localisation.

---

Edit: I did have a look at: https://github.com/iamkun/dayjs a while ago. ... It seems to be much smaller and we may be able to borrow most of what we need.

[pmario]

I disagree. Filter functions can parse string formatted dates as required without any difficulties. The special treatment of modified and created to make them read as JS date objects was a mistake for reasons that we've covered elsewhere, and so I wouldn't want to extend the problem to further fields.

OK. ... But the time-zone problem doesn't go away. If we transfer tiddlers between users, that live in different time zones, we have to deal with that problem in one way or the other. ... created and modified are always UTC .. we know that.

To calculate with user-dates in the right way we need something similar.

An other big problem for me atm is, that the existing date-format is not human readable. IMO 20211123110521480 is a pain.

eg: 2021-11-23T11:05:21.480Z ... imo is still painful, but more readable and it is also ISO standardized

[CodaCodr]

FWIW, I'm with @pmario on this, 100%. I've experienced date issues managing customer systems across the globe (and dev code in multiple timezones, though that's less problematic) and quirky issues found only in a military context. Dates normalised to UTC is the best solution -- by far.

moment.js? Too big - like waaaay too big. Plus...

https://github.com/moment/moment/#project-status

[Jermolene]

I did have a look at: https://github.com/iamkun/dayjs a while ago. ... It seems to be much smaller and we may be able to borrow most of what we need.

Even that has a large API, with little prospect of the core ever using more than a tiny fraction of it.

But the time-zone problem doesn't go away

Adding date arithmetic operators doesn't affect the timezone problem. Everything is still implemented via those two functions $tw.utils.parseDate() and $tw.utils.stringifyDate(). I think the fix for the timezone issue is to extend our format to add a timezone marker.

An other big problem for me atm is, that the existing date-format is not human readable. IMO 20211123110521480 is a pain.

I think that's best discussed as a separate issue, it doesn't directly relate to the OP.

[pmario]

Even that has a large API, with little prospect of the core ever using more than a tiny fraction of it.

That's why I wrote at https://github.com/Jermolene/TiddlyWiki5/issues/6267#issuecomment-976435610 that they add a lot of code, that we already have in TW. ...

I wouldn't want to include the library, I would want to extract the code we actually need, and leave the library alone.

[stobot]

FWIW despite the current time-zone annoyance, to me there is a large convenience factor in how it currently works with a long number string. I (and I perceive to be other users that post) will commonly hand-enter dates and times. Sometimes I can just enter the YYYYMMDD and sometimes I can add hours, or minutes etc. depending on the need. I only mention because opening the can of worms of date vs. datetime (from my experience in other languages) gets complicated. Very interesting discussion though, and I appreciate your consideration of the feature!

[AnthonyMuscio]

I hope we can progress this a little. I would just add when adding new dates and adhering to the tiddler date/time stamp format, and knowing when to use UTC, and some date pickers is already enough for robust date handling as Eric's work https://tiddlytools.com/timer.html

A lot could be gained from adequately documenting some best practices for tiddlywiki's used across time zones and some simple add/subtract against code already in the core to handle the details. Evans formulae plugin did this as have a few others so it is achievable.

It would be good if we don't want these improvements in the core to have a core plugin for date more advanced manipulation and perhaps it could include a little more about internationalisation - perhaps with guidance for designing multi-language "capable" solutions and plugins. I may not speak French well, but if someone who does, knows how to convert my solution/plugin to include a French translation they may be happy to do so.

We seem to have plenty of English/German and English/French speakers here but other languages are no doubt neglected, I for one don't know how to build language aware solutions.

Perhaps one day more accessibility features or customisations as well.

[AnthonyMuscio]

Post script, if the days operator allowed us to extract the resulting date(s) ie the "limits used" it could be hacked to do basic add and subtract days if nothing else.

eg; days:all[-7]] would not look at existing date fields but all possible dates, and return the date 7 days ago.

with a months and years operators also with this feature we would at least have a way to achieve most simple date manipulation (without regard to UTC)

[pmario]

There is some new info about better date-handling and calculation. https://blog.openreplay.com/is-it-time-for-the-javascript-temporal-api .. It won't be available for some time in browsers, but IMO it would be a nice plugin once some of them support it.

[AnthonyMuscio]

Hi, I just want to bump this, the solution need not be in the core but it needs to be in the available core plugins, even if for now we use a larger than desirable library, pending a cut down version. Is there an existing solution we can add to the core plugins?

TiddlyWiki has a lot of strengths, even with dates, but not having some basic date maths is a weak link.

I would also point out the ability to simply add or subtract from a given tiddlywiki serial date would make date manipulation much more usable to everyday users; Keep in mind the data maths is on the absolute date value be it UTC or not.

Add/Subtract N, milliseconds, seconds, minutes, hours, days, weeks, months, quarters, years for example where any of these can exceed their typical values eg; add:seconds[90000] seconds is 25hours or 1 day 1 hour ago or ahead.

Ideally this would understand the most significant first allowing partial dates to be used eg add 25hours to YYYY0MM0DD will assume YYYY0MM0DD0hh0mm adding one day and one hour resulting in 202204250100000000 always returning the full serial number the user can truncate if needed. ie zero fill to right before adding/subtracting.

Perhaps if there is no input date, it uses now;

{{{ [add:hours[2]] }}} is two hours from now, {{{ [{!!date-field}add:hours[2]] }}} is two hours more than the date/time in date-field.

Note: I have a partial work around, just should someone arrive here, for standard periodical handling is to use a selected date or now time stamp eg weekly-review-date and in the list rather than look for dates falling on today use the days[-7]] to list tiddlers with the weekly-review-date 7 days old. On completing that item - time stamp it again with "now" and the "weekly review" list will not show it until another 7 days passes. This has the added advantage of indicating in the date field when last actioned.

You can subtract "day of month" from Now to get the number to use in the days operator. eg: if today is the 15th of the month use days[-15] to list dates from the beginning of the month. You could do something similar with day of year but need to handle the year change.