search

VladimirMarkelov / ttdl

TTDL - Terminal Todo List Manager
MIT License
209 stars 17 forks source link

[Feature Request] Allow to rewrite the data before displaying by external commands via speical key/value tags #26

Closed eugnma closed 4 years ago

eugnma commented 4 years ago

Currently, it is hard to customize for ttdl, I suggest that allow to rewrite the data before displaying by external commands via speical key/value tags, e.g. the item This is an example tagpart1/tagpart2:value, it will call the external command ttdl-tagpart1-tagpart2 to rewrite the item, is it possible?

VladimirMarkelov commented 4 years ago

It looks feasible. But it is good to learn more about details and use cases.

As far as I understand, if there is a tag in a special format, before printing out the line, ttdl takes value, calls a program from tag name, and replace tag:value with output. Correct?

Questions:

  1. I'd like to see an example of the feature usage. E.g, it is interesting if the use case can be solved using environment variables and substituting them ina todo line (instead of heavy external app call)
  2. Performance-wise, if many lines contains such construction, it can be a bit slow. Though, I do not think anyone displays tens of todos that contains similar tags. But for a few lines it should not be a trouble
  3. ttdl-tagpart1-tagpart2 looks like too specific for ttdl. Can an arbitrary binary be called? E.g, git:branch as a tag.
  4. Taking into account previous question, it is good to have a special prefix/suffix/inner symbol to detect such replaceable tags automatically. E.g, $git:branch or $git:$branch(I am not sure if dollar sign is OK to use in tags - it is just an example). Maybe, an extra command line option to turn the feature on would be good to add.
  5. What if a tool's output contains a few lines? Join them with a space? Take only the first line?
  6. What to do if too fails(e.g, binary with such name does not exist)? Keep old tag? Show a placeholder?
eugnma commented 4 years ago

Good to hear that it's feasible to implement it, the behavior I though was like below:

Pre-condition

Assume the input and output for external commands is JSON format.

Behavior example

For example, the item This is an example tag1:value tag2p1/tag2p2:value, the steps for calling external commands as below:

  1. The first time pass the input {"description": "This is an example", "specialTags":[{"tag1": "value"}]} to ttdl-tag1, then it returns {"description": "This is an example", "specialTags":[{"tag1": "value"}]};
  2. The second time pass the input {"description": "This is an example", "specialTags":[{"tag1": "value"}, {"tag2p1/tag2p2": "value"}]} to ttdl-tag1p1-tag2p2, then it returns {"description": "This is an example", "specialTags":[{"tag1": "value"}, {"tag2p1/tag2p2": "value"}]}.

Please be noted that:

  1. The order calling external commands is same with the order of special tags;
  2. All field of output might be changed via external commands.

Usage example

The are some many calendars are using now, sometimes we may use it as a date, it may be added like This is an example due/calendar1:2020-02-02 rec/calendar1:1m, it the feature is implemented, then we can create two plugins ttdl-due-calendar1 and ttdl-due-calendar1 to handle these items.

Your Questions

  1. I'd like to see an example of the feature usage. E.g, it is interesting if the use case can be solved using environment variables and substituting them ina todo line (instead of heavy external app call) A: Please see Usage example;
  2. Performance-wise, if many lines contains such construction, it can be a bit slow. Though, I do not think anyone displays tens of todos that contains similar tags. But for a few lines it should not be a trouble A: I don't think performance is big problem as git also support the similar behavior. It's OK for me calling external commands only if special tags are start with a special char, e.g. !.
  3. ttdl-tagpart1-tagpart2 looks like too specific for ttdl. Can an arbitrary binary be called? E.g, git:branch as a tag. A: I think it's better to define the input and output format of external commands, so cannot be arbitrary binary.
  4. Taking into account previous question, it is good to have a special prefix/suffix/inner symbol to detect such replaceable tags automatically. E.g, $git:branch or $git:$branch(I am not sure if dollar sign is OK to use in tags - it is just an example). Maybe, an extra command line option to turn the feature on would be good to add. A: I was though the external commands will be: replace the / to - then prepend ttdl-. According the todo.txt standard, it only limit the char :.
  5. What if a tool's output contains a few lines? Join them with a space? Take only the first line? A: Please see Behavior example.
  6. What to do if too fails(e.g, binary with such name does not exist)? Keep old tag? Show a placeholder? A: I think it's fine to report as errors.
VladimirMarkelov commented 4 years ago

Thanks! It is clearer now. Yes, if ttdl- prefix is added always, it cannot be an arbitrary binary. Though, some stuff needs to be explained a bit more.

  1. When are external commands called? When running ttdl list? or ttdl list <special-option>?
  2. Should ttdl process all tags, or only those which have / in their names? The latter case may make sense if we transform the text when running simple list command
  3. What is "description" in your examples? The entire todo's text without tags or the text before the first tag?
  4. From your example I understand that every "plugin" gets its own tag and all previous ones. But due to input and output are the same in examples, it is not clear what a "plugin" receives: just list of all original tags+values, or the result of the previous call with added tag for the current "tag"?

Correct me, if I am wrong in my example (I assume that every "plugin" returns its entire argument with one or more fields changed):

  1. Original todo in todo.txt: "Fix bug due:2019-10-10 /due:2019-11-11 /overdue:2019-11-18"
  2. First tag with / is /due:2019-11, so the first "plugin" is ttdl-due and it gets the initial argument with description and the first tag: {"description": "Fix bug due:2019-10-10", "due": "2019-11-11"}
  3. Let's assume it returns {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11"}
  4. We take the result, add the tag for the second "plugin. So, the second "plugin" ttdl-overdue receives {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11", "overdue": "2019-11-18"} (or `{"description": "Fix bug due:2019-10-10", "due": "2019-11-11"}?).
  5. The seconds returns {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11", "overdue": "2019-Nov-18"}
  6. Since no "plugin" failed, ttdl combines all tags from last returned JSON and a user sees on the screen: Fix bug due:2019-10-10 due:2019-Nov-11 overdue:2019-Nov-18 (please, note that all / are removed as well).
  7. If any "plugin" failed, it shows original todo line and prints an error to stderr
eugnma commented 4 years ago

Thank you for your attention, I add some comments base on your comment, please correct me if I am wrong.

Your Questions

  1. When are external commands called? When running ttdl list? or ttdl list \<special-option>?

Both are OK for me.

  1. Should ttdl process all tags, or only those which have / in their names? The latter case may make sense if we transform the text when running simple list command

I think just process all tags start with ! can be used widely, which likes git style and vi style.

  1. What is "description" in your examples? The entire todo's text without tags or the text before the first tag?

From the todo.txt format standard, my understanding for the "description" is Review Tim's pull request for the item x 2011-03-02 2011-03-01 Review Tim's pull request +TodoTxtTouch @github, please correct me if I am wrong.

  1. From your example I understand that every "plugin" gets its own tag and all previous ones. But due to input and output are the same in examples, it is not clear what a "plugin" receives: just list of all original tags+values, or the result of the previous call with added tag for the current "tag"?

Please check Your example.

Your example

  1. Original todo in todo.txt: "Fix bug due:2019-10-10 /due:2019-11-11 /overdue:2019-11-18"

I would suggest use Fix bug due:2019-10-10 !due:2019-11-11 !overdue:2019-11-18.

  1. First tag with / is /due:2019-11, so the first "plugin" is ttdl-due and it gets the initial argument with description and the first tag: {"description": "Fix bug due:2019-10-10", "due": "2019-11-11"}

I would suggest save all special tags in speicalTags as a special tag can be named description, so it will be {"description": "Fix bug", "speicalTags":[{"due":"2019-10-10"},{"!due": "2019-11-11"}]}.

  1. Let's assume it returns {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11"}

I assume the ttdl-due is for changing the due time, so it will return {"description": "Fix bug", "speicalTags":[{"due":"2019-11-11"}]} or {"description": "Fix bug", "speicalTags":[{"due":"2019-11-11"},{"!due": "2019-11-11"}]} if we want to keep the tag from the command ttdl list.

  1. We take the result, add the tag for the second "plugin. So, the second "plugin" ttdl-overdue receives {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11", "overdue": "2019-11-18"} (or `{"description": "Fix bug due:2019-10-10", "due": "2019-11-11"}?).

Let's asset it returns {"description": "Fix bug", "speicalTags":[{"due":"2019-11-11"},{"!due": "2019-11-11"}]} from step 3, so the second plugin ttdl-overdue will receive {"description": "Fix bug", "speicalTags":[{"due":"2019-11-11"},{"!due": "2019-11-11"},{"!overdue":"2019-11-18"}]}.

  1. The seconds returns {"description": "Fix bug due:2019-10-10", "due": "2019-Nov-11", "overdue": "2019-Nov-18"}

I assume the ttdl-overdue aslo is for changing the due time, so it will return {"description": "Fix bug", "speicalTags":[{"due":"2019-11-18"},{"!due": "2019-11-11"}]} if we don't want to keep the tag from the command ttdl list.

  1. Since no "plugin" failed, ttdl combines all tags from last returned JSON and a user sees on the screen: Fix bug due:2019-10-10 due:2019-Nov-11 overdue:2019-Nov-18 (please, note that all / are removed as well).

It shows Fix bug due:2019-11-18 !due:2019-11-11 if the final result is from step 5.

If any "plugin" failed, it shows original todo line and prints an error to stderr

I agree with you.

VladimirMarkelov commented 4 years ago

Good idea about ! - I like it.

From the todo.txt format standard, my understanding for the "description" is Review Tim's pull request for the item x 2011-03-02 2011-03-01 Review Tim's pull request +TodoTxtTouch @github, please correct me if I am wrong.

I agree with you. The question was more about a case when tags are in the middle. But we can think of this case as of rare one and just take everything until the first tag.

The last questions (I hope it is the last one :) ):

What if every "plugin" will receive all the tags? instead of every "plugin" gets all previous tags and its own one? So, we can do this: generate JSON once, then pass it to plugin one by one, and every time pass to the next "plugin" the result of previous one. What we get at the end we use to construct the line for displaying.

Let me summarize the algorithm(assuming we create JSON once in the beginning):

  1. If a todo has no tags with leading "!" - print it as is.
  2. Otherwise, generate a JSON from all tags + description
  3. Call plugin one by one and pass them this dynamically changing("dynamically" - because every plugin may change it and return changed one) JSON
  4. After the last plugin finishes, construct the new todo line only from the final plugin result and display the new line
eugnma commented 4 years ago

I total agree with you, hope the custom way I was thinking you would like it.

What if every "plugin" will receive all the tags? instead of every "plugin" gets all previous tags and its own one? So, we can do this: generate JSON once, then pass it to plugin one by one, and every time pass to the next "plugin" the result of previous one. What we get at the end we use to construct the line for displaying.

Generate JSON once is easier, but it's better to keep the original sequence.

VladimirMarkelov commented 4 years ago

Generate JSON once is easier, but it's better to keep the original sequence.

Since `"speicalTags"`` is an array, all tags would be in the original sequence. The only difference is that the every plugin gets all the tags.

eugnma commented 4 years ago

At last, I would highlight that the external commands calling from ttdl should base on the lasted JSON, so for the item Fix bug due:2019-10-10 !due:2019-11-11 !overdue:2019-11-18:

  1. So the first time pass {"description": "Fix bug", "speicalTags":[{"due":"2019-10-10"},{"!due": "2019-11-11"},{"!overdue":"2019-11-18"}]} to ttdl-due;
  2. Let's assume ttdl-due changes due date and removes the !overdue tag;
  3. Then ttdl get the final result {"description": "Fix bug", speicalTags":[{"due":"2019-11-11"}]} and won't call ttdl-overdue.
VladimirMarkelov commented 4 years ago

Interesting feature. In this case we have to track already executed plugins (to avoid a forever loop if, e.g, a plugin returns the list shuffled or reversed) and check the result to skip removed tags.

Another thing came to my mind: a config file needs one more option. I think plugins can be either binaries or scripts. In case of scripts ttdl cannot just execute them(at least on Windows), so config should provide a name and options for shell to run commands. If config does not contain it, the shell should default to cmd.exe /c in Windows, and to sh -cu in other OS.

Ok, let's do it :)

VladimirMarkelov commented 4 years ago

By the way, since ! is used to detect plugins, do we still need to use / that should be replaced with -? Maybe we can use -, e.g !tagp1-tagp2:value instead of !tagp1/tagp2:value? So, we can just build a plugin name as ttdl-+tagName. Or / must be in tag name?

eugnma commented 4 years ago

In this case, we don’t need it.

Sent from my iPhone

On 12 Jan 2020, at 12:22, Vladimir Markelov notifications@github.com wrote:

 By the way, since ! is used to detect plugins, do we still need to use / that should be replaced with -? Maybe we can use -, e.g !tagp1-tagp2:value instead of !tagp1/tagp2:value? So, we can just build a plugin name as ttdl-+tagName. Or / must be in tag name?

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

VladimirMarkelov commented 4 years ago

Do you build TTDL from sources? Or you wait for crate.io to be updated?

I added the feature and tested it little on Windows(a new section "Custom formatting" appears in the README). So, I am not sure if it does what it was intended to do. That was why I have not created a new release and not updated the crate.

If you build from sources, please try the new version and share our experience. If you want to use cargo and an updated crate, I'll refresh the crates as soon as I test the app on Linux and build both Windows binary and deb-package.

eugnma commented 4 years ago

Great work! I've just built form source, everything is fine except tags are not used for the final display, for example, assume there is an item This is a test !due:2020-02-02 and the ttdl-due returns {"description": "This is a test", "specialTags": [{"due":"2020-02-02"}], I would like it displays:

# D P Created    Finished   Due        Threshold  Spent  Subject
-----------------------------------------------------------------
1                           2020-02-02                   This is a test
-----------------------------------------------------------------
1 todos (of 1 total)

But it actually displayed:

# D P Created    Finished   Due        Threshold  Spent  Subject
-----------------------------------------------------------------
1                                                        This is a test due:2020-02-02
-----------------------------------------------------------------
1 todos (of 1 total)
VladimirMarkelov commented 4 years ago

I see. That is because I call custom formatting when description is going to be printed. It is possible to allows anyone to rewrite any tag/column(except ID one - it does not make sense and it is dangerous). But with limitation: because column widths are calculated beforehand, the custom valued would be truncated to already calculated column width.

And it brings another question: since any column can be customized, does it mean that a plugin must get not only simple tags, but also "due", "created", "priority" and so on? In this case I had to add them to tag list manually because underlining library I use removes standard tags from both todo description and tag list.

eugnma commented 4 years ago

It is possible to allows anyone to rewrite any tag/column(except ID one - it does not make sense and it is dangerous). But with limitation: because column widths are calculated beforehand, the custom valued would be truncated to already calculated column width.

Yes, ID definitely cannot be changed. I think it's OK truncate the final result.

And it brings another question: since any column can be customized, does it mean that a plugin must get not only simple tags, but also "due", "created", "priority" and so on? In this case I had to add them to tag list manually because underlining library I use removes standard tags from both todo description and tag list.

According the todo.txt standard, my understanding is the special tags like due, t and rec belong to special tags, and priority should be a normal property since it is not a tag, please correct me if i am wrong.

BTW, I updated the previous comment related the output from ttdl-due which is not correct, please check, thanks!

VladimirMarkelov commented 4 years ago

Thanks for an example. I have not implemented replacing anything except description yet. I'll fix it.

Yes, you are right: due is a special tag, while priority is not. I confused them because the library I use removes due from tag list as it does for priority. We can pass priority, creation & finished dates, and done mark as optional array. In this case even done mark x can be customized as, e.g, √. So, I can generate optional items and pass them in the same JSON.

VladimirMarkelov commented 4 years ago

@eugnma

I've updated the code. Changes:

You can try the new version. Meantime I need to refactor the code a bit (no changes in algorithm) and update README to describe new JSON field and customizing gothchas

eugnma commented 4 years ago

I have just tried with the latest version, seems like the description is missed, not sure I did something wrong.

  1. A normal item without special tags starting with !:
$ cat todo.txt
This is a test due:2020-01-02
$ ttdl
# D P Created    Finished   Due        Threshold  Spent  Subject
-----------------------------------------------------------------
1                           2020-01-02                   
-----------------------------------------------------------------
1 todos (of 1 total)
  1. A item with special tags starting with !: 
    
$ cat todo.txt
This is a test !due:2020-01-02
$ cat /usr/local/bin/ttdl-due 
#!/usr/bin/env sh

echo '{"description":"This is a test","specialTags":[{"due":"2020-02-02"}]}' $ ttdl

D P Created Finished Due Threshold Spent Subject

1 2020-01-02

1 todos (of 1 total)

VladimirMarkelov commented 4 years ago

I'm sorry. It looks like you tried refactored code. Thank you! I found the bug: TTDL read all dates from "created", and because it was missing in result JSON it defaulted to value from original TODO.

Fixed in the latest sources:

eugnma commented 4 years ago

Cool, it likes a magic!

VladimirMarkelov commented 4 years ago

Thanks!

It seems it is time to update docs and package on crates.io :)

eugnma commented 4 years ago

Can I confirm is the JSON passed via stdin to external plugins? I cannot get it from stdin or parameters.

VladimirMarkelov commented 4 years ago

It seems there was a bug in passing arguments to a plugin. It should be fixed in the latest sources. Testing a naive script that prints dates of the current year without YYYY part:

$ cat todo.txt 
2020-01-13 fix bug #2 !cal:test
2019-12-20 2020-01-06 fix bug #1 !cal:test

$ cat ttdl-cal 
#!/bin/sh
echo "$*" | sed 's/2020-//'

$ ./target/release/ttdl list
# D P Created    Finished   Due        Threshold  Spent  Subject
-----------------------------------------------------------------
1     01-13                                              fix bug #2 !cal:test
2     01-06      2019-12-20                              fix bug #1 !cal:test
-----------------------------------------------------------------
2 todos (of 2 total)
eugnma commented 4 years ago

Thanks, I can get it from the first parameter now, but I would suggest pass the it via stdin as it easier for testing (e.g. piping).

echo '{"description":"This is a test","specialTags":[{"due":"2020-02-02"}]' | ttdl-plugin1 | ttdl-plugin2
VladimirMarkelov commented 4 years ago

I'll look into it. Passing an argument was easier to implement at first sight. But it appears that it has its drawbacks. Maybe using stdin would be better.

VladimirMarkelov commented 4 years ago

I've changed interaction with plugins: now they must read from stdin - no argument provided in command line. On Windows it seems working.

Now I don't need to escape anything manually :), so it seems a better way