Special characters

[o-pteron]

Hello todotxt

Thank you for creating this wonderful tool. I have problems with todo.txt-cli concerning sorting and special characters like æ, ø and å. Please bear with me as I'm neither a programmer nor a native english speaker.

When listing tasks the sorting seems broken when the task contains special characters like æ, ø and å. Even when I've used the following from Tips & Tricks in todo.cfg: export TODOTXT_SORT_COMMAND='env LC_COLLATE=C sort -k 2,2 -k 1,1n'

Reproduce the error: 1) Insert a task containing "ÆØÅ" in todo.txt, like this:

(A) Task
(B) Task 2
(C) Task 3
Task 4
ÆØÅ

2) Tasks will be listed like this:

opteron@thor:~/K-Drive/todo.txt$ t
1 (A) Task 1
5 ÆØÅ
2 (B) Task 2
3 (C) Task 3
4 Task 4
--
TODO: 5 of 5 tasks shown

I would expect tasks without priority being placed below prioritized tasks.

My setup: KDE Neon latest version

opteron@thor:~/K-Drive/todo.txt$ t -V
TODO.TXT Command Line Interface v2.10

First release: 5/11/2006
Original conception by: Gina Trapani (http://ginatrapani.org)
Contributors: http://github.com/ginatrapani/todo.txt-cli/network
License: GPL, http://www.gnu.org/copyleft/gpl.html
More information and mailing list at http://todotxt.com
Code repository: http://github.com/ginatrapani/todo.txt-cli/tree/master

todo.txt downloaded from https://github.com/todotxt/todo.txt-cli/releases/download/v2.10.0/todo.txt_cli-2.10.tar.gz

opteron@thor:~/K-Drive/todo.txt$ tail -n 5 ~/.bashrc 
PATH=$PATH:"/home/opteron/K-Drive/todo.txt/"
export TODOTXT_DEFAULT_ACTION=ls
alias t='todo.sh -d /home/opteron/K-Drive/todo.txt/todo.cfg'
source todo_completion
complete -F _todo t

opteron@thor:~/K-Drive/todo.txt$ cat todo.cfg
# === EDIT FILE LOCATIONS BELOW ===

# Your todo.txt directory
#export TODO_DIR="/Users/gina/Documents/todo"
export TODO_DIR=$(dirname "$0")

# Your todo/done/report.txt locations
export TODO_FILE="$TODO_DIR/todo.txt"
export DONE_FILE="$TODO_DIR/done.txt"
export REPORT_FILE="$TODO_DIR/report.txt"

# You can customize your actions directory location
#export TODO_ACTIONS_DIR="$HOME/.todo.actions.d"

# == EDIT FILE LOCATIONS ABOVE ===

# === COLOR MAP ===

## Text coloring and formatting is done by inserting ANSI escape codes.
## If you have re-mapped your color codes, or use the todo.txt
## output in another output system (like Conky), you may need to
## over-ride by uncommenting and editing these defaults.
## If you change any of these here, you also need to uncomment
## the defaults in the COLORS section below. Otherwise, todo.txt
## will still use the defaults!

 export BLACK='\\033[0;30m'
 export RED='\\033[0;31m'
 export GREEN='\\033[0;32m'
 export BROWN='\\033[0;33m'
 export BLUE='\\033[0;34m'
 export PURPLE='\\033[0;35m'
 export CYAN='\\033[0;36m'
 export LIGHT_GREY='\\033[0;37m'
 export DARK_GREY='\\033[1;30m'
 export LIGHT_RED='\\033[1;31m'
 export LIGHT_GREEN='\\033[1;32m'
 export YELLOW='\\033[1;33m'
 export LIGHT_BLUE='\\033[1;34m'
 export LIGHT_PURPLE='\\033[1;35m'
 export LIGHT_CYAN='\\033[1;36m'
 export WHITE='\\033[1;37m'
 export DEFAULT='\\033[0m'

# === COLORS ===

## Uncomment and edit to override these defaults.
## Reference the constants from the color map above,
## or use $NONE to disable highlighting.
#
# Priorities can be any upper-case letter.
# A,B,C are highlighted; you can add coloring for more.
#
# export PRI_A=$YELLOW        # color for A priority
# export PRI_B=$GREEN         # color for B priority
# export PRI_C=$LIGHT_BLUE    # color for C priority
# export PRI_D=...            # define your own
# export PRI_X=$WHITE         # color unless explicitly defined

# There is highlighting for tasks that have been done,
# but haven't been archived yet.
#
export COLOR_DONE=$LIGHT_GREY

# There is highlighting for projects and contexts.
#
export COLOR_PROJECT=$CYAN
export COLOR_CONTEXT=$PURPLE

# === BEHAVIOR ===

## customize list output
#
# TODOTXT_SORT_COMMAND will filter after line numbers are
# inserted, but before colorization, and before hiding of
# priority, context, and project.
#
#export TODOTXT_SORT_COMMAND='env LC_COLLATE=C sort -f -k2'
export TODOTXT_SORT_COMMAND='env LC_COLLATE=C sort -k 2,2 -k 1,1n'
# TODOTXT_FINAL_FILTER will filter list output after colorization,
# priority hiding, context hiding, and project hiding. That is,
# just before the list output is displayed.
#
#export TODOTXT_FINAL_FILTER='cat'

opteron@thor:~/K-Drive/todo.txt$ bash --version
GNU bash, version 4.3.48(1)-release (x86_64-pc-linux-gnu)
Copyright (C) 2013 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>

This is free software; you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Thank you for your time Sincerely Kasper

[munkee]

I can't reproduce.

:~$ t -V | head -1
TODO.TXT Command Line Interface v2.10

:~$ cat tmp/sort.test.txt
(A) Task
ÆØÅ
(B) Task 2
(C) Task 3
Task 4

:~$ t lf tmp/sort.test.txt
1 (A) Task
3 (B) Task 2
4 (C) Task 3
5 Task 4
2 ÆØÅ
--
SORT: 5 of 5 tasks shown

:~$ cat .todo/config
export TODO_DIR="$HOME/Dropbox/todo"
export TODO_FILE="$TODO_DIR/todo.txt"
export DONE_FILE="$TODO_DIR/done.txt"
export REPORT_FILE="$TODO_DIR/report.txt"
export TMP_FILE="/tmp/todo.tmp"
export TODOTXT_DEFAULT_ACTION=ls

I was going to suggest you try a stripped down config file, but I can't reproduce even with your provided config file. That makes me think it's a locale issue.

What does sort --version say?

What does locale say?

[o-pteron]

Thank you for your help.

As requested:

opteron@thor:~$ sort --version
sort (GNU coreutils) 8.25
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later <http://gnu.org/licenses/gpl.html>.
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law.

Written by Mike Haertel and Paul Eggert.
opteron@thor:~$ locale
LANG=en_DK.UTF-8
LANGUAGE=en_US
LC_CTYPE="en_DK.UTF-8"
LC_NUMERIC="en_DK.UTF-8"
LC_TIME="en_DK.UTF-8"
LC_COLLATE="en_DK.UTF-8"
LC_MONETARY="en_DK.UTF-8"
LC_MESSAGES="en_DK.UTF-8"
LC_PAPER="en_DK.UTF-8"
LC_NAME="en_DK.UTF-8"
LC_ADDRESS="en_DK.UTF-8"
LC_TELEPHONE="en_DK.UTF-8"
LC_MEASUREMENT="en_DK.UTF-8"
LC_IDENTIFICATION="en_DK.UTF-8"
LC_ALL=en_DK.UTF-8

I'll play around with some virtual machines and see if I can reproduce the error on other systems.

[o-pteron]

I found the problem. LC_ALL=en_DK.UTF-8 seems to mess up the sorting. You can reproduce the error using: export LC_ALL=en_DK.UTF-8

[munkee]

Yep, that will do it. How is LCALL being set? It shouldn't be set in any configuration files and should only be set by/for individual scripts or programs that need it. It overrides all of the other LC settings. See this stackoverflow answer https://unix.stackexchange.com/a/87763

[o-pteron]

LC_ALL is not being set by todo.txt, I might have done it myself. Thank you very much for your help and the link about LC_ALL

[munkee]

Happy to help. Gave me a good excuse to learn more about locale variables. If you've fixed it on your end you can close this issue.