Handling empty field when joining tables

[MostafaYA]

Prerequisites

• [ ] make sure you're are using the latest version by csvtk version
• [ ] read the usage

Describe your issue

• [ ] describe the problem
• [ ] provide a reproducible example Hi, I wonder how to join multiple tables with frequently missing data. Header of all tables is the same. Here is an example:

echo -ne "1\t2\t3\t4\na\tb\n" > file1
echo -ne "1\t2\t3\t4\na\tb\tc\n" > file2
echo -ne "1\t2\t3\t4\na\tb\tc\td\n" > file3

 csvtk join -t file1 file2 file3
[ERRO] record on line 2: wrong number of fields

desired output

1       2       3       4
a       b       -       -
a       b       c       -
a       b       c       d

Thank you

I'm grateful to users who have greatly helped to report bugs and suggested new features.

I may respond to issues or fix bugs quickly, but I usually implement new features periodically (two or more weeks).

[shenwei356]

The result of echo -ne "1\t2\t3\t4\na\tb\n" is not a valid TSV file.

If they are, use csvtk concat.

[shenwei356]

You can fix the TSV file with code from @lskatz :

for i in *.tsv; do \
    tabs=$(cat $i | perl -F'\t' -lane 'print(scalar(@F))' | sort | uniq | sort -nr | head -n 1); \
    cat $i | perl -F'\t' -lane 'while(@F < '$tabs'){push(@F,"");} print join("\t", @F);' > tmp.tsv && mv -v tmp.tsv $i; \
done;

[shenwei356]

The new csvtk fix is v0.26.0 can also fix this.

$ echo -ne "1\t2\t3\t4\na\tb\n" | csvtk fix  -t | csvtk pretty -Ht -S bold
[INFO] the maximum number of columns in all 2 rows: 4
┏━━━┳━━━┳━━━┳━━━┓
┃ 1 ┃ 2 ┃ 3 ┃ 4 ┃
┣━━━╋━━━╋━━━╋━━━┫
┃ a ┃ b ┃   ┃   ┃
┗━━━┻━━━┻━━━┻━━━┛