taxonkit lca gives error `[ERRO] bufio.Scanner: token too long' sometimes

[taylorreiter]

Prerequisites

• [x] make sure you're are using the latest version by taxonkit version: I'm using taxonkit v0.14.1 (installed via conda)
• [x] read the usage

Describe your issue

• [x] describe the problem: taxonkit lca gives error [ERRO] bufio.Scanner: token too long on some files. I ran this command on 64k of these files, and it gave this error for 71 of them. I've attached one file below.
• [x] provide a reproducible example

# install taxonkit
conda install taxonkit
# download taxid -> lineage file. required for taxonkit
wget https://ftp.ncbi.nlm.nih.gov/pub/taxonomy/taxdump.tar.gz
tar xf taxdump.tar.gz
taxonkit lca --data-dir . -i 2 -s ";" -o test_lca.txt test.txt

test.txt: A TSV-formatted input file

[shenwei356]

It's due to some long rows, similar to https://github.com/shenwei356/seqkit/issues/214.

[shenwei356]

Hi Taylor, thanks for reporting this. I've just increased the default size of the line buffer from 4096 to 1M. It works now.

• taxonkit_darwin_arm64.tar.gz
• taxonkit_linux_amd64.tar.gz
• taxonkit_darwin_amd64.tar.gz

If the error still occurs, you can use a bigger buffer size:

  -b, --buffer-size string   size of line buffer, supported unit: K, M, G. You need to increase the
                             value when "bufio.Scanner: token too long" error occured (default "1M")

[taylorreiter]

thank you so much @shenwei356! I've tried this on both Mac and Linux and it solved the problem. Thank you!!!!!

[shenwei356]

Please don't hesitate to let me know if you encounter any additional problems.