Open bingkunyao opened 2 months ago
Do you have a tmp
subdirectory in the of the directory where you have the dataset you are "indexing" / using as a reference for checking for duplication?
If you look here in the script creating the suffix array, there is a relative path for tmp/out.table.bin
:
I guess it is a bug, probably it should have been \tmp
as in other places in the codebase. I also bumped into this problem yesterday, maybe it helps you or other people trying to use the tool.
Just same as "A full end-to-end single file deduplication example" in readme file, When I tried to run "bash scripts/deduplicate_single_file.sh /home/user/deduplicate-text-datasets/test_reduce/testfile.csv /home/user/deduplicate-text-datasets/test_reduce/test_result 400 4", I encountered the error below:
I am sure that the file and the path mentioned both exists. I also run "ulimit -Sn 100000" but it dis not work. Mention that the CSV file is large (About 2.0 GB). Could anyone help me with this problem?