search

allenai / dolma

Data and tools for generating and inspecting OLMo pre-training data.
https://allenai.github.io/dolma/
Apache License 2.0
1.02k stars 108 forks source link

dedupe.documents.attribute_name does not work #166

Open mathCrazyy opened 6 months ago

mathCrazyy commented 6 months ago

96 has already been mentioned

and my version is tag 1.0.3, my command is :

dolma dedupe --documents "study/samples/v0/documents//*" --dedupe.documents.attribute_name 'bff_duplicate_documents' --dedupe.documents.key "metadata.id" --dedupe.skip_empty --bloom_filter.file /tmp/deduper_bloom_filter.bin --no-bloom_filter.read_only --bloom_filter.estimated_doc_count '6_000_000' --bloom_filter.desired_false_positive_rate '0.0001' --processes 188

but I also get the wrong info : thread '' panicked at src/deduper.rs:156:26: called Result::unwrap() on an Err value: Custom { kind: Other, error: "Failed to parse rule: --> 1:1\n |\n1 | metadata.id\n | ^---\n |\n = expected chain" } note: run with RUST_BACKTRACE=1 environment variable to display a backtrace

EdwinRou commented 5 months ago

got the same issue today, i dont know why too

Edit : nevermind the #96 was the solution