Closed markus-stoll closed 1 year ago
Hi @markus-stoll can you try to follow those instructions : https://visual-layer.readme.io/docs/v1-api#fastdup.engine.Fastdup.run namely run with input_dir
pointing to a folder and subset
argument is a list of the subset of files you like to work on from this folder. Let us know if this works.
If that does not work, you can try v0 API (https://visual-layer.readme.io/docs/v02xx-api) namely run with
import fastdup
fastdup.run(input_dir=[put your list here], work_dir='/path/to/work/dir')
Let us know if this works. Once we get your response we will try to clarify documentation to make it simpler to use when using huggingface datasets.
Hi @markus-stoll we found the source of the issue and we will release version 0.928 fix by tomorrow.
Sounds good. You could also checkput my pull request #188. It fixed the problem forme. Maybe it helps.
@markus-stoll you are a mind reader! I did almost 100% the same fix:
What happened?
Using a list of local files in fastdup.create( input_dir=input_files, ... ) causes an Assertion Error:
AssertionError: Failed to extract any files from list
I will provide a fix in a Fork and a Pull Request.
What did you expect to see?
######################################################################################## Dataset Analysis Summary:
.....
What version of fastdup were you runnning on?
0.927
What version of Python were you running on?
Python 3.9
Operating System
Ubuntu
Reproduction steps
Install HuggingFace library of datasets
pip install datasets
Relevant log output
Attach a screenshot [Optional]
No response
Contact Details [Optional]