clip-retrievel end2end throws "TypeError: sequence item 0: expected str instance, list found" on Windows

ghost commented 2 years ago

Running Windows 10,

0it [00:00, ?it/s]File sharded in 0 shards Downloading starting now, check your bandwidth speed (with bwm-ng)your cpu (with htop), and your disk usage (with iotop)! 0it [00:00, ?it/s] Traceback (most recent call last):
File "d:\miniconda3\envs\ldo\lib\runpy.py", line 194, in _run_module_as_main return _run_code(code, main_globals, None,
File "d:\miniconda3\envs\ldo\lib\runpy.py", line 87, in _run_code exec(code, run_globals)
File "D:\miniconda3\envs\ldo\Scripts\clip-retrieval.exe__main__.py", line 7, in File "d:\miniconda3\envs\ldo\lib\site-packages\clip_retrieval\cli.py", line 16, in main fire.Fire(
File "d:\miniconda3\envs\ldo\lib\site-packages\fire\core.py", line 141, in Fire component_trace = _Fire(component, args, parsed_flag_args, context, name)
File "d:\miniconda3\envs\ldo\lib\site-packages\fire\core.py", line 466, in _Fire component, remaining_args = _CallAndUpdateTrace(
File "d:\miniconda3\envs\ldo\lib\site-packages\fire\core.py", line 681, in _CallAndUpdateTrace component = fn(*varargs, **kwargs)
File "d:\miniconda3\envs\ldo\lib\site-packages\clip_retrieval\clip_end2end.py", line 37, in clip_end2end clip_inference(
File "d:\miniconda3\envs\ldo\lib\site-packages\clip_retrieval\clip_inference\main.py", line 40, in main input_dataset = list(braceexpand(input_dataset))
File "d:\miniconda3\envs\ldo\lib\site-packages\braceexpand__init.py", line 98, in return (_flatten(t, escape) for t in parse_pattern(pattern, escape))
File "d:\miniconda3\envs\ldo\lib\site-packages\braceexpand\init__.py", line 216, in _flatten s = ''.join(l)
TypeError: sequence item 0: expected str instance, list found

I think this has something to do with how parameters are handled on windows, I was getting a similar error before when using the example from https://github.com/rom1504/img2dataset/blob/main/dataset_examples/laion-aesthetic.md

img2dataset --url_list laion2B-en-aesthetic --input_format "parquet"\ --url_col "URL" --caption_col "TEXT" --output_format webdataset\ --output_folder laion2B-en-aesthetic-data --processes_count 16 --thread_count 64 --image_size 384\ --resize_only_if_bigger=True --resize_mode="keep_ratio" --skip_reencode=True \ --save_additional_columns '["similarity","hash","punsafe","pwatermark","aesthetic"]' --enable_wandb True

--save_additional_columns was not working and had to be removed

lonleyodd commented 2 years ago

excuse me, have you solved it?

rom1504 commented 2 years ago

The solution is to use a better shell. Wsl for example

mathyouf commented 2 years ago

I am having the same issue running this on Ubuntu using ZSH.

lizhiustc commented 2 years ago

Same error on ubuntu

m-bain commented 1 year ago

Same here :(

mathyouf commented 1 year ago

For my exactly situation (not having captions) adding --enable_text=False solved my issue. Hope this helps someone, likely not the solution to everyone's problem, but perhaps related.