Henry1iu
commented
2 years ago Hi,
Sorry for this late reply. I had a vacation for the Chinese Lunar New Year. I haven't met this error. before. Have you seen those preprocessed sequence files under the "interm_data" folder?
Besides, the error message implies the loading path was incorrect as it shouldn't contain two "train" folders. I will check my latest push for you.
Best regards, Jianbang
Hi! Thank you for your work.
I tried to preprocess the Argoverse dataset using the script
./scripts/preprocessing.bash. It took all 128 cores and about 96 hours to reach the last batch of the dataset and face the error below.Have you faced this issue before? Any suggestion on how to fix it?
Thank you!
Full trace:
(vectornet_env_py38_torch18) fkitashov@perelman:~/repositories/TNT-Trajectory-Predition$ ./scripts/preprocessing.bash 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████▉| 12871/12872 [62:14:23<00:17, 17.41s/it] Traceback (most recent call last): File "core/util/preprocessor/argoverse_preprocess_v2.py", line 414, in
for i, data in enumerate(tqdm(loader)):
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/tqdm/std.py", line 1180, in iter
for obj in iterable:
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 517, in next
data = self._next_data()
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1179, in _next_data
return self._process_data(data)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/dataloader.py", line 1225, in _process_data
data.reraise()
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/_utils.py", line 429, in reraise
raise self.exc_type(msg)
OSError: Caught OSError in DataLoader worker process 7.
Original Traceback (most recent call last):
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/_utils/worker.py", line 202, in _worker_loop
data = fetcher.fetch(index)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in fetch
data = [self.dataset[idx] for idx in possibly_batched_index]
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/torch/utils/data/_utils/fetch.py", line 44, in
data = [self.dataset[idx] for idx in possibly_batched_index]
File "core/util/preprocessor/argoverse_preprocess_v2.py", line 58, in getitem
df = copy.deepcopy(seq.seq_df)
File "/home/fkitashov/repositories/motion-prediction/argoverse-api/argoverse/data_loading/argoverse_forecasting_loader.py", line 77, in seq_df
return _read_csv(self.current_seq)
File "/home/fkitashov/repositories/motion-prediction/argoverse-api/argoverse/data_loading/argoverse_forecasting_loader.py", line 24, in _read_csv
return pd.read_csv(path, *args, kwargs)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/pandas/io/parsers.py", line 686, in read_csv
return _read(filepath_or_buffer, kwds)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/pandas/io/parsers.py", line 452, in _read
parser = TextFileReader(fp_or_buf, kwds)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/pandas/io/parsers.py", line 946, in init
self._make_engine(self.engine)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/pandas/io/parsers.py", line 1178, in _make_engine
self._engine = CParserWrapper(self.f, self.options)
File "/home/fkitashov/anaconda3/envs/vectornet_env_py38_torch18/lib/python3.8/site-packages/pandas/io/parsers.py", line 2008, in init
self._reader = parsers.TextReader(src, kwds)
File "pandas/_libs/parsers.pyx", line 382, in pandas._libs.parsers.TextReader.cinit
File "pandas/_libs/parsers.pyx", line 674, in pandas._libs.parsers.TextReader._setup_parser_source
OSError: [Errno 40] Too many levels of symbolic links: '/home/fkitashov/repositories/TNT-Trajectory-Predition/dataset/raw_data/train/train'