zhaone
commented
3 years ago Dear authors, thank you for sharing your code. I download the dataset from argoverse, then I want to preprocess the data. When I use 'python preprocess_data.py -m lanegcn', it take 5 hours but nothing output, the cpu occupancy rate is high but gpu is low. Thank you very much!
It is a problem but not an error.
The first thing you misunderstand is that the processing part should not consume GPU resource because it does not running on GPU (it has no much matrix computation) but on CPU. Therefore your CPU occupancy rate is high.
In my opinion, the data preprocessing strategy the author provided is not so elegant. I had preprocessed data using author's script, it did work but as you said, it toke a long time (merely evaluation set) and consumed much memory. The original strategy author use is saving all the preprocessed samples in memory and dumping a very very big output file at once, this causes 3 problems:
- As you said, you cannot see any output result until you finishing preprocessing all samples
- It consumes much memory (Author is working in Uber and he can use Uber's powerful server but these servers are not available for me: my server will report memory overflow when I preprocessed the train set)
- You can only use the whole dataset (hard to use partial dataset to debug)
Later I improve the data preprocess script my self. If you really want to preprocess the data by your self, I can give you some advice:
- Do not save outputs of all samples output in memory and dump them to a huge file at once. Process one (or several) sample, dump it out immediately. In this way, you can get the output files faster (many output files) and use part of the dataset for debug.
- Adopt multiprocessing. Different samples can be processed in parallel if you use the strategy above. This could speed up processing.
Hope authors can update preprocessing part.
walterchenchn
Dear authors, thank you for sharing your code. I download the dataset from argoverse, then I want to preprocess the data. When I use 'python preprocess_data.py -m lanegcn', it take 5 hours but nothing output, the cpu occupancy rate is high but gpu is low. Thank you very much!