search

RichardHGL / WSDM2021_NSM

Improving Multi-hop Knowledge Base Question Answering by Learning Intermediate Supervision Signals. WSDM 2021.
130 stars 22 forks source link

preprocess_step1 takes too long time #12

Open qiaopr opened 2 years ago

qiaopr commented 2 years ago

Hi,How fast are you preprocess_step1.py files running?Mine has take several hours to process only 150 piece of data. It's unbeliviable!

RichardHGL commented 2 years ago

Did you modify any parameters? Such as graph size. I think it won't be so slow. When I processed the data, I just keep it run in the backend and get the data the next day. I guess any dataset can be processed in one day.

qiaopr commented 2 years ago

I exactly followed the preprocess >Freebase>README.md.I didn't modify any parameters. The data I'm processed is CWQ.

LLLiaomeng commented 2 years ago

I also faced this problem.

JasonCen-sweetdreams commented 2 years ago

Also ran into this problem. It seems that the preprocess_step1 takes a lot of time to output only a few, while my CPU, GPU and memory usage are in a healthy state. sad :(

RichardHGL commented 2 years ago

Okay, I'll check this problem next month. You can also try to look into ppr_util.py, I think the majority of time is spent on the calculation of ppr for every graph.

RichardHGL commented 2 years ago

The preprocessed datasets can be found in Readme of this repo. If you find any possible ways to improve the efficiency of preprocessing, please kindly let me know.