Jones-match commented 5 months ago

(wukong) root@autodl-container-46914989bd-be6f2b06:~/autodl-tmp/v3/v3wukong# PYTHONPATH="." python src/preprocess/dataset_generator.py /root/miniconda3/envs/wukong/lib/python3.8/site-packages/pytorch_lightning/metrics/init.py:43: LightningDeprecationWarning: pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5 rank_zero_deprecation( Global seed set to 7 testcases: 100%|██████████████████████████████████████████████████████████████| 20175/20175 [17:35<00:00, 19.11it/s] total 999630 XFGs! xfgs: 0%|▎ | 4930/1005201 [00:00<02:00, 8322.60it/s] Traceback (most recent call last): File "src/preprocess/dataset_generator.py", line 176, in xfg_unique_paths = unique_data(config.dataset.name, config.data_folder) File "src/preprocess/dataset_generator.py", line 51, in unique_data xfg_dict = unique_xfg_sym(xfg_paths) File "/root/autodl-tmp/v3/v3wukong/src/utils.py", line 141, in unique_xfg_sym ln_md5 = getMD5(str(xfg.nodes[ln]["code_sym_token"])) KeyError: 'code_sym_token'

在第一步可以使用PDG正常生成XFG，但在第二步的时候，还是会出现找不到code_sym_token属性。想请教一下应该如何解决

Jones-match commented 3 months ago

joern 是直接在README.md 中给出的地址下载的源码https://github.com/ives-nx/dwk_preprocess/tree/main/joern_slicer/joern

在将joern的源码下载后，复制到wukong 项目的根目录下然后将根目录下的./config/dwk.yaml 中的joern_path 修改

# preprocess keys
joern_path: "/root/autodl-tmp/v3/v3wukong/dwk_preprocess-main/joern_slicer/joern/joern-parse"
split_token: false

修改这些之后运行项目

不知道是否缺少了其他配置步骤，希望得到老师指点

mcf20 commented 3 months ago

joern 是直接在 README.md 中给出的地址下载的源码 https://github.com/ives-nx/dwk_preprocess/tree/main/joern_slicer/joern

在将joern的源码下载后，复制到wukong 项目的根目录下然后将根目录下的./config/dwk.yaml 中的joern_path 修改
# preprocess keys
joern_path: "/root/autodl-tmp/v3/v3wukong/dwk_preprocess-main/joern_slicer/joern/joern-parse"
split_token: false
修改这些之后运行项目

不知道是否缺少了其他配置步骤，希望得到老师指点

感谢您的回复这是在ubuntu里面跑的么

Jones-match commented 3 months ago

是的，ubuntu18.04 cuda11.1

Jones-match commented 3 months ago

服务器

lsq163@126.com

664730 commented 2 months ago

(wukong) root@autodl-container-46914989bd-be6f2b06:~/autodl-tmp/v3/v3wukong# PYTHONPATH="." python src/preprocess/dataset_generator.py /root/miniconda3/envs/wukong/lib/python3.8/site-packages/pytorch_lightning/metrics/init.py:43: LightningDeprecationWarning: pytorch_lightning.metrics.* module has been renamed to torchmetrics.* and split off to its own package (https://github.com/PyTorchLightning/metrics) since v1.3 and will be removed in v1.5 rank_zero_deprecation( Global seed set to 7 testcases: 100%|██████████████████████████████████████████████████████████████| 20175/20175 [17:35<00:00, 19.11it/s] total 999630 XFGs! xfgs: 0%|▎ | 4930/1005201 [00:00<02:00, 8322.60it/s] Traceback (most recent call last): File "src/preprocess/dataset_generator.py", line 176, in xfg_unique_paths = unique_data(config.dataset.name, config.data_folder) File "src/preprocess/dataset_generator.py", line 51, in unique_data xfg_dict = unique_xfg_sym(xfg_paths) File "/root/autodl-tmp/v3/v3wukong/src/utils.py", line 141, in unique_xfg_sym ln_md5 = getMD5(str(xfg.nodes[ln]["code_sym_token"])) KeyError: 'code_sym_token'

在第一步可以使用PDG正常生成XFG，但在第二步的时候，还是会出现找不到code_sym_token属性。想请教一下应该如何解决

请问 KeyError: 'code_sym_token'这个问题您解决了吗？

bysota commented 1 month ago

USE_CPU = cpu_count()

USE_CPU = 2

jumormt / DeepWukong

Symbolize and Split Dataset步骤时遇到了KeyError: 'code_sym_token' #19

服务器

USE_CPU = cpu_count()