Ying-1106
commented
3 months ago this is the code about generating Dataset:
base_dir = os.path.join(now_dir,'HGBl_base_dir')
construct the Ondiskdataset from existed dglgraph
graph_file_path = '/data/zzh/TEST_DIR/HGBl_dir/HGBl-amazon_DGLGraph.bin' HGBl_Graph = dgl.load_graphs(filename=graph_file_path)[0][0]
feature = HGBl_Graph.ndata['h'] product_feat_np = feature.numpy() product_feat_file = os.path.join(base_dir,'product_feat_file.npy') np.save(file=product_feat_file,arr=product_feat_np)
src,dst = HGBl_Graph.edges(etype=('product','product-product-0','product') ) src = src.numpy() dst = dst.numpy() P0P_npy = np.stack((src, dst)) P0P_npy_file = os.path.join(base_dir,'P0P.npy') np.save(file=P0P_npy_file,arr=P0P_npy)
src,dst = HGBl_Graph.edges(etype=('product','product-product-1','product') ) src = src.numpy() dst = dst.numpy() P1P_npy = np.stack((src, dst)) P1P_npy_file = os.path.join(base_dir,'P1P.npy') np.save(file=P1P_npy_file,arr=P1P_npy)
The edge information numpy files in train_set, val_set, and test_set have been stored locally, and each set includes the source and target node IDs of two types of edges, P-0-P and P-1-P
Train set
train_set_POP_path = "/data/zzh/TEST_DIR/HGBl_base_dir/train_set_P0P.npy" train_set_P1P_path = "/data/zzh/TEST_DIR/HGBl_base_dir/train_set_P1P.npy"
val set
val_set_POP_path = "/data/zzh/TEST_DIR/HGBl_base_dir/val_set_P0P.npy" val_set_P1P_path = "/data/zzh/TEST_DIR/HGBl_base_dir/val_set_P1P.npy"
test set
test_set_POP_path = "/data/zzh/TEST_DIR/HGBl_base_dir/test_set_P0P.npy" test_set_P1P_path = "/data/zzh/TEST_DIR/HGBl_base_dir/test_set_P1P.npy"
yaml_content = f""" dataset_name: HGBl_amazon_GB graph: nodes:
-
type: product num: 10099
edges: -
type: "product:product-product-0:product" format: numpy path: {os.path.basename(P0P_npy_file)}
-
type: "product:product-product-1:product" format: numpy path: {os.path.basename(P1P_npy_file)}
feature_data:
- domain: node type: product name: feat format: numpy in_memory: false path: {os.path.basename(product_feat_file)}tasks:
-
name: link_prediction num_classes: 100 train_set:
-
type: "product:product-product-0:product" data:
- name: seeds format: numpy path: {os.path.basename(train_set_POP_path)}
-
type: "product:product-product-1:product" data:
- name: seeds format: numpy path: {os.path.basename(train_set_P1P_path)}
validation_set:
-
type: "product:product-product-0:product" data:
- name: seeds format: numpy path: {os.path.basename(val_set_POP_path)}
-
type: "product:product-product-1:product" data:
- name: seeds format: numpy path: {os.path.basename(val_set_P1P_path)}
test_set:
-
type: "product:product-product-0:product" data:
- name: seeds format: numpy path: {os.path.basename(test_set_POP_path)}
-
type: "product:product-product-1:product" data:
- name: seeds format: numpy path: {os.path.basename(test_set_P1P_path)}
-
"""
metadata_path = os.path.join(base_dir, "metadata.yaml") with open(metadata_path, "w") as f: f.write(yaml_content)
dataset = gb.OnDiskDataset(base_dir).load() graph = dataset.graph.to(device) feature = dataset.feature.to(device) tasks = dataset.tasks link_pred_task = tasks[0]
datapipe = gb.ItemSampler(link_pred_task.train_set, batch_size=16, shuffle=True) datapipe = datapipe.copy_to(device) datapipe = datapipe.sample_uniform_negative(graph, 1) datapipe = datapipe.sample_neighbor(graph, [-1, -1,-1]) datapipe = datapipe.fetch_feature( feature, node_feature_keys={"product": ["feat"]} )
dataloader = gb.DataLoader(datapipe,num_workers=0)
-
Skeleton003
Rhett-Ying
github-actions[bot]
🐛 Bug
When I was using GraphBolt for a heterogeneous graph link prediction task, errors frequently occurred during batch generation. I created a dataset called HGBl-amazon, which includes one type of node: product, and two types of edges: Product-0-Product and Product-1-Product. I constructed a link prediction task and stored edge information in the train_set, val_set and test_set like GraphBolt examples. However, I always encountered errors while iterating through the dataloader.
code: self.model.train() loss_all = 0.0 for i, data in enumerate(self.train_dataloader): # this line always raise error.