Model dimension error in inference

[glennliu]

Hi,

I finally generated a few scene graph data from SGAlign.

In runing your inference, scripts/inference_reg.py. It generate error at gnn.py.

    def forward(self, node_feat, edge_index, edge_feat):

        # MLP for node and edge features
        if self.use_feat_mlp:
            node_emb = self.node_feat_mlp(node_feat)
            edge_emb = self.edge_feat_mlp(edge_feat)

It says emb feature dimension error. And I checked the feature dimension as below.

Thanks

[glennliu]

The error occurs here, https://github.com/dfki-av/sg-pgm/blame/233d852cbc8e40de051352da55d49b3e4466461f/modules/layers/gnn.py#L55C2-L55C2

[EryiXie]

Hi, it looks like the input edge feature's dimension is incorrect. It expects the input edge feature to be Nx82, as the config file specified. I am not sure where the 210x18 comes from. You can take these lines In the dataset.py as reference https://github.com/dfki-av/sg-pgm/blob/233d852cbc8e40de051352da55d49b3e4466461f/data/datasets.py#L268 . I hope this can help you.

[glennliu]

Hi

I checked the dataset.py. And I highlight the feature dimensions of the realted features as below.

Is the edge_attr looks correct to you? Do you need to load any relationship dictionary like bag of words? I'm thinking maybe some steps in my data generation is inconsistent with yours and cause the error.

Thanks

[glennliu]

I my dataset does not provide any relationship information, can I skip the relationship embedding process? Do you have such configured interface?

Thanks

[glennliu]

Hi

I tryied the debug from data generation in 3RScan. So, the related process can be summarized as below.

1. preprocess.py in 3RScan. At line316 It encode a entity_edge_feats in shape of $N \times 9$. N is the number of nodes in the graph. And the relationship.txt defines $9$ categories. Each node construct an entity_edge_feat, a feature vector with 9 length. It basically create a histogram that summarize the 9 relationship s associated with the node. This step looks fine to me. And all the edge relationships are set to None in my program.
2. data/datasets.py in SG-PGM, At line532. Here, your program load the previously saved edge features. And you construct a sg_pair from the loaded edge feature. sg_pair.edge_attr_q has a shape $(E_t,18)$, where $E_t$ is the number of edges in the target graph. Each edge feature concatenate the two relationship feature from the last step. So, each of them is a feature vector at $18$ length.

Both of the steps look fine to me. Except I set the relationsihp to None. It may be different with your original setting.

So, can you help debug?

Thanks

[EryiXie]

Hi,

I think I might know what is the problem here.

You see in SGAligner's config file https://github.com/sayands/sgaligner/blob/9e31be38dc09993eb81956babc80ce7ebebe60f6/configs/scan3r/scan3r_ground_truth.yaml#L23-L26 the relationship's dimension is definitely not 9 or 18.

I think the 9 or 18 may come from the metafile of 3RScan: relationship.txt. There are two versions of it, one has 8 simple relationships, and the other has 41. This is because the 3RScan's scene graph annotation has two versions, the complex original version and the simplified version from SceneGraphFusion. This also confused me last year for quite a while.

You might need to look at the 3RScan repository to see what went wrong there.

Best regards

[glennliu]

Hi,

I have two more question regarding theScan3RDataset in the data/dataset.py.

1. What is insg_match? I assume it is the ground-truth match between graph nodes. It is in the shape $M \times 2$. Each row is a pair of [SRC_IDX, REF_IDX]. Is it right?
2. How to construct a tot_bow_vec_obj_attr_feats? It should be constructed from the vertex attribution $\mathcal{V}$. But in your paper, I cannot find how to construct $\mathbf{X}$ from $\mathcal{V}$. In my dataset, I have {semantic label, bounding box, point cloud} for each node. How should I construct the initial node features $\mathbf{X}$ in my dataset?

Thanks& Regards

[EryiXie]

Hi,

I have two more question regarding theScan3RDataset in the data/dataset.py.

1. What is insg_match? I assume it is the ground-truth match between graph nodes. It is in the shape M×2. Each row is a pair of [SRC_IDX, REF_IDX]. Is it right?
2. How to construct a tot_bow_vec_obj_attr_feats? It should be constructed from the vertex attribution V. But in your paper, I cannot find how to construct X from V. In my dataset, I have {semantic label, bounding box, point cloud} for each node. How should I construct the initial node features X in my dataset?

Thanks& Regards

Hi,

1. sg_match: yes, you are right.

2. For the mathematical notion of the graph G = (V, E, X, A), the V is just a set of nodes, likes V = {1,2,3,4,5,6} and the X is the node attribute e.g. $X \in R^{6\times C}$, for the 6 nodes and each node have an attribute vector with the dimension of C.

   In your case, I suggest that semantic label, bounding box, and point cloud should all be treated as node attributes, but I am not sure simply combining them into one vector is a good idea. Maybe you will need to balance and normalize these attributes first if you want to use them all for graph matching.

[glennliu]

I'm still confused about the node attribute feature $\mathbf{X}$. I understand a node attribute $\mathbf{x}_i \in R^{C}$. But how do you initialize it?

[EryiXie]

Oh, sorry I forget to answer that question. You can take a look at the SGAligner code to know more about the "bow_vec": https://github.com/sayands/sgaligner/blob/9e31be38dc09993eb81956babc80ce7ebebe60f6/preprocessing/scan3r/preprocess.py

[glennliu]

Thanks. It looks like it just make a histogram of semantic category. Bow is constructed from the histogram.