xionluhnis
commented
1 year ago I believe that the files in syntax are not used by the current code on Github.
It refers to what we mentioned in the discussion section 7:
From a program synthesis perspective, we have access to a set of constraints on valid instruction interactions (e.g. Stack is always paired with a Move instruction reaching it). This conditional dependency is referred to as context in semantic segmentation, and there have been many efforts to explicitly tackle this by Conditional Random Field (CRF) (Zheng et al., 2015; Chen et al., 2018; Rother et al., 2004). They clean up spurious predictions of a weak classifier by favoring same-label assignments to neighboring pixels, e.g., Potts model. For our problem, we tried a first-order syntax compatibility loss, but there was no noticeable improvement. However we note that (Yu & Koltun, 2016) observed that a CNN with a large receptive field but without CRF can outperform or compare similarly to its counterpart with CRF for subsequent structured guidance (Zheng et al., 2015; Chen et al., 2018). While we did not consider any CRF post processing in this work, sophisticated modeling of the knittability would be worth exploring as a future direction.
The contents of those files are co-occurrence matrices that we computed from the dataset with the intent of checking whether we could use that as a mechanism for regularizing the outputs. The rationale was that we know from the dataset what the likelihood of an output instruction is given its neighbors. Thus we would have hoped that this enables us to at least invalidate some pairs (i.e., it we know that a pair of neighboring instructions have 0 co-occurrence, then we can safely reject it).
Unfortunately, we did not end up with any significant improvement using these and thus we did not include the mechanism in this repository. But this was a short attempt: it may be that we did not try long enough and did not use the proper mechanism to integrate such information. I believe this can be used for at least cleaning the outputs, but we did not investigate further.
From what I remember:
- the files numbers
1to8represent neighborhoods of a pixel (i.e., which neighboring pixel) - the file contents correspond to a co-occurrence matrix
- it is 17x17 because there are 17 instructions we worry about
- the values are unnormalized occurrence counts
++AK
1033216625
May I ask what the data in the syntax folder in the dataset is used for, and what do these txt files represent and how are they generated?