Open WenxiongLiao opened 2 years ago
The spj_rand
is the model you'll want to use for end-to-end evaluation. This is what is reported in the paper and has the negative random sampling augmentation in training. The variant spj
has no negative sampling and doesn't perform well when testing with external data.
The ssg+spj
model is taking the output of the SSG model and inputting this into the pre-trained spj_rand
model.
The atomic/join is orthogonal to these query types and wasn't reported in the ACL paper. Just the VLDB and arxiv pre-prints. You could add a feature to each instance counting the number of facts used. If this is == 1, then it is atomic, if it is >= 1 it is join. This should be sufficient.
2.I'm going to reproduce the results of the ACL paper, I have executed 'bash scripts/experiments_ours.sh v2.4_25' and 'bash scripts/experiments_baseline.sh v2.4_25'. After execute 'python -m neuraldb.final_scoring' I got this results:
The result here is only minmax/set/bool/count. How to output the results related to Atomic and join?