What is the latest version of Smatch?

[goodmami]

Two related questions: What is the version of the current software? And what is considered the "official" fork (if there is one)? Some points to consider:

• setup.py says "1.0"
• The latest release on PyPI is 1.0.1 (the only differences between the version on PyPI and the current master branch are the changes introduced by #20 and #21)
• SemEval 2016 refers to versions 2.0, 2.0.1, and 2.0.2, but as far as I can tell those versions predate the 1.0.0 and 1.0.1 releases on PyPI
• There is also the ISI-NLP fork which adds an ILP solver but is otherwise behind this repo by 7 commits. It's inclusion in the ISI-NLP group makes it look like official version of Smatch, but it's not the version on PyPI.

@danielhers and @snowblink14 are listed as maintainers on PyPI, so what do you think? Should we bump the current version to, say, 2.0.3 so it's at least numerically consistent with the SemEval 2016 release? And perhaps the README could state whether this is the official (or at least preferred) Smatch code or if another repository should be used instead. I'm happy to do the work and provide a PR for these things but I cannot make the decision.

[danielhers]

Bumping the current version to 2.0.3 would imply that it contains the fixes included in other places, which I'm not sure it does. The SemEval 2016 version (by Guntis Barzdins and Didzis Gosko) is based on a separate fork, called pSMATCH, isn't it?

I don't even know if there is an "official" version any more, or who decides. For what it's worth, in the CoNLL 2019 shared task we used the version of smatch included in the mtool repository, which is based on the one here, but does not include #20 and #21 and instead includes some other changes (to allow programmatic usage).

[goodmami]

@danielhers thanks for explaining. I put "official" in scare-quotes (at first mention) because I understand it's not as if some authority decides, except perhaps in cases like with mtool where you fix the version for the task.

My concern is with reproducibility and result comparability; that is, to avoid some of the versioning issues that plagued BLEU. If a paper says they use "Smatch" for evaluation, it's really not clear which version or which fork they are using. Specifying the version number is a start, but it's somewhat meaningless unless combined with a URL to the repository/package where that version number is relevant. I'm also guilty of this: in a recent paper I put a URL to this repo but did not include a version or even date of access.

Regarding this repository and the SemEval 2016 2.0.2 version, if I check out a5fdf292e290599ffb7aeab112f916c2f0f78e68 (the initial commit) from this repository, then smatch.py is identical to the one distributed as 2.0.2 from SemEval 2016, so I think this includes all those changes. The pSMATCH code should be separate, then.

[nschneid]

There's also https://github.com/mdtux89/amr-evaluation/tree/master/smatch. @mdtux89, is this identical to aaf97e0?

[snowblink14]

This directory was created hoping to bring all the efforts in the community together for smatch, but like all open-source projects, it develops multiple branches/forks, so it might be difficult to define "official". That being said, I think it's a bit confusing if we take PyPi version into account. Maybe worth doing 1) Update the release of PyPi and make it at least consistent with this repository 2) Encourage the community to include more details on using smatch to improve reproducibility by updating README (difficult in practice though)

[goodmami]

Thanks @snowblink14. I now regret using the word "official", but the idea is: what do we recommend to someone who is not familiar with what the various forks do differently.

But I really agree with your point about updating the README with notes about reproducibility. It looks like @danielhers is already moving on the versioning issue, but I'll take a stab at a better README.

[mdtux89]

There's also https://github.com/mdtux89/amr-evaluation/tree/master/smatch. @mdtux89, is this identical to aaf97e0?

Yes, that's correct.

[goodmami]

@snowblink14 you recommended the following above:

1. Update the release of PyPi and make it at least consistent with this repository
2. Encourage the community to include more details on using smatch to improve reproducibility by updating README (difficult in practice though)

These things (and more) have been accomplished:

• The repository is now sync'd with the PyPI version, and new versions are published when there is a new git tag (e.g., creating a "release" on GitHub). @danielhers can you confirm that this is the recommended process?
• The README has been updated with notes on the version history and for encouraging reproducible research
• The update_log (renamed CHANGELOG.md) now more clearly indicates changes related to release versions

I think this issue can now be closed, or perhaps upon the next successful release (to confirm that publishing works as expected).

[danielhers]

@goodmami, thanks for these documentation improvements! I think they're very helpful to everyone in the community. As for releases, a small detail is that they are currently actually created automatically only when a tag is created on my fork, because the PyPI token is encrypted with the public key associated with it:

https://github.com/snowblink14/smatch/blob/a4f2e28a16666860bf70c4274803a60c67bac2cb/.travis.yml#L12

This means the process for a release is

1. @snowblink14 creates a release (and thereby automatically a tag) on https://github.com/snowblink14/smatch
2. I replicate this tag on https://github.com/danielhers/smatch and a Travis CI job automatically deploys to PyPI

To get rid of step (2), @snowblink14 would need to enable Travis CI for https://github.com/snowblink14/smatch, and then update the encrypted token in .travis.yml. I'll be glad to advise how to do that but I don't mind doing step (2) myself so I think the current situation is fine.

[goodmami]

I'm closing this issue since my question was resolved. Regarding the release process, I've moved the discussion to a new issue (#29).