Require copilots to update the challenge specification

[lstkz]

In almost every challenge the forum discussion changes the initial specification. There are a "rule" that says the discussion always overrides the requirements. Unfortunately, copilots never update the specification, and it can be very confusing for both submitters and reviewers.

Example situations: 1. spec: implement unit tests forum: unit tests are not required -> the copilot should remove unit tests requirement from the spec

2. spec: no info if image sprites are required or no forum: confirmation that we must use image sprites -> the copilot should add this requirement to the spec

3. spec: describes XYZ feature forum: marks it as optional -> the copilot should update the spec and mention that XYZ feature is optional

4. forum: discussion about edge cases that are marked as required -> the copilot should try to describe it explicitly in the spec, so reviewers don't miss it

5. forum: discussion about edge cases that are marked as out of scope -> no need to update the spec

It would be nice if we can host the specification in GitHub repo. Members can submit pull requests so that the copilot will have less work. We have currently similar system but with Github tickets. I believe it will reduce the number of invalid responses in the Review phase. Currently, it's very easy to miss important requirements or misunderstood something if there are many threads.

[ThomasKranitsas]

@lsentkiewicz you may want to take a look here https://www.youtube.com/watch?v=bfiRnrnJgsE In the new challenge details page concept there is the idea of a Notification feed where we (registrants) we can see all the updates that happened in a challenge like updated requirements, removed requirements etc.

So I guess this one is kind of To-do or In-progress :)

@hokienick & @vic-topcoder can give us more details here.

[lstkz]

Which minute exactly? The new layout looks great, but it will take months to make it live.

[ThomasKranitsas]

@ 21:10 you can see the Notifications & Feed section on the sidebar.

[lstkz]

First, we must ask copilots to update the spec if there are some changes or clarifications. The notification widget will be just an extra feature.

[ThomasKranitsas]

I believe it would be better if instead of updating the challenge spec, to add another section (eg. Clarifications) where copilot will define all the clarifications or decisions he/she made from the forum discussions.

If copilot update the spec all the time then competitors will always have to find what was changed in the spec but if there is a list of clarifications it would be much easier.

Example: Spec:

1. Implement requirement A.
2. Implement requirement B.
3. Implement requirement C.

Clarifications:

1. Requirement B is optional
2. Both X, Y approaches are acceptable for requirement C.

[callmekatootie]

I am of the opinion that unless the participants can see a changelog (generated automatically) for the changes made to the specification once the contest went live, the spec should not get updated.

The reviewer can go through the spec first and then through the discussions in the forum, which in itself, serves as a pseudo changelog for the spec.

[ThomasKranitsas]

@callmekatootie but in many (if not in most) cases there are too many questions/clarifications in forum or different members asking the same question in different threads which makes it even harder to understand what the final decision/clarification was. IMO Adding a change log right after the original spec without touching the original spec would be the best option.

[callmekatootie]

I am not against a change log but I would not consider it to be a must have.

Too may questions / discussion => Spec is written poorly. Co-Pilot should learn from it and prepare a better spec the next time.
Asking same questions in different threads => Co-Pilot should point the member to the earlier discussion and warn the member to read the threads carefully and not duplicate their questions.

[lstkz]

@callmekatootie There are many complaints about the review process and reviewers. I am trying to suggest improvements that will make life easier for them. Also, some things are confusing for submitters.

For example: In my last challenge, there was a discussion about some edge case. The copilot suggested a solution for that problem, and I simply assumed that we must implement it. After the review phase, he wrote Why are you trying to make something that wasn't in the spec required?. He didn't confirm it explicitly, but he didn't say it was out of scope either. It was weird for me because what is the point of discussion something if it's not required?

Reviewers also don't have an easy job. They read the spec, some points are valid, some points are invalid, and the forum discussion overrides some points. It happens very often that they miss something. As a submitter, I sometimes need to spend a few minutes to find a proper thread and appeal invalid response.

I remember some copilots tried to update specification and keep it up to date in the past. I don't think it requires much effort.

@ThomasKranitsas I don't like a separate changelog file. It's better to have only one document than two documents.

[talesforce]

@lsentkiewicz - I like the idea of updating the spec to be consistent when needed. A lot of submitters in fact ask for the spec to be updated based on forum clarifications and I believe it's a good way to ensure that the spec and forums are in sync with each others.

At the same time, I don't think the spec can be updated for every forum clarification. It should only be done for any requirements - the examples you quoted e.g. unit tests, sprites etc. are good examples when when it should be done.

It will be helpful to have the spec updates highlighted in some manner on the challenge page.

[lstkz]

@talesforce Not for every clarification, but only for major issues.

Example from my recent challenge as a reviewer https://www.topcoder.com/challenge-details/30057374/?type=develop It was required to implement MovieFetcher. In the forum, the copilot said that we can remove it from the scope, but in my opinion, it wasn't explicitly confirmed. 3 of 4 submitters implemented it, and I am not sure if they all were aware of it.

IMO the copilot should update the gitlab ticket and say "it's out of scope, but it can be considered as an extra feature".