search

xlab-uiuc / IDoCT

Illinois Dataset of Configuration Tests
3 stars 23 forks source link

inconsistency between the description of the onboarding task and the CI scripts #28

Open xylian86 opened 1 year ago

xylian86 commented 1 year ago

As mentioned in the Pull Request #27, there is an inconsistency between the description of the onboarding task and the CI scripts Currently, the onboarding task instructs students to randomly select five tests, while the CI scripts generate an error if the chosen tests have already been selected by other students.

To address this issue, we can implement the following solutions:

  1. Update the onboarding task: Modify the instructions to ask students to select only untouched tests, i.e., tests that have not been chosen by any other student.
  2. Update the CI scripts: Instead of throwing an error when duplicate tests are detected, change the behavior to issue a warning. This will allow the CI process to proceed without blocking the execution, but still notify users about the potential issue of duplicate test selection.
tianyin commented 1 year ago

Thanks @Alex-Lian !

I personally don't like workarounds of the CI scripts, because it will end up with repeated efforts and redundant results. So I don't particularly like #2. But if you think it's fine, it's ok for me.

Ideally we should ask students to work on separate tests. So I think what we should do is to find a way to achieve that via the onboarding tasks. I like #1 on changing the onboarding task. But it does not prevent students to concurrently choose two untouched tests (there is no lock to prevent data races).

tianyin commented 1 year ago

@Alex-Lian Also, is IDoCT specific to Hadoop?

I think we should expand our repo sets, especially given that there are lots of efforts in https://github.com/xlab-uiuc/openctest to integrate with more repos.

xylian86 commented 1 year ago

@tianyin

But it does not prevent students to concurrently choose two untouched tests (there is no lock to prevent data races).

Yes, it is possible. One approach we can take is to modify the CI script approval process for each pull request, ordering them based on their open time. By implementing this change, the student who opens the pull request earlier will have the priority to choose the tests.

Also, is IDoCT specific to Hadoop?

Yes, the current description of the onboarding task and the CI scripts is only for hadoop-common.

I think we should expand our repo sets, especially given that there are lots of efforts in https://github.com/xlab-uiuc/openctest to integrate with more repos.

I can do it when I am available (after the paper ddl) or we can ask undergraduate students or student enrolled in 591 SE or students takeCS527 and want bonus point.

tianyin commented 1 year ago

By implementing this change, the student who opens the pull request earlier will have the priority to choose the tests.

yes but it's not ideal as it will inevitably run into the situation that someone worked hard with results but invalidated by the others.

xylian86 commented 1 year ago

yes but it's not ideal as it will inevitably run into the situation that someone worked hard with results but invalidated by the others.

Indeed, to avoid such situation. We can suggust students to take the following steps before investing effort into the chosen Ctests. First, they can open a pull request to assert ownership of the selected tests by including this information in the description of the PR. Then, they can proceed to update the pull request with their results.