Run Specimin on all methods in a project in CI

NiharikaJamble commented 2 months ago

This change will execute ASHE to check Specimin's accuracy (defined as ability to produce compilable output) on any push or pull requests made to main branch

kelloggm commented 2 months ago

@NiharikaJamble is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review".

NiharikaJamble commented 2 months ago

Hi Professor

I am working on two comments that you mentioned. Rest of them are fixed.I have the following issues/questions regarding the same.

Looking through the logs of the CI run, this doesn't appear to be quite working as intended:
- the resulting accuracy is 0%, which is not what we expect
- the cause might be that the logs indicate that Specimin is failing to run with some kind of problem related to the auto-formatter Spotless. Why is that happening?
- the logic uses a deprecated mechanism for storing results set-update. The logs have instructions for how to use the new, preferred mechanism ("environment files"), which you should look into.
the latest working log I had shows 90.07%, Can i get the log for which you encountered this issue?
In .github/workflows/gradle.yml https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*discussion_r1679630354__;Iw!!DLa72PTfQgg!LGE3Duu6fOGQFBMz6Fo6YAQPluvpcLEbEIlYqZlBSOvLLgyRQ_kqrBFktqd47yzKwVFDwWIdIEfixBqS90PpZQNWlw$ :

@@ -71,7 +71,7 @@ jobs: uses: @.*** with: distribution: 'corretto'

java-version: '17'

java-version: '21'

What's the reason for changing the Java version used for the regular CI runs? I don't think the Java version here should have any impact on the new CI job that you've added.

Also, why is the Java version important? Does something fail if we use Java 17? If so, that should be fixed: Specimin should run under Java 17+.

I changed the java version to 17. I am encountering a gradle version deprecated warning post which the CI stops. I am currently fixing the same.

This is what I encountered.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.***.

Thanks and regards

Niharika

On Wed, Jul 17, 2024 at 12:47 PM Martin Kellogg @.***> wrote:

@NiharikaJamble is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review". — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this

@NiharikaJamble https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgZLXdecQ$ is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review".

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2233751945__;Iw!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgpujcy3g$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNHNU2XTBVDBQ475KDDZM2N3RAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTG42TCOJUGU__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQiYZu7LQw$ . You are receiving this because you were mentioned.Message ID: @.***>

NiharikaJamble commented 2 months ago

Apologies for the incorrect error.Below is the error I wanted to mention for point 2 in previous mail:

On Wed, Jul 17, 2024 at 4:11 PM Jamble, Niharika @.***> wrote:

Hi Professor

I am working on two comments that you mentioned. Rest of them are fixed.I have the following issues/questions regarding the same.

Looking through the logs of the CI run, this doesn't appear to be quite working as intended:

the resulting accuracy is 0%, which is not what we expect

the cause might be that the logs indicate that Specimin is failing to run with some kind of problem related to the auto-formatter Spotless. Why is that happening?

the logic uses a deprecated mechanism for storing results set-update. The logs have instructions for how to use the new, preferred mechanism ("environment files"), which you should look into.

the latest working log I had shows 90.07%, Can i get the log for which you encountered this issue?

In .github/workflows/gradle.yml https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*discussion_r1679630354__;Iw!!DLa72PTfQgg!LGE3Duu6fOGQFBMz6Fo6YAQPluvpcLEbEIlYqZlBSOvLLgyRQ_kqrBFktqd47yzKwVFDwWIdIEfixBqS90PpZQNWlw$ :

@@ -71,7 +71,7 @@ jobs: uses: @.*** with: distribution: 'corretto'

java-version: '17'

java-version: '21'

What's the reason for changing the Java version used for the regular CI runs? I don't think the Java version here should have any impact on the new CI job that you've added.

Also, why is the Java version important? Does something fail if we use Java 17? If so, that should be fixed: Specimin should run under Java 17+.

I changed the java version to 17. I am encountering a gradle version deprecated warning post which the CI stops. I am currently fixing the same.

This is what I encountered.

Deprecated Gradle features were used in this build, making it incompatible with Gradle 9.***.

Thanks and regards

Niharika

On Wed, Jul 17, 2024 at 12:47 PM Martin Kellogg @.***> wrote:

@NiharikaJamble is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review". — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this

@NiharikaJamble https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgZLXdecQ$ is this ready to review again? If so, please either post a comment that says so or use the GitHub UI to "request another review".

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2233751945__;Iw!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQgpujcy3g$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNHNU2XTBVDBQ475KDDZM2N3RAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEMZTG42TCOJUGU__;!!DLa72PTfQgg!Kf7Af4y4qxwx13L8NEaCyz-XRYUyGzT1sZA2fg3oKoDIPeNKhTwRuI3JT6S3XXvjWGRpFQWFha31ZkYEZQiYZu7LQw$ . You are receiving this because you were mentioned.Message ID: @.***>

kelloggm commented 2 months ago

Apologies for the incorrect error.Below is the error I wanted to mention for point 2 in previous mail

I see - it looks like ASHE is compiling itself with Java 21, so its binaries don't work with Java 17. That's a good reason.

the latest working log I had shows 90.07%, Can i get the log for which you encountered this issue?

I'm referring to the log of the run of the CI job on this PR. In particular, I think I was looking at this one: https://github.com/njit-jerse/specimin/actions/runs/9952758388/job/27513834573

kelloggm commented 2 months ago

@NiharikaJamble CI is not running in this PR because there are conflicts with the main branch that you need to resolve. Once the conflicts are resolved CI should run again.

NiharikaJamble commented 2 months ago

Hi Professor

I did check on that. The error indicates that origin/main is not a branch present. However that is not the case since we do have a main branch. Also in the run you mentioned , only the gradle in use for ASHE was of a different version w.r.t the successful run. Given that gradle versions also impact the plugins used I have changed the path to clone the directories into directory of ASHE project so while running the gradle used is the one mentioned in gradle wrapper of ASHE project.

Also I am unable to run my updated changes in the PR.

So I wanted to request a run before we meet so I could have some outputs for the meeting

Thanks and regards Niharika

On Wed, Jul 24, 2024 at 3:09 PM Martin Kellogg @.***> wrote:

@ NiharikaJamble CI is not running in this PR because there are conflicts with the main branch that you need to resolve. Once the conflicts are resolved CI should run again. — Reply to this email directly, view it on GitHub, or unsubscribe.

@NiharikaJamble https://urldefense.com/v3/__https://github.com/NiharikaJamble__;!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy28LQZbTQQ$ CI is not running in this PR because there are conflicts with the main branch that you need to resolve. Once the conflicts are resolved CI should run again.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2248721585__;Iw!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy29szH0Erw$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNGYDW2JJJ7SN3A3ES3ZN73WFAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYG4ZDCNJYGU__;!!DLa72PTfQgg!KcwZzq5c7HrHeRNegCtXRYPOcyflNKQn1MmTtYa-Cs70n7-YA9S880qJbFQJMxpGaVQlrb96gR3Ui15iy28fSbHo_w$ . You are receiving this because you were mentioned.Message ID: @.***>

kelloggm commented 2 months ago

I did check on that. The error indicates that origin/main is not a branch present. However that is not the case since we do have a main branch

I don't know what you mean by this. There is clearly a conflict in .github/workflows/gradle.yml that you need to resolve; as soon as you push a merge commit that does so, CI will run automatically in this PR.

kelloggm commented 2 months ago

NiharikaJamble commented 2 months ago

Hi Professor

As per our discussion yesterday I have raised another PR. Can you please review the same?

Along with this I have noticed by pulling the newly added code the accuracy has dropped to 88.08 % from 90.07 %. I would like to check my run in the PR but seems like I cannot do it without your review.( some with write access needs to approve it)

Thanks & Regards Niharika Jamble

On Wed, Jul 24, 2024 at 4:16 PM Martin Kellogg @.***> wrote:

Screen. Shot. 2024-07-24. at. 4. 16. 09. PM. png (view on web) — Reply to this email directly, view it on GitHub, or unsubscribe. You are receiving this because you were mentioned. Message ID: <njit-jerse/specimin/pull/323/c2248822328@ github. com>

Screen.Shot.2024-07-24.at.4.16.09.PM.png (view on web) https://urldefense.com/v3/__https://github.com/user-attachments/assets/7efc8508-f537-4e5a-a0e2-c5b1234cc99e__;!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxAYRrqnmQ$

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2248822328__;Iw!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxCahU--8Q$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNDYOX22LX6M4TCZRSTZOADSJAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENBYHAZDEMZSHA__;!!DLa72PTfQgg!IplLMv7iNBC59SAnAqf7C0HVNhy_CVjiEYrvZSVbn6QOUfR9V3rEOAbAOn7KkrhoeIOzHX3pc8DBlw3yxxAEAZuhQA$ . You are receiving this because you were mentioned.Message ID: @.***>

kelloggm commented 2 months ago

As per our discussion yesterday I have raised another PR. Can you please review the same?

In the future, please request a review directly on the PR that needs to be reviewed, either by tagging me in a comment or by using the "request a review" feature in the GitHub UI.

Along with this I have noticed by pulling the newly added code the accuracy has dropped to 88.08 % from 90.07 %.

We've been making changes to Specimin, so it's possible that we've introduced a regression. Please don't worry about the exact value at this time.

kelloggm commented 2 months ago

I would like to check my run in the PR but seems like I cannot do it without your review.( some with write access needs to approve it)

I'm not sure where you're seeing this. Looking at your PR #330, it looks like CI is failing on your added job because there is some missing file: https://github.com/njit-jerse/specimin/actions/runs/10084952318/job/27884789451. You should have sufficient permissions to trigger CI (as seen by the fact that CI does run on #330). I'd prefer not to review a PR until CI passes.

NiharikaJamble commented 2 months ago

Hi Professor The script has completed its first successful run in the PR with 88.08%. Here is the run: https://github.com/njit-jerse/specimin/actions/runs/10101037185

Regards Niharika Jamble

On Thu, Jul 25, 2024 at 3:31 PM Martin Kellogg @.***> wrote:

I would like to check my run in the PR but seems like I cannot do it without your review. ( some with write access needs to approve it) I'm not sure where you're seeing this. Looking at your PR #330, it looks like CI is failing on your added

I would like to check my run in the PR but seems like I cannot do it without your review.( some with write access needs to approve it)

I'm not sure where you're seeing this. Looking at your PR #330 https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/330__;!!DLa72PTfQgg!MF_cdAzEYobRaOoMpQaeJ0hmNNgZL6yMaGYyUnuka7W77EIsnxark9N2DDey36W9ge8NZLBb7hWO789u6wkR6ojQgQ$, it looks like CI is failing on your added job because there is some missing file: https://github.com/njit-jerse/specimin/actions/runs/10084952318/job/27884789451 https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/actions/runs/10084952318/job/27884789451__;!!DLa72PTfQgg!MF_cdAzEYobRaOoMpQaeJ0hmNNgZL6yMaGYyUnuka7W77EIsnxark9N2DDey36W9ge8NZLBb7hWO789u6wmaI5xtgw$. You should have sufficient permissions to trigger CI (as seen by the fact that CI does run on #330 https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/330__;!!DLa72PTfQgg!MF_cdAzEYobRaOoMpQaeJ0hmNNgZL6yMaGYyUnuka7W77EIsnxark9N2DDey36W9ge8NZLBb7hWO789u6wkR6ojQgQ$). I'd prefer not to review a PR until CI passes.

— Reply to this email directly, view it on GitHub https://urldefense.com/v3/__https://github.com/njit-jerse/specimin/pull/323*issuecomment-2251254407__;Iw!!DLa72PTfQgg!MF_cdAzEYobRaOoMpQaeJ0hmNNgZL6yMaGYyUnuka7W77EIsnxark9N2DDey36W9ge8NZLBb7hWO789u6wm6tScqIg$, or unsubscribe https://urldefense.com/v3/__https://github.com/notifications/unsubscribe-auth/BIZGHNAZYK57IKZ2NDRZK4TZOFHANAVCNFSM6AAAAABK575Q7WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDENJRGI2TINBQG4__;!!DLa72PTfQgg!MF_cdAzEYobRaOoMpQaeJ0hmNNgZL6yMaGYyUnuka7W77EIsnxark9N2DDey36W9ge8NZLBb7hWO789u6wmutvmtgg$ . You are receiving this because you modified the open/close state.Message ID: @.***>

njit-jerse / specimin

Run Specimin on all methods in a project in CI #323