Tests failing with `FileNotFoundException`

[saurabharora90]

We've been using Roborazzi for a while. Recently we've started to see flakiness appear in our screenshot tests on Github CI.

Execution failed for task ':features:myModule::finalizeTestRoborazziDebug'.
> java.io.FileNotFoundException: /home/runner/work/myapp/features/mymodule/build/test-results/roborazzi/results/1760836821606_com.myapp.DetailUiScreenshotTest.empty state.json (No such file or directory)

We need to retry a bunch of times before it goes through. We've updated to the latest 1.16.1 version and continue to face this issue.

We have build cache setup on CI and even tried clearing the build cache but didn't have any luck with this.

[takahirom]

@saurabharora90 Thank you for reporting this. Could you provide the --info (You can filter "Roborazzi" logs) and --stacktrace logs for this issue?

[saurabharora90]

Can you provide an email address to share the logs?

Btw we updated to 1.17.0 and its the same issue

[takahirom]

@SaurabhArora90 Thank you for checking with the latest version. This is my email address: takam.dev@gmail.com

[saurabharora90]

Emailed the --info logs

[takahirom]

@saurabharora90 Thank you. I was able to understand the situation from the log you provided. Could you also provide a --stacktrace log or point out where the crash occurred? I think it occurred here; is that correct?

io.github.takahirom.roborazzi.RoborazziPlugin

              val results: List<CaptureResult> = resultDirFileTree.get().mapNotNull {
                if (it.name.endsWith(".json")) {
                  CaptureResult.fromJsonFile(it.path) // **here**
                } else {
                  null
                }
              }

[takahirom]

📝

              val results: List<CaptureResult> = resultDirFileTree.get().mapNotNull {
                if (it.name.endsWith(".json")) {
                  CaptureResult.fromJsonFile(it.path) // **here**
                } else {
                  null
                }
              }

fun fromJsonFile(filePath: String): CaptureResult {
      val jsonElement = json.parseToJsonElement(FileReader(filePath).readText())
      return json.decodeFromJsonElement<CaptureResult>(jsonElement)
    }

  const val resultDirPathFromBuildDir = "test-results/roborazzi/results/"

      val resultDirFileProperty =
        project.layout.buildDirectory.dir(RoborazziReportConst.resultDirPathFromBuildDir)
      val resultDirFileTree =
        resultDirFileProperty.map { it.asFileTree }

[takahirom]

@saurabharora90 I believe your tests are running testReleaseUnitTest and testDebugUnitTest simultaneously. Is this intentional? You might resolve this by excluding testReleaseUnitTest using -x testReleaseUnitTest. Ideally, however, Roborazzi should create a separate directory for each variant.

[saurabharora90]

I believe your tests are running testReleaseUnitTest and testDebugUnitTest simultaneously. Is this intentional? You might resolve this by excluding testReleaseUnitTest using -x testReleaseUnitTest. Ideally, however, Roborazzi should create a separate directory for each variant.

Yes we are running both variants. It is intentional for the time being. Though we plan to migrate away from that in the long run.

For stack trace, I'll try to get it if you need them. Given the error doesn't happen all the time, it might take some time.

[saurabharora90]

We've disabled release for the past one week and have stopped seeing this issue

[takahirom]

Thanks. I tried to reproduce the problem in the integration tests but was unable to do so. Therefore, I'll wait for the person who has a project that can reproduce it. https://github.com/takahirom/roborazzi/pull/380