search

ndtp / android-testify

Add screenshots to your Android tests
Other
99 stars 5 forks source link

Support sharding with Gradle managed devices #187

Open sergio-sastre opened 7 months ago

sergio-sastre commented 7 months ago

Describe the bug Recording screenshots with andoid-testify and gradle managed devices works fine, unless we use sharding, e.g. using -Pandroid.experimental.androidTest.numManagedDeviceShards=2

In that case, only the sceenshots of one of the emulators is pulled out, for 2 shards it would be half of the expected screenshots

This issue relates to:

To Reproduce

  1. configure android-testify to use gradle managed devices, for instance

    managedDevices {
devices {
  pixel3api30(ManagedVirtualDevice) {
     device = "Pixel 3"
     apiLevel = 30
     systemImageSource = "aosp"
  }
}
}

  2. execute it with more than 1 shard, e.g. ./gradlew pixel3api30DebugAndroidTest -Pandroid.experimental.androidTest.numManagedDeviceShards=2

  3. Only the screenshots of one "shard" can be found in the corresponding gradle managed device folder

Expected behavior The screenshots of all "shards" are in the corresponding gralde managed device folder

Target Android Device (please complete the following information):

Additional context It works if numManagedDeviceShards=1

sergio-sastre commented 7 months ago

Describe the bug Recording screenshots with andoid-testify and gradle managed devices works fine, unless we use sharding, e.g. using -Pandroid.experimental.androidTest.numManagedDeviceShards=2

In that case, only the sceenshots of one of the emulators is pulled out, for 2 shards it would be half of the expected screenshots

This issue relates to:

  • [ ] The Kotlin library
  • [x] The Gradle plugin
  • [ ] The IntelliJ Platform plugin
  • [ ] The sample code
  • [ ] The documentation

To Reproduce

  1. configure android-testify to use gradle managed devices, for instance
managedDevices {
   devices {
      pixel3api30(ManagedVirtualDevice) {
         device = "Pixel 3"
         apiLevel = 30
         systemImageSource = "aosp"
      }
   }
}
  1. execute it with more than 1 shard, e.g. ./gradlew pixel3api30DebugAndroidTest -Pandroid.experimental.androidTest.numManagedDeviceShards=2
    1. Only the screenshots of one "shard" can be found in the corresponding gradle managed device folder

Expected behavior The screenshots of all "shards" are in the corresponding gralde managed device folder

Target Android Device (please complete the following information):

  • Device: Any
  • Physical or Virtual: Gradle Managed device
  • API Level: Any

Additional context It works if numManagedDeviceShards=1

I'll tag @olegosipenko, since he might have some suggestion for this ;)

DanielJette commented 7 months ago

Ha! I didn't even know they supported shards. Sounds like we need to read and respect the numManagedDeviceShards flag -- I imagine it's a matter of iterating over multiple directories instead of just one.

sergio-sastre commented 6 months ago

@DanielJette Could you tell me which classes should be adjusted to support this? I could try to fix it and create the corresponding PR

DanielJette commented 6 months ago

Hi @sergio-sastre I haven't had a chance to look into this, I'm not entirely sure what might be needed here.

These two PRs added the bulk of the GMD support so they might offer some good places to start looking:

The key file is Library/src/main/java/dev/testify/output/TestStorageDestination.kt

If I had to guess, the issue is with the value returned by getTestStoragePath. I would guess that perhaps it's writing files for each shard to a unique directory that's not being considered by Testify.

TestStorage has this getOutputProperties method ... maybe that has information about where the files are stored when sharded?

If you share what you find, I can try to help out with this.

sergio-sastre commented 6 months ago

Hi @sergio-sastre I haven't had a chance to look into this, I'm not entirely sure what might be needed here.

These two PRs added the bulk of the GMD support so they might offer some good places to start looking:

177 is the one where I added the support for properly saving the test storage files in a way that was compatible with Testify. So I'd focus on those changes.

The key file is Library/src/main/java/dev/testify/output/TestStorageDestination.kt

If I had to guess, the issue is with the value returned by getTestStoragePath. I would guess that perhaps it's writing files for each shard to a unique directory that's not being considered by Testify.

TestStorage has this getOutputProperties method ... maybe that has information about where the files are stored when sharded?

If you share what you find, I can try to help out with this.

After further investigating, it looks like the problem is in Gradle Managed Devices itself. I've already reported it: https://github.com/android/testing-samples/issues/502