[bootstraponline] number of changes to the object exceeds the rate limit.

[bootstraponline]

renas commented on f074037 a minute ago @bootstraponline I'm getting this error now when I run test suites containing large amount of tests:

Uploading [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to Firebase Test Lab... ERROR: (gcloud.firebase.test.android.run) Could not copy [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk] to [gs://walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/] ResponseError 429: The total number of changes to the object walmart-fb668-flank/2017-06-12_17:52:57.388000_ShhK/walmart-android-qa-debug.apk exceeds the rate limit. Please reduce the rate of create, update, and delete requests..

Before when we only had a root project we didn't experience this error. Seems to have come with the sub-folder object that we now create.

https://github.com/TestArmada/flank/commit/f074037403c0173cc33cd7340f0956921bd643fd#commitcomment-22496961

moving this to an issue so it can be tracked properly.

[bootstraponline]

The total number of changes to the object walmart-android-qa-debug.apk exceeds the rate limit.

@renas are you sure we are uploading that APK exactly once and then using the google storage location for all future references? In the firebase slack one of the engineers mentioned that we're constantly uploading the APKs. If that's true then it seems like a defect in the runner.

[bootstraponline]

I'm using the same folder structure via gcloud CLI and not encountering any issues.

• gcloud.rb

[bootstraponline]

the "apk" object should only be created once, never updated, and deleted once.

Please reduce the rate of create, update, and delete requests..

[renas]

From what I can see we only upload them once. Not sure if we are doing something wrong somewhere.

[bootstraponline]

I'm currently wrapping up another project. I'll be able to work on Flank again starting tomorrow. I'm happy to investigate this further then.

[renas]

That would be great @bootstraponline. Thanks!

[bootstraponline]

this log shows upload being called multiple times. my guess is that's what's happening.

will hopefully be an easy fix.

[renas]

Keeping my fingers crossed :)

On Tue, Jun 13, 2017 at 7:17 AM, bootstraponline notifications@github.com wrote:

this log shows upload being called multiple times https://gist.github.com/thalescm/7e64bc80fc2a32dcb8eef50fc0a96aac#file-flank-shard-error-L51-L56. my guess is that's what's happening.

will hopefully be an easy fix.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TestArmada/flank/issues/34#issuecomment-308130963, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKGpf1tf57VgkII3_gzcMRSHJhnAZ9Pks5sDpoOgaJpZM4N3hhL .

[bootstraponline]

I have checked the SDK and this is what's happening.

log.status.Print('Uploading [{f}] to Firebase Test Lab...'.format(f=path))
    try:
      if path.startswith(GCS_PREFIX):
        # Perform a GCS object to GCS object copy

for some reason the CLI is copying the object even when we're already on GCS. I'll talk to the firebase team about this behavior.

[renas]

Great. Thanks.

On Tue, Jun 13, 2017 at 7:25 AM, bootstraponline notifications@github.com wrote:

I have checked the SDK and this is what's happening.

log.status.Print('Uploading [{f}] to Firebase Test Lab...'.format(f=path)) try: if path.startswith(GCS_PREFIX):

Perform a GCS object to GCS object copy

for some reason the CLI is copying the object even when we're already on GCS. I'll talk to the firebase team about this behavior.

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub https://github.com/TestArmada/flank/issues/34#issuecomment-308133453, or mute the thread https://github.com/notifications/unsubscribe-auth/AAKGpcfA899RgPojI16ukHIbRBV5ngGpks5sDpvCgaJpZM4N3hhL .

[bootstraponline]

I resolved the issue:

diff --git a/gcloud/google-cloud-sdk/lib/surface/firebase/test/android/run.py b/gcloud/google-cloud-sdk/lib/surface/firebase/test/android/run.py
index 256c164..fc509da 100755
--- a/gcloud/google-cloud-sdk/lib/surface/firebase/test/android/run.py
+++ b/gcloud/google-cloud-sdk/lib/surface/firebase/test/android/run.py
@@ -166,11 +166,16 @@ class _BaseRun(object):
     bucket_ops = results_bucket.ResultsBucketOps(
         project, args.results_bucket, args.results_dir,
         tr_client, tr_messages, storage_client)
-    bucket_ops.UploadFileToGcs(args.app)
-    if args.test:
+    print('Checking for gcs prefix on app apk, test apk, and obb files')
+    GCS_PREFIX = 'gs://'
+
+    if not args.app.startswith(GCS_PREFIX):
+      bucket_ops.UploadFileToGcs(args.app)
+    if args.test and not args.test.startswith(GCS_PREFIX):
       bucket_ops.UploadFileToGcs(args.test)
     for obb_file in (args.obb_files or []):
-      bucket_ops.UploadFileToGcs(obb_file)
+      if not obb_file.startswith(GCS_PREFIX):
+        bucket_ops.UploadFileToGcs(obb_file)
     bucket_ops.LogGcsResultsUrl()

     tr_history_picker = history_picker.ToolResultsHistoryPicker(

In the test execution log there's no reference to Uploading x to Firebase Test Lab.... The duplicate copies are successfully skipped.

The default SDK doesn't provide any control over this. I'm not sure how we'll want to ship a patch to the SDK.

[renas]

Not sure how we should proceed. Maybe you can have a chat with the Firebase engineers?

[bootstraponline]

I have messaged them. They're in a different time zone. I expect a response later today. At work we've just forked the SDK and use that instead of upstream.

[bootstraponline]

I talked with the team. A feature request has been opened to optionally disable the apk copying. No ETA. The long term strategy is using the API directly (as mentioned in #14).

I never ran into this with my Python runner because I'm using the existing API. We'd get the benefit of using 1 process for all the shards which is a win.

I think for now the easy solution is to add a shard.py and then copy it into the existing gcloud SDK folder. Then the flank jar will invoke that instead of the regular run.py.

Alternatively we replace the entire Java codebase with a gcloud-cli Python extension. Thoughts?

[renas]

Running it one process would be really nice. We could copy shard.py to the gcloud sdk folder just as you describe. Can we somehow tell shard.py which shards we want it to create (the tests we want in each chard)? E.g. something like this: [test1, test2, test3], [test4, test5, test6, test7], etc

[bootstraponline]

Can we somehow tell shard.py which shards we want it to create (the tests we want in each chard)?

Yep. shard.py is a script that we would provide so it can do anything. It'd be up to us to create it.

The initial shard.py will just contain a fix for the apk copy issue. Behavior will be otherwise identical to the existing test android run command. After that we can investigate expanding into a single process model which is a bit of a larger project.

The idea is that shard.py becomes the entry point. It'd run dex-test-parser, determine the shards, and finally output the results. flank.jar wouldn't exist anymore. Assuming that sounds good, I'm happy to work on this.

Are you on the firebase slack? It'd be nice to chat there sometime instead of always via a GitHub issue.

[bootstraponline]

Fixed by using Java API instead of the gcloud-cli.

[traviswestbrook]

Hate to comment on a closed issue but wasn't sure where the best place to follow up was. It seems like there is both a java poc and a kotlin poc I was wondering which one if either will be used to replace flank.jar? Also is there any outline from moving the rest of the functionality over to the poc?

[bootstraponline]

I think eventually the Kotlin proof of concept will replace flank.jar. The issue is there are no dedicated development resources for flank so any progress is slow. It's missing a bunch of features and has some known issues.

I'm currently working on other projects and flank hasn't been a priority. I still think flank is great though.