run-crank / cog-iterable

Iterable Cog for use with Crank
https://crank.run
MIT License
0 stars 2 forks source link

Create step can sometimes return before contact is actually created #20

Closed iamEAP closed 4 years ago

iamEAP commented 4 years ago

Repro Steps

  1. Run the following scenario many times with unique email addresses each time:
scenario: Contact Create is Async?
description: Shows that contact creation appears to be async.

tokens:
  test.email: async-contact-235235@thisisjust.atomatest.com

steps:
- step: When I create or update an iterable contact
  data:
    contact:
      email: '{{test.email}}'
      firstName: Async
      lastName: Test
- step: Then the firstName field on iterable contact {{test.email}} should be Async
- step: And the firstName field on iterable contact {{test.email}} should be Async
- step: And the firstName field on iterable contact {{test.email}} should be Async
- step: And the firstName field on iterable contact {{test.email}} should be Async
- step: Finally, delete the {{test.email}} iterable contact

If you are using a POSIX-compliant shell, this script will do the trick:

#!/bin/bash
crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com" \
  && crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com" \
  && crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com" \
  && crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com"  \
  && crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com" \
  && crank run async.crank.yml --token test.email="async-contact-$(awk -v min=0 -v max=9999999 'BEGIN{srand(); print int(min+rand()*(max-min+1))}')@thisisjust.atomatest.com"

Expected: The scenario should pass.

Actual: Very occasionally, the first or several of the first check steps will fail, while subsequent check steps pass (even though the steps are identical).

Screen Shot 2020-02-17 at 4 31 58 PM

iamEAP commented 4 years ago

It's possible this is just a quirk of the Iterable API: perhaps on the back-end, they batch/queue contact creation and respond with a success before the contact is actually consistently available.

...But if so, we'll want to account for this, as it can make tests flakey (as seen above).