Add a way to configure ackDeadlineSeconds

[jung-kim]

fixes: #532

Does no validation of ackDeadlineSeconds, I believe acceptable range is t >= 0, and let the exception to be thrown at the random subscription creation time.

[googlebot]

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

:memo: Please visit https://cla.developers.google.com/ to sign.

Once you've signed, please reply here (e.g. I signed it!) and we'll verify. Thanks.

---

• If you've already signed a CLA, it's possible we don't have your GitHub username or you're using a different email address. Check your existing CLA data and verify that your email is set on your git commits.
• If you signed the CLA as a corporation, please let us know the company's name.

[jung-kim]

I signed it!

[googlebot]

CLAs look good, thanks!

[dhalperi]

Hi @CodingTwinky -- this will not actually affect behavior when reading from Pub/Sub on Google Cloud Dataflow. Is this intended to change testing behavior using the InProcessPipelineRunner?

[jung-kim]

My intention was to add a way for user to configure ackDeadlineSeconds when no subscription is defined and random one is created via dataflow. Am I missing something?

[dhalperi]

@codingtwinky the code that creates the temporary subscription in the Java SDK is only used when running with the InProcessPipelineRunner. the DataflowPipelineRunner uses different code.

Can you say more information about why you want to change the ack deadline? You might want to reach out to the Dataflow support team: https://cloud.google.com/dataflow/support if you are seeing issues when running on the DataflowPipelineRunner that you believe are attributable to this parameter.

[jung-kim]

Reason why we were experimenting with various ack deadline was and experimentation to see dataflow's behavior during high backlog situation, which we've been having hard time with. We were able to test by creating subscription manually but if we could configure ack deadline for throwaway subscription we can create on dataflow creation.

[reuvenlax]

Dataflow should automatically extend acks if processing takes longer than the deadline. What exactly were the effects you saw by changing the deadline on the subscription?

[jung-kim]

For our use case, once dataflow failed to keep up with pubsub's publish rate, ~18k/sec, it gets into weird "ack deadline passed" death spiral and only way to recover is to pubsub's publish rate to go down.

Increasing pubsub deadline ack timeout was an attempt at delaying when this death spiral happens.

Considering I can manually create a pubsub subscription with desired configuration I will close this issue but it would be nice to be able to configure this.