Non-deterministic code in orchestrator fails silently

[anthonychu]

Describe the bug When non-deterministic code in an orchestrator is encountered during replay, no error message is displayed, the replay fails silently, and the orchestrator stays in a running state.

Investigative information

• Durable Functions extension version: 2.0.0
• durable-functions npm module version: 1.3.1
• Language (JavaScript/TypeScript) and version: JS
• Node.js version: 10.16.3

To Reproduce Steps to reproduce the behavior:

1. Clone this repo: https://github.com/anthonychu/test-durable-js

2. npm i
   func extensions install
   cp local.settings.SAMPLE.json local.settings.json
   # Add storage connection string to local.settings.json
   func start

3. Start the BadTimer orchestrator: http://localhost:7071/api/orchestrators/BadTimer.
4. A timer is scheduled to fire in 30 seconds. It uses new Date() so it should fail with a non-deterministic error.
5. During replay, there are no error messages. The status URL indicates the orchestration is still running.

@ConnorMcMahon @cgillum

/cc @christiannwamba this is the problem you ran into this weekend.

[ConnorMcMahon]

I'm curious if this results in a non-determinism error in C#. From my knowledge, we would only throw an error in C# if the different timer execution results in us to attempt to make future durable calls in a different order.

We could definitely add some checks on the Durable JS side of things to see if the timer wakeup time on replay is the same as the originally scheduled time.

[anthonychu]

I always thought that if the inputs to callActivity and createTimer during replay don't match the inputs in the original call, it's a non-deterministic error. Maybe that's not the case.

In any event, this is failing silently and should probably be fixed.