Major issue with callback calling order

[boris-petrov]

Note the following code:

class Test
  constructor: ->
    @foo =
      bar: (callback) ->
        console.log 2
        do callback
        console.log 4

  testBAD: ->
    console.log 1
    await @foo.bar defer()
    console.log 3

  testOK: ->
    console.log 1
    @foo.bar ->
      console.log 3

do (new Test).testBAD
do console.log
do (new Test).testOK

This outputs:

1
2
4
3

1
2
3
4

Which, obviously, is seriously wrong. :) Or am I horribly mistaken as to what await/defer mean? :)

% iced --version                                   
IcedCoffeeScript version 1.4.0a

[andrusha]

Agreed, documentation clearly says that

  await $.getJSON url, defer json
  cb json.results

translates to

  $.getJSON url, (json) ->
    cb json.results

hence the second test of yours should be correct behaviour.

[boris-petrov]

Any development on this one?

[maxtaco]

I'm pretty sure that this is not a bug. First of all, I wouldn't have my code depend on the order of interleaving as in this example. In my mind, I draw an analogy to threaded code, in which case it's up to the scheduler to figure out which task to schedule next.

In this particular example, the runtime behaves the way it does to prevent code like this from overrunning the stack:

for i in [0...10000]
  if i % 1000 is 0
    await setTimeout defer(), 100

Callbacks aren't called directly but rather are scheduled via process.nextTick.

This feature and the determinism you want are at odds implementation-wise, and I think the feature I want is more important.

Can you paste in an example with more context? In other words, why does this matter to you?

[boris-petrov]

Well, as far as I understand the semantics of await/defer, defer is a function that returns a callback. This callback is a NORMAL function, which means that when I call it (which is line 6 in my original example - do callback I should get a SYNCHRONOUS call of the "function", which in this case is everything AFTER the await line (line 11). This has absolutely nothing to do with the setTimeout example which you gave, where, obviously, no determinism is possible. Here, however, you should have 100% determinism and synchronousness.

I could probably paste an example, but I need to remember what was my use case. I think it was connected with some tests that I was writing... I was expecting the await to finish exactly when I call the defer callback, but it wasn't and my tests were failing or something.

I love IcedCoffeeScript and want it to be perfect. :) I definitely vote for this to be fixed, if possible, as it is definitely not what people would expect.

By the way, I'm not sure how .NET 4.5 and the await features in C# 5 implement this (as this is the only other language I know that supports that kind of stuff, aside from the async monad in Haskell and the stuff in F# that is similar). You could check if you are interested and post your findings. :)

Thanks!

[maxtaco]

This idea and semantics are based on the C++ system available here: https://github.com/okws/sfslite .

Show me your example if you get around to it, thanks...

[boris-petrov]

Well, here's something like the code I had with which I found the issue:

class A
  constructor: ->
    @doNotCall = false

  do: (callback) ->
    return if @doNotCall
    callback =>
      @doNotCall = true

  doWrong: (callback) ->
    return if @doNotCall
    await callback defer()
    @doNotCall = true

a = new A

timesCalled = 0
f = (cb) ->
  assert false if ++timesCalled is 2
  do cb
  a.do f

a.do f

The issue is visible if you change the last two lines to use doWrong instead of do.

As I said, this is very close to what I had. You could imagine the class as the thing I want to test and the bottom code is the test itself. I want to test the fact that do (or doWrong) won't call their own callbacks twice. So in the "test" I call do, call cb BEFORE the second call to do expecting that @doNotCall will be set immediately to true because it should be a synchronous function call. However, in the doWrong case, this doesn't happen and the "test" fails.

This might seem as a contrived example to you, but it was very real and I really hit this issue and was debugging it for some time until I found it was a "bug". :)

[maxtaco]

Thanks Boris. If I were writing doWrong (correctly?), I would do this:

  doWrong: (callback) ->
    return if @doNotCall
    @doNotCall = true
    await callback defer()

I think as a practical matter, one shouldn't expect anything await'ed to callback right away, since that's presumably the reason one awaited in the first place. If the code changes down the road to not callback right away, you'll have a bug since your previous expectations are now unmet.

In other words, code with your assumption is less maintainable....

I'm going to close this issue and mark it "won't fix."

[boris-petrov]

Well, I couldn't put the @doNotCall = true line before the callback as there were conditions that had to be met for me to do that (and they depended on results coming from the callback), but anyway. :) Thanks for taking the time to review this.

[andrusha]

@maxtaco could you explain why is it expected behaviour? Does it only happen if we defer no variables?

[maxtaco]

I'm saying that the "spec" doesn't specify when the code after an await block will be scheduled, so your program should not encode any assumptions about it. The runtime reserves the right to schedule the block that comes after an await block either immediately, or at process.nextTick.

In the case of Boris's example, the block in question is console.log 3, which comes directly after an await block. The runtime can schedule this block immediately (and therefore before console.log 4) or at process.nextTick (and therefore after console.log 4). You should treat this scheduling decision as non-deterministic in all cases.

It is my strong opinion that well-written IcedCoffeeScript code shouldn't care about this scheduling decision.

The runtime needs this flexibility. If it didn't have this flexibility, and it had to schedule blocks following awaits immediately, then it would overrun node's stack in many real-world examples.

[andrusha]

Thanks, now I got. Makes sense.