onyx-platform / onyx-datomic

Onyx plugin for Datomic
http://www.onyxplatform.org/
Eclipse Public License 1.0
37 stars 14 forks source link

Allow read-log to complete when there's no more txes #27

Closed jeremyheiler closed 7 years ago

jeremyheiler commented 7 years ago

This PR addresses #26. The issue I was having was that the job would never complete when there was an end-tx. This happened because end-tx was never checked before trying to load more transactions, and it would get stuck in a loop.

My use case that discovered this problem is an integration test where I want to read the last few transactions from the log. In a streaming situation, I don't think this would be an issue because it would just load more transactions and eventually complete.

I tested this using my integration test topology, and I'd be happy to include a proper test. However, I want to see if this makes sense before I do that.

Thanks!

lbradstreet commented 7 years ago

Thanks. This works. I'm moderately tempted to make a breaking change and make the end-tx inclusive so we don't have to wait for that extra transaction. I think if we did we would have to change the task-map keys so we don't catch anybody out.

That said, there's no downside to your change vs the status quo, so I will probably go with it instead.

jeremyheiler commented 7 years ago

I think this does work with end-tx being inclusive. For the test, when the end-tx is 1002, it has to look up the next tx (1007) in order to know that 1002 doesn't exist.

lbradstreet commented 7 years ago

Ah, I just meant that it kinda sucks to have to wait for an extra transaction if all you want to do is process the whole db. For example, say you want to do a once off transformation/filter and didn't want to have to transact one extra time so that you can just have the job finish.

lbradstreet commented 7 years ago

Merging this in to prepare for the next release.

jeremyheiler commented 7 years ago

I thought I had replied to this, but I guess not. My use case was to be able to have the start and end tx be the same, so it only processes that one. So, I think it works as to what you're saying.

jeremyheiler commented 7 years ago

And, thanks!