search

NationalSecurityAgency / lemongraph

Log-based transactional graph engine
Other
1.14k stars 150 forks source link

lg-lite branch status #25

Closed d4tocchini closed 5 years ago

d4tocchini commented 5 years ago

lg-lite seems to compile & function properly, any details / status on this branch? Should this be viewed this as preferred direction moving forward or experimental not likely to be merged?

NSA-LGDev2 commented 5 years ago

Yes, lg-lite will get pushed to master at some point. I need to decide if tasks should have additional states and if so, how best to structure that. At the moment, tasks are either pending, or do not exist. It might be useful to be able to mark a task as completed (or errored?), and optionally retain original task contents. Thoughts are welcome!

d4tocchini commented 5 years ago

I'll check it out, thanks

d4tocchini commented 5 years ago

on a related note, can you clarify how multi-threading / processes fits in with tasks?

In the lg-lite branch, is the REST service the same structure: with a master proc, sync proc and potential worker req-handler procs?

Thinking about it, task state, other than completed or not, is probably best left to task-specific concerns, right? A task could always hit the disk or mutate shared memory systemV style or whatever. I don't see needing more than stateless configuration for things like job priority, how worker-threads match with job-types, locking worker-threads to processors, etc. A solid example in this area being Naughty Dog Engine's fiber-based task scheduler see Christian Gyrling GDC Talk. Potentially usable implementation here.

NSA-LGDev2 commented 5 years ago

Sure - tasks have nothing to do with threads/processes - they are a batch of records from a single job that were pulled by an adapter client. If you haven't yet, take a look here - that explains the purpose of the lg-lite branch.

Same process topology for the web service though - master proc, sync proc, and multiple web workers.

d4tocchini commented 5 years ago

I probably should've digested that doc more as I was clearly conflating the concept of tasks with jobs and hypothetical job execution concerns. I think a high level overview from input through internals to output with tangible use-cases would help me & prob others better contribute here. Flushing out the Design section of LG_LITE.md and sprinkling a few sample code-blocks would likely suffice. Obviously, understanding can be gleamed in the tests, examples and source, but @ayende's 7-part lemongraph code-review sheds light on such an attempt.

Thanks for clarifying the status of the lg-lite branch. I'll close this issue & if I land on tangible insights or conceptual-blocks regarding tasks & the like I'll open a PR to help move the docs along.