How libuv put callback function into event queue?

[rpf5573]

How libuv put callback function into event queue with response value ???

[davisjam]

I am not sure I understand your question.

Are you asking about the implementation of libuv, or how to use it to achieve a certain goal, or something else?

[rpf5573]

Sorry, I'm studying NodeJS and I found that I should understand Libuv to understand NodeJS. So I started from I/O Device ~ Non-Blocking ~ Event Loop etc. Many people say that callback function is enqueued in Event Queue and event loop iterate that queue and process that callback function. But ! (1) Who put callback function to event queue ? (2) And is there really only callback function in event queue? how about result of I/O Operation - The value which will be used in callback function as a argument value ? (3) Is there javascript string ( my callback function ) in event queue ? or some pointer to my js callback function?

I'm really curious.... really.

Please shine a light on my head.

[davisjam]

This is not a super polished response. I'm hoping it's enough to help you get started and reach a better understanding of the code.

(1) Who put callback function to event queue?

Node.js is a libuv "embedder". Node.js uses libuv for (1) platform-independent (asynchronous) system call support, and (2) its event loop / worker pool paradigm.

See nodejs/src/node.cc::Start for the call to uv_run that drives the libuv event loop that Node.js uses.

This enters (in Linux) libuv/src/unix/core.c where uv_run is defined. Note the call to uv__io_poll within which heads over to libuv/src/unix/linux-core.c where epoll_pwait is issued.

(2) And is there really only callback function in event queue? how about result of I/O Operation - The value which will be used in callback function as a argument value?

Everything the libuv event loop handles is a "callback". But the callbacks the libuv event loop invokes (function pointers) are not always the same as the callbacks in Node.js (JavaScript code). Node.js sometimes (always?) interposes its own C++ code as callbacks. For example, fs.readFile is converted by Node.js into several uv_fs_X operations under the hood, and only after stat, open, potentially many read's, and close will the user's JS callback be invoked. Each of those FS operations runs asynchronously and spends some time on the event loop. C++-land Node.js libraries (and I believe the code re-enters the JS fs library along the way?) manage all that for you.

To see how code moves from Node.js JavaScript into Node.js C++, take a look for example at node/src/node_file.cc::Initialize. These calls tell V8 how to map from JS calls into the appropriate Node.js C++ bindings.

[rpf5573]

Thank you so much ! I'm reading source code of libuv with my very weak knowledge of C++ and thread, OS.

I have some question about communication between worker thread and main thread(event loop). I tracked down the file system function from var fs = require('fs') to uv_async_send(&w->loop->wq_async) And! many people say that uv_async_send wakes up the event loop and calls the async handle's callback (https://github.com/joyent/libuv/issues/1453) I think that uv_async_send just write some simple string to async_wfd(async work file descriptor ?).

1. How does this uv_async_send wake up event loop of main thread ? I searched async_wfd keyword in a whole source code of nodejs, but,, I think event loop does not access that asycn_wfd... Am I wrong?

2. Is this related with unicorn function ?

3. Is this related with this orange box?

I'm really curious, Check please !

[davisjam]

In threadpool.c, the worker function (thread) issues uv_async_send(&w->loop->wq_async) whenever it completes work.

This wakes up the event loop, which calls the async handler's callback from the event loop. This callback is uv__work_done, which invokes the uv__work's done CB, which is uv__queue_done (set in uv_queue_work), which (finally) invokes the uv_work_t's after_work_cb.

Note that uv__work_done is not declared static, which is a hint that it is externally visible. In src/unix/loop.c:uv_loop_init you can see that loop->wq_async member is registered with callback uv__work_done.

How does this uv_async_send wake up event loop of main thread ?

Look at src/unix/async.c:uv__async_send. Note the write call which affects an fd that is in the set of fds monitored by the event loop.

[rpf5573]

davisjam, thanks to you, the mystery seems to be slowly solved. Now I may understand what "wake up event loop" means.

I found some information about this in a libuv design overview.

Pending callbacks are called. All I/O callbacks are called right after polling for I/O, for the most part. There are cases, however, in which calling such a callback is deferred for the next loop iteration. If the previous iteration deferred any I/O callback it will be run at this point.

Yes, If I call fs.readFile(), this happens.

1. fs.readFile( ) in javascript land
2. jump to c/c++ land
3. insert that file read request into work queue.
4. one of thread in thread pool process that request.
5. w->work() ( == uv__fs_work() ) is called in worker() function 5.QUEUE_INSERT_TAIL( &w->loop->wq, &w->wq ) // I don't know the purpose of this code. But this looks important.
6. uv__async_send( )
7. write some information to async_wfd file description.

And in event loop.

1. uv__run_timers() ~~ uv__run_prepare() are called in uv_run()
2. uv__io_poll( loop, timeout ) is called
3. in uv__io_poll, monitoring(loop) many file descriptions and, run callback function related with that file description( w->cb(loop, w, pe->events) )
4. Makecallback ~~ InternalMakeCallback ~~ anyway, my javascript callback funciton is called with result value( file string )

There are cases,however, in which calling such a callback is deferred for the next loop iteration. If the previous iteration deferred any I/O callback it will be run at this point.

So, If file read operation is end after uv__io_poll() is called, the callback function is inserted into pending_queue, and, called in the uv__run_pending() right?

My question is this

At what part of nodejs source code, was the deferred callback function inserted into pending_queue? and how?

[davisjam]

At what part of nodejs source code, was the deferred callback function inserted into pending_queue? and how?

See node/lib/fs.js and node/lib/internal/fs/read_file_context.js. There are a series of stages (stat, open, read-read-...-read, close), some handled in the ReadFileContext class, and concluding with ReadFileContext::readFileAfterClose which invokes callback(null, buffer) on success. That callback is user code that was assigned in node/lib/fs.js::readFile.

The callback invoked at the conclusion of fs.readFile is defined by Node.js-JS lib. Once inside the Node.js libs it invokes the appropriate user callback.

You can look at the bindings called from JS-land. These are defined in nodejs/src/node_file.cc.

Look for calls to uv_fs_X in nodejs/src/node_file.cc to see how Node.js-C++ uses the libuv APIs.

Not sure if this answers your question directly, but hopefully it's enough to get you going in the right direction.

[rpf5573]

Thanks you davisjam.

See node/lib/fs.js and node/lib/internal/fs/read_file_context.js. There are a series of stages (stat, open, read-read-...-read, close), some handled in the ReadFileContext class, and concluding with ReadFileContext::readFileAfterClose which invokes callback(null, buffer) on success. That callback is user code that was assigned in node/lib/fs.js::readFile. => yes, I read that part. there is callback(null, buffer) as you said.

And I think that the whole process of fs.readFile() is this.

1. fs.readFile(path, options, callback)

2. new ReadFileContext(callback, options.encoding) // this is what you pointed to me

3. const req = new FSReqCallback(); req.context = context; req.oncomplete = readFileAfterOpen;

4. now, context have my js callback and readFileAfterOpen function is pointing to uv_fs_fstat()

5. binding.open(pathModule.toNamespacedPath(path), stringToFlags(options.flag || 'r'), 0o666, req); // this is the end of readFile() function in js land.

6. Open() is called in node_file.cc

7. AsyncCall() -> AsyncDestCall()

8. req_wrap->Dispatch(fn, fn_args..., after) -> Call() -> fn is called. fn was uv_fs_open.

9. in uv_fs_open, uv__work_submit is called.

10. in uv__work_submit, post(&w->wq) is called.

11. in post, QUEUE_INSERT_TAIL(&wq, q) is called // insert my work into work queue.

12. worker is running and iterating the work queue already, and in that funciton, w->work is called. my work was uv__fs_work. so uv__fs_work is called

13. in uv__fs_work, uv__fs_open is called because req->fs_type was OPEN.

14. and open is called, I think this open is function of linux. the result return to w->work(w) again in worker.

15. after w->work, uv_async_send -> uv__async_send is called.

16. in uv__async_send, write some string in async_wfd. uv__io_poll in uv_run is checking all file description by using epoll_wait so,,, the main thread will know that fs.readFile is done.

17. in uv__io_poll, w->cb is called. w->cb is uv__async_io.

18. async_cb is called, and async_cb is uv__work_done. so uv__work_done is called and in that function w->done is called. w->done is uv__fs_done.

19. in uv__fs_done, req->cb(req) is called. req->cb was AfterInteger. This was set in uv_fs_open.( I'm not sure.. )

20. AfterInteger call req_wrap->Resolve(Integer::New(req_wrap->env()->isolate(), req->result)). the req->result was the result of uv__fs_open.

21. MakeCallback... I don't know what this function exactly does. But It seems that this function make connection with javascript callback function.

22. My js callback function may be called in readFileAfterClose as you said.

OK ! This is tooo long process. But ! I didnt find any connection with pending_queue in this whole process except uv__io_s structure!

My question is this.

1. How is the callback function of fs.readFile( filename, callback ) inserted into pending_queue ? Many people say that If I/O Operation complete, the callback function is inserted into pending_queue and uv__run_pending will iterate that pending_queue and call my callback function. Is this true?

2. What is the role of MakeCallback ?

3. If you think this question is unclear or I would not understand your answer because I have no big picture of Libuv, just tell me only the result please. Is my callback function which I set into fs.readFile( filename, callback ) will be inserted into pending_queue which uv__run_pending() function manage ?

Then, In what case does this happend ?

Check please !

[davisjam]

Your steps sound OK. Nice sleuthing.

OK ! This is tooo long process.

Indeed.

How is the callback function of fs.readFile( filename, callback ) inserted into pending_queue ?

It isn't, at least not directly.

I believe the (only?) callbacks libuv executes, certainly in this case, are functions defined in the Node-C++ libraries. The C++ library callbacks return into the Node-JS library. Once back in JS-land, the callback you provided to fs.readFile is triggered.

What is the role of MakeCallback ?

I don't know for certain. Ask over in the nodejs repo.

[rpf5573]

davisjam, thanks for your answer. I checked that open -> stat -> read -> close process in source code but, I didn't see any code to insert callback function into pending_queue.

I found uv_insert_pending_req function in nodejs source code. Yes ! this is what I expected. But, that is in a window only. How about Linux ? Is there the same or similar function with uv_insert_pending_req in Linux ???

Check please!