search

ros2 / ros2cli

ROS 2 command line interface tools
Apache License 2.0
173 stars 159 forks source link

make handles not inheritable to prevent from blocking durning tab-completion #852

Closed iuhilnehc-ynos closed 8 months ago

iuhilnehc-ynos commented 12 months ago

to fix https://github.com/ros2/ros2cli/issues/851

Please refer to the https://github.com/ros2/ros2cli/pull/852#issuecomment-1716847919

fujitatomoya commented 12 months ago

(TODO: why?)

big TODO though 😅 i will take a look.

iuhilnehc-ynos commented 12 months ago

This is definitely not the final solution as Foxy works well with the spawn_daemon.

iuhilnehc-ynos commented 12 months ago

@fujitatomoya Sorry, I need to turn it into a draft.

iuhilnehc-ynos commented 12 months ago

Mark: It's related to https://github.com/ros2/ros2cli/pull/652.

fujitatomoya commented 12 months ago

@hidmic by any chance, do you have any idea why https://github.com/ros2/ros2cli/pull/652 leads to https://github.com/ros2/ros2cli/issues/851?

hidmic commented 11 months ago

@fujitatomoya hmm, whatever it is, it is not obvious. Perhaps something funky about processes started to gather autocompletion options? A backtrace at the time of blocking would help.

iuhilnehc-ynos commented 11 months ago

Let me share my understanding,

  1. When the Tab key is pressed for ros2 topic echo /<tab> in the bash, the bash will fork a child bash to do the complete
gen_completion_matches

```shell (gdb) bt #0 __libc_fork () at ./posix/fork.c:41 #1 0x00005602d660c6c4 in make_child (command=command@entry=0x0, flags=4) at jobs.c:2175 #2 0x00005602d6619e29 in command_substitute ( string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6371 #3 0x00005602d661c8c9 in param_expand ( string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863 #4 0x00005602d661fa8c in expand_word_internal (word=, quoted=quoted@entry=0, isexp=isexp@entry=0, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329 #5 0x00005602d6621e1d in shell_expand_word_list (tlist=, eflags=30) at subst.c:11890 #6 0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8495890) at subst.c:12014 #7 expand_words_no_vars (list=list@entry=0x5602d8495890) at subst.c:11366 #8 0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:547 #9 0x00005602d66330ef in assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) at arrayfunc.c:820 #10 assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) at arrayfunc.c:810 #11 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:473 #12 0x00005602d661b70f in do_compound_assignment (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., name=0x5602d84aba10 "COMPREPLY") at subst.c:3190 --Type for more, q to quit, c to continue without paging--c #13 do_assignment_internal (word=, expand=expand@entry=1) at subst.c:3298 #14 0x00005602d66269ea in do_word_assignment (flags=0, word=) at subst.c:3345 #15 expand_word_list_internal (eflags=31, list=) at subst.c:11980 #16 expand_words (list=) at subst.c:11357 #17 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d849f510, async=0, pipe_out=-1, pipe_in=-1, simple_command=) at execute_cmd.c:4381 #18 execute_command_internal (command=0x5602d8498fd0, asynchronous=, pipe_in=-1, pipe_out=, fds_to_close=0x5602d849f510) at execute_cmd.c:846 #19 0x00005602d65f9e88 in execute_connection (command=0x5602d849c310, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:2718 #20 0x00005602d65f555a in execute_command_internal (command=0x5602d849c310, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:1020 #21 0x00005602d65f8207 in execute_command (command=0x5602d849c310) at execute_cmd.c:395 #22 0x00005602d65f9e4b in execute_connection (command=0x5602d8159610, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:2709 #23 0x00005602d65f555a in execute_command_internal (command=0x5602d8159610, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1020 #24 0x00005602d65f55a2 in execute_command_internal (command=0x5602d849d3d0, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1012 #25 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d849a650, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103 #26 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210) at execute_cmd.c:5162 #27 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d848e010, cs=0x5602d848e010, nw=, foundp=, cw=, lwords=0x5602d84924d0, ind=, line=0x5602d818e590 "ros2 topic echo /", text=0x5602d81bc9d0 "/", cmd=0x5602d848cc10 "ros2") at pcomplete.c:1155 #28 gen_compspec_completions (cs=cs@entry=0x5602d848e010, cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422 #29 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d848cc10 "ros2", cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594 #30 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649 #31 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bc9d0 "/", start=, end=17) at bashline.c:1665 #32 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bc9d0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 , found_quote=, quote_char=) at complete.c:1222 #33 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031 #34 0x00005602d66737cb in _rl_dispatch_subseq (key=, map=0x5602d66e22e0 , got_subseq=) at readline.c:887 #35 0x00005602d6673d98 in _rl_dispatch (map=, key=) at readline.c:833 #36 readline_internal_char () at readline.c:645 #37 0x00005602d667471d in readline_internal_charloop () at readline.c:694 #38 readline_internal () at readline.c:706 #39 readline (prompt=) at readline.c:385 #40 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488 #41 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422 #42 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358 #43 0x00005602d65e407a in read_token (command=) at /Users/chet/src/bash/src/parse.y:3290 #44 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254 #45 yylex () at /Users/chet/src/bash/src/parse.y:2797 #46 yyparse () at y.tab.c:1835 #47 0x00005602d65dd56b in parse_command () at eval.c:347 #48 0x00005602d65dd707 in read_command () at eval.c:391 #49 0x00005602d65dd92a in reader_loop () at eval.c:138 #50 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811 ```

  1. The child bash continue to fork to execute /usr/bin/python3 /home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2 to call the TopicNameCompleter
waitchld

```shell (gdb) bt #0 0x00007f4d764ea45a in __GI___wait4 (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffe56c1b560, options=options@entry=0, usage=usage@entry=0x0) at ../sysdeps/unix/sysv/linux/wait4.c:30 #1 0x00007f4d764ea41b in __GI___waitpid (pid=pid@entry=-1, stat_loc=stat_loc@entry=0x7ffe56c1b560, options=options@entry=0) at ./posix/waitpid.c:38 #2 0x00005602d660b6ca in waitchld (block=block@entry=1, wpid=) at jobs.c:3812 #3 0x00005602d660cf8a in wait_for (pid=2731517, flags=flags@entry=0) at jobs.c:2992 #4 0x00005602d65f7bc2 in execute_command_internal (command=0x5602d849a990, asynchronous=asynchronous@entry=0, pipe_in=pipe_in@entry=-1, pipe_out=pipe_out@entry=-1, fds_to_close=fds_to_close@entry=0x5602d81bd890) at execute_cmd.c:891 #5 0x00005602d664f979 in parse_and_execute ( string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., from_file=from_file@entry=0x5602d669eac5 "command substitution", flags=flags@entry=20) at evalstring.c:489 #6 0x00005602d661a5b4 in command_substitute ( string=string@entry=0x5602d84aca10 "IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6517 #7 0x00005602d661c8c9 in param_expand ( string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863 #8 0x00005602d661fa8c in expand_word_internal (word=, quoted=quoted@entry=0, isexp=isexp@entry=0, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329 #9 0x00005602d6621e1d in shell_expand_word_list (tlist=, eflags=30) at subst.c:11890 #10 0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8495890) at subst.c:12014 #11 expand_words_no_vars (list=list@entry=0x5602d8495890) at subst.c:11366 #12 0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:547 #13 0x00005602d66330ef in assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) at arrayfunc.c:820 #14 assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) --Type for more, q to quit, c to continue without paging--c at arrayfunc.c:810 #15 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:473 #16 0x00005602d661b70f in do_compound_assignment (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., name=0x5602d84aba10 "COMPREPLY") at subst.c:3190 #17 do_assignment_internal (word=, expand=expand@entry=1) at subst.c:3298 #18 0x00005602d66269ea in do_word_assignment (flags=0, word=) at subst.c:3345 #19 expand_word_list_internal (eflags=31, list=) at subst.c:11980 #20 expand_words (list=) at subst.c:11357 #21 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d849f510, async=0, pipe_out=-1, pipe_in=-1, simple_command=) at execute_cmd.c:4381 #22 execute_command_internal (command=0x5602d8498fd0, asynchronous=, pipe_in=-1, pipe_out=, fds_to_close=0x5602d849f510) at execute_cmd.c:846 #23 0x00005602d65f9e88 in execute_connection (command=0x5602d849c310, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:2718 #24 0x00005602d65f555a in execute_command_internal (command=0x5602d849c310, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849f510) at execute_cmd.c:1020 #25 0x00005602d65f8207 in execute_command (command=0x5602d849c310) at execute_cmd.c:395 #26 0x00005602d65f9e4b in execute_connection (command=0x5602d8159610, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:2709 #27 0x00005602d65f555a in execute_command_internal (command=0x5602d8159610, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1020 #28 0x00005602d65f55a2 in execute_command_internal (command=0x5602d849d3d0, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d849a650) at execute_cmd.c:1012 #29 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d849a650, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103 #30 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d8499210) at execute_cmd.c:5162 #31 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d848e010, cs=0x5602d848e010, nw=, foundp=, cw=, lwords=0x5602d84924d0, ind=, line=0x5602d818e590 "ros2 topic echo /", text=0x5602d81bc9d0 "/", cmd=0x5602d848cc10 "ros2") at pcomplete.c:1155 #32 gen_compspec_completions (cs=cs@entry=0x5602d848e010, cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422 #33 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d848cc10 "ros2", cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594 #34 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d848cc10 "ros2", word=word@entry=0x5602d81bc9d0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649 #35 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bc9d0 "/", start=, end=17) at bashline.c:1665 #36 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bc9d0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 , found_quote=, quote_char=) at complete.c:1222 #37 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031 #38 0x00005602d66737cb in _rl_dispatch_subseq (key=, map=0x5602d66e22e0 , got_subseq=) at readline.c:887 #39 0x00005602d6673d98 in _rl_dispatch (map=, key=) at readline.c:833 #40 readline_internal_char () at readline.c:645 #41 0x00005602d667471d in readline_internal_charloop () at readline.c:694 #42 readline_internal () at readline.c:706 #43 readline (prompt=) at readline.c:385 #44 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488 #45 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422 #46 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358 #47 0x00005602d65e407a in read_token (command=) at /Users/chet/src/bash/src/parse.y:3290 #48 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254 #49 yylex () at /Users/chet/src/bash/src/parse.y:2797 #50 yyparse () at y.tab.c:1835 #51 0x00005602d65dd56b in parse_command () at eval.c:347 #52 0x00005602d65dd707 in read_command () at eval.c:391 #53 0x00005602d65dd92a in reader_loop () at eval.c:138 #54 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811 ```

  1. And then TopicNameCompleter will call the spawn_daemon
call spawn_daemon

```shell (gdb) bt #0 0x00007fe55b71b7ed in __GI___select (nfds=nfds@entry=0, readfds=readfds@entry=0x0, writefds=writefds@entry=0x0, exceptfds=exceptfds@entry=0x0, timeout=timeout@entry=0x7ffcb21d6600) at ../sysdeps/unix/sysv/linux/select.c:69 #1 0x0000560fb71b6766 in pysleep (secs=) at ../Modules/timemodule.c:2076 #2 time_sleep (self=, obj=) at ../Modules/timemodule.c:370 #3 0x0000560fb709a854 in cfunction_vectorcall_O (func=, args=0x560fb97abb28, nargsf=, kwnames=) at ../Objects/methodobject.c:516 #4 0x0000560fb70898a2 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x560fb97abb28, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114 #5 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x560fb97abb28, callable=) at ../Include/cpython/abstract.h:123 #6 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcb21d6790, tstate=) at ../Python/ceval.c:5893 #7 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4181 #8 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb97ab940, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/daemonize.py, line 138, in daemonize (callable_=, tags={'name': 'ros2-daemon', 'ros_domain_id': 28, 'rmw_implementation': 'rmw_fastrtps_cpp'}, timeout=None, debug=False, prog='from ros2cli.daemon.daemonize import main; main()', cmd=['/usr/bin/python3', '-c', 'from ros2cli.daemon.daemonize import main; main()', '--name', 'ros2-daemon', '--ros-domain-id', '28', '--rmw-implementation', 'rmw_fastrtps_cpp'], name='rmw_implementation', value='rmw_fastrtps_cpp', flag='--rmw-implementation', kwargs={'stdin': -1, 'stdout': -3, 'stderr': -3, 'close_fds': False}, process=, _input=None, _communication_started=False, args=[...], stdin=<_io.BufferedWriter at remote 0x7fe530b771c0>, stdout=None, stderr=None, pid=2731550, returncode=None, encoding=None, errors=None, pipesize=-1, _devnull=6, text_mode=None, _sigint_wait_secs=<...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46 #9 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fe530b7a720, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #10 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #11 0x0000560fb70850d1 in _PyObject_VectorcallTstate (kwnames=('tags', 'timeout', 'debug'), nargsf=, args=, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114 #12 PyObject_Vectorcall (kwnames=('tags', 'timeout', 'debug'), nargsf=, args=, callable=) at ../Include/cpython/abstract.h:123 #13 call_function (kwnames=('tags', 'timeout', 'debug'), oparg=, pp_stack=, trace_info=0x7ffcb21d6970, tstate=) at ../Python/ceval.c:5893 #14 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4231 #15 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe559c15e40, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/node/daemon.py, line 139, in spawn_daemon (args= for more, q to quit, c to continue without paging--c , _registries={'action': {None: , 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None:...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46 #16 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fe530b7aba0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #17 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #18 0x0000560fb7083e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fe5575877c0, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114 #19 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fe5575877c0, callable=) at ../Include/cpython/abstract.h:123 #20 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcb21d6b50, tstate=) at ../Python/ceval.c:5893 #21 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4213 #22 0x0000560fb7090784 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe557587640, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/node/strategy.py, line 35, in __init__ (self=, args=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': , args=, locals=0x0, con=0x7fe530b7acc0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #24 _PyFunction_Vectorcall (kwnames=0x0, nargsf=, stack=, func=) at ../Objects/call.c:342 #25 _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=, args=, nargsf=, kwargs=) at ../Objects/call.c:142 #26 0x0000560fb70a5744 in _PyObject_Call_Prepend (kwargs=0x0, args=(, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, callable=, tstate=0x560fb90c0840) at ../Objects/call.c:431 #27 slot_tp_init (self=self@entry=, args=args@entry=(, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[) at ../Objects/typeobject.c:1135 #29 _PyObject_MakeTpCall (tstate=0x560fb90c0840, callable=, args=, nargs=, keywords=0x0) at ../Objects/call.c:215 #30 0x0000560fb7089c66 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fe5595d7290, callable=, tstate=) at ../Include/cpython/abstract.h:112 #31 _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fe5595d7290, callable=, tstate=) at ../Include/cpython/abstract.h:99 #32 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fe5595d7290, callable=) at ../Include/cpython/abstract.h:123 #33 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcb21d6e60, tstate=) at ../Python/ceval.c:5893 #34 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4213 #35 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe5595d7100, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2topic/ros2topic/api/__init__.py, line 74, in __call__ (self=, prefix='/', parsed_args=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , argcount=, args=, locals=0x0, con=0x7fe530b88ef0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #37 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #38 0x0000560fb709082d in _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=, args=, nargsf=, kwargs=) at ../Objects/call.c:153 #39 0x0000560fb70a654c in _PyObject_Call_Prepend (tstate=0x560fb90c0840, callable=, obj=, args=, kwargs={'prefix': '/', 'action': , 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, args=(), kwds={'prefix': '/', 'action': , 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, args=, nargs=, keywords=('prefix', 'action', 'parser', 'parsed_args')) at ../Objects/call.c:215 #42 0x0000560fb708a908 in _PyObject_VectorcallTstate (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=, args=, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:112 #43 _PyObject_VectorcallTstate (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=, args=, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:99 #44 PyObject_Vectorcall (kwnames=('prefix', 'action', 'parser', 'parsed_args'), nargsf=, args=, callable=) at ../Include/cpython/abstract.h:123 #45 call_function (kwnames=('prefix', 'action', 'parser', 'parsed_args'), oparg=, pp_stack=, trace_info=0x7ffcb21d71e0, tstate=) at ../Python/ceval.c:5893 #46 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4231 #47 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb9812a30, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 384, in _complete_active_option (self=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, argcount=, args=, locals=0x0, con=0x7fe55aeb3650, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #63 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #64 0x0000560fb7083f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x560fb92bf8c8, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114 #65 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x560fb92bf8c8, callable=) at ../Include/cpython/abstract.h:123 #66 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcb21d7780, tstate=) at ../Python/ceval.c:5893 #67 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4198 #68 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb92bf6a0, for file /home/chenlh/.local/lib/python3.10/site-packages/argcomplete/finders.py, line 176, in __call__ (argument_parser=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, argcount=, args=, locals=0x0, con=0x7fe55aeb35c0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #70 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #71 0x0000560fb709082d in _PyObject_FastCallDictTstate (tstate=0x560fb90c0840, callable=, args=, nargsf=, kwargs=) at ../Objects/call.c:153 #72 0x0000560fb70a654c in _PyObject_Call_Prepend (tstate=0x560fb90c0840, callable=, obj=, args=, kwargs={'exclude': ['-h', '--help']}) at ../Objects/call.c:431 #73 0x0000560fb71bf1e0 in slot_tp_call (self=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, nargs=, keywords=('exclude',)) at ../Objects/call.c:215 #75 0x0000560fb708a908 in _PyObject_VectorcallTstate (kwnames=('exclude',), nargsf=, args=, callable=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, args=, callable=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, args=, callable=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _actions=[, pp_stack=, trace_info=0x7ffcb21d7b00, tstate=) at ../Python/ceval.c:5893 #79 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4231 #80 0x0000560fb709b70c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x560fb92aa470, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/cli.py, line 62, in main (script_name='ros2', argv=None, description='ros2 is an extensible command-line tool for ROS 2.', extension=None, parser=, 'store': , 'store_const': , 'store_true': , 'store_false': , 'append': , 'append_const': , 'count': , 'help': , 'version': , 'parsers': , 'extend': }, 'type': {None: }}, _a...(truncated), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46 #81 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fe55b0490a0, tstate=0x560fb90c0840) at ../Python/ceval.c:5067 #82 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #83 0x0000560fb7083e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fe55b5439b8, callable=, tstate=0x560fb90c0840) at ../Include/cpython/abstract.h:114 #84 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fe55b5439b8, callable=) at ../Include/cpython/abstract.h:123 #85 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcb21d7ce0, tstate=) at ../Python/ceval.c:5893 #86 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4213 #87 0x0000560fb7174e56 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fe55b543840, for file /home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2, line 33, in (), tstate=0x560fb90c0840) at ../Include/internal/pycore_ceval.h:46 #88 _PyEval_Vector (tstate=0x560fb90c0840, con=, locals=, args=, argcount=, kwnames=) at ../Python/ceval.c:5067 #89 0x0000560fb7174cf6 in PyEval_EvalCode (co=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, locals=) at ../Python/ceval.c:1134 #90 0x0000560fb719f7d8 in run_eval_code_obj (tstate=0x560fb90c0840, co=0x7fe55b41cf50, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }) at ../Python/pythonrun.c:1291 #91 0x0000560fb71990bb in run_mod (mod=, filename=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, flags=, arena=) at ../Python/pythonrun.c:1312 #92 0x0000560fb719f525 in pyrun_file (fp=fp@entry=0x560fb909e530, filename=filename@entry='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', start=start@entry=257, globals=globals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, locals=locals@entry={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , '__file__': '/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', '__cached__': None, 're': , 'sys': , '__requires__': 'ros2cli', 'distribution': , 'importlib_load_entry_point': , 'load_entry_point': }, closeit=closeit@entry=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:1208 #93 0x0000560fb719ea08 in _PyRun_SimpleFileObject (fp=0x560fb909e530, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', closeit=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:456 #94 0x0000560fb719e653 in _PyRun_AnyFileObject (fp=0x560fb909e530, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', closeit=1, flags=0x7ffcb21d7f88) at ../Python/pythonrun.c:90 #95 0x0000560fb719141e in pymain_run_file_obj (skip_source_first_line=0, filename='/home/chenlh/Projects/ROS2/ros2-master/install/bin/ros2', program_name='/usr/bin/python3') at ../Modules/main.c:353 #96 pymain_run_file (config=0x560fb909f3a0) at ../Modules/main.c:372 #97 pymain_run_python (exitcode=0x7ffcb21d7f84) at ../Modules/main.c:587 #98 Py_RunMain () at ../Modules/main.c:666 #99 0x0000560fb7167cad in Py_BytesMain (argc=, argv=) at ../Modules/main.c:720 #100 0x00007fe55b629d90 in __libc_start_call_main (main=main@entry=0x560fb7167c70

, argc=argc@entry=2, argv=argv@entry=0x7ffcb21d8198) at ../sysdeps/nptl/libc_start_call_main.h:58 #101 0x00007fe55b629e40 in __libc_start_main_impl (main=0x560fb7167c70
, argc=2, argv=0x7ffcb21d8198, init=, fini=, rtld_fini=, stack_end=0x7ffcb21d8188) at ../csu/libc-start.c:392 #102 0x0000560fb7167ba5 in _start () ```

  1. The daemon is spawned with inheriting the handles from the parent process.
wait pickler to load data and call callable_

```shell (gdb) bt #0 0x00007fc026718d7f in __GI___poll (fds=0x7fc02658bf60, nfds=1, timeout=timeout@entry=7200000) at ../sysdeps/unix/sysv/linux/poll.c:29 #1 0x00007fc0267366c2 in __poll_chk (fds=, nfds=, timeout=timeout@entry=7200000, fdslen=fdslen@entry=18446744073709551615) at ./debug/poll_chk.c:27 #2 0x000055c2494c622d in poll (__timeout=, __nfds=, __fds=) at /usr/include/x86_64-linux-gnu/bits/poll2.h:39 #3 select_poll_poll_impl (timeout_obj=, self=0x7fbff4c683f0) at ../Modules/selectmodule.c:638 #4 select_poll_poll (self=0x7fbff4c683f0, args=, nargs=) at ../Modules/clinic/selectmodule.c.h:223 #5 0x000055c249388b28 in method_vectorcall_FASTCALL (func=, args=0x7fbffba9d3e8, nargsf=, kwnames=0x0) at ../Objects/descrobject.c:386 #6 0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fbffba9d3e8, callable=, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114 #7 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fbffba9d3e8, callable=) at ../Include/cpython/abstract.h:123 #8 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcafd3da90, tstate=) at ../Python/ceval.c:5893 #9 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4198 #10 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fbffba9d240, for file /usr/lib/python3.10/selectors.py, line 416, in select (self=}, _map=<_SelectorMapping(_selector=<...>) at remote 0x7fbff4c683d0>, _selector=) at remote 0x7fbff4c2bc70>, timeout=7200000, ready=[]), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46 #11 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fc026211400, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067 #12 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #13 0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fbff4c6c1d8, callable=, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114 #14 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fbff4c6c1d8, callable=) at ../Include/cpython/abstract.h:123 #15 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcafd3dc70, tstate=) at ../Python/ceval.c:5893 #16 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4198 #17 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fbff4c6c040, for file /usr/lib/python3.10/socketserver.py, line 294, in handle_request (self=, 'system.methodSignature': , 'system.methodHelp': , 'get_name': , 'get_namespace': , 'get_node_names_and_namespaces': , 'get_node_names_and_namespaces_with_enclaves': , 'get_topic_names_and_types': , 'get_service_names_and_types': , 'get_action_names_and_types': for more, q to quit, c to continue without paging--c te 0x7fbff4c64160>, 'get_publisher_names_and_types_by_node': , 'get_publishers_info_by_topic': , 'get_subscriber_names_and_types_by_node': , 'get_subscriptions_...(truncated), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46 #18 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fbffbc13380, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067 #19 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #20 0x000055c249361f52 in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x55c24bc8d3a0, callable=, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114 #21 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x55c24bc8d3a0, callable=) at ../Include/cpython/abstract.h:123 #22 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcafd3de50, tstate=) at ../Python/ceval.c:5893 #23 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4198 #24 0x000055c249374ca8 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x55c24bc8d1e0, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/__init__.py, line 127, in serve (server=, 'system.methodSignature': , 'system.methodHelp': , 'get_name': , 'get_namespace': , 'get_node_names_and_namespaces': , 'get_node_names_and_namespaces_with_enclaves': , 'get_topic_names_and_types': , 'get_service_names_and_types': , 'get_action_names_and_types': , 'get_publisher_names_and_types_by_node': , 'get_publishers_info_by_topic': , 'get_subscriber_names_and_types_by_node': , argcount=, args=, locals=0x0, con=0x7fbffba87800, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067 #26 _PyFunction_Vectorcall (kwnames=, nargsf=, stack=, func=) at ../Objects/call.c:342 #27 _PyObject_VectorcallTstate (tstate=0x55c24b5d5170, callable=, args=, nargsf=, kwnames=) at ../Include/cpython/abstract.h:114 #28 0x000055c249361e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fc02652eeb0, callable=, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114 #29 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc02652eeb0, callable=) at ../Include/cpython/abstract.h:123 #30 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcafd3e040, tstate=) at ../Python/ceval.c:5893 #31 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4213 #32 0x000055c24937970c in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fc02652ed40, for file /home/chenlh/Projects/ROS2/ros2-master/build/ros2cli/ros2cli/daemon/daemonize.py, line 81, in main (callable_=), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46 #33 _PyEval_Vector (kwnames=, argcount=, args=, locals=0x0, con=0x7fbffba87ad0, tstate=0x55c24b5d5170) at ../Python/ceval.c:5067 #34 _PyFunction_Vectorcall (func=, stack=, nargsf=, kwnames=) at ../Objects/call.c:342 #35 0x000055c249361e0d in _PyObject_VectorcallTstate (kwnames=0x0, nargsf=, args=0x7fc02652e188, callable=, tstate=0x55c24b5d5170) at ../Include/cpython/abstract.h:114 #36 PyObject_Vectorcall (kwnames=0x0, nargsf=, args=0x7fc02652e188, callable=) at ../Include/cpython/abstract.h:123 #37 call_function (kwnames=0x0, oparg=, pp_stack=, trace_info=0x7ffcafd3e220, tstate=) at ../Python/ceval.c:5893 #38 _PyEval_EvalFrameDefault (tstate=, f=, throwflag=) at ../Python/ceval.c:4213 #39 0x000055c249452e56 in _PyEval_EvalFrame (throwflag=0, f=Frame 0x7fc02652e020, for file , line 1, in (), tstate=0x55c24b5d5170) at ../Include/internal/pycore_ceval.h:46 #40 _PyEval_Vector (tstate=0x55c24b5d5170, con=, locals=, args=, argcount=, kwnames=) at ../Python/ceval.c:5067 #41 0x000055c249452cf6 in PyEval_EvalCode (co=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, locals=) at ../Python/ceval.c:1134 #42 0x000055c24947d7d8 in run_eval_code_obj (tstate=0x55c24b5d5170, co=0x7fc0264333c0, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }) at ../Python/pythonrun.c:1291 #43 0x000055c2494770bb in run_mod (mod=, filename=, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, flags=, arena=) at ../Python/pythonrun.c:1312 #44 0x000055c249470461 in PyRun_StringFlags (str=, start=257, globals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, locals={'__name__': '__main__', '__doc__': None, '__package__': None, '__loader__': , '__spec__': None, '__annotations__': {}, '__builtins__': , 'main': }, flags=0x7ffcafd3e440) at ../Python/pythonrun.c:1183 #45 0x000055c249470311 in PyRun_SimpleStringFlags (command=0x7fc02645e390 "from ros2cli.daemon.daemonize import main; main()\n", flags=0x7ffcafd3e440) at ../Python/pythonrun.c:503 #46 0x000055c24946f4d5 in pymain_run_command (command=) at ../Modules/main.c:248 #47 pymain_run_python (exitcode=0x7ffcafd3e434) at ../Modules/main.c:578 #48 Py_RunMain () at ../Modules/main.c:666 #49 0x000055c249445cad in Py_BytesMain (argc=, argv=) at ../Modules/main.c:720 #50 0x00007fc026629d90 in __libc_start_call_main (main=main@entry=0x55c249445c70

, argc=argc@entry=9, argv=argv@entry=0x7ffcafd3e648) at ../sysdeps/nptl/libc_start_call_main.h:58 #51 0x00007fc026629e40 in __libc_start_main_impl (main=0x55c249445c70
, argc=9, argv=0x7ffcafd3e648, init=, fini=, rtld_fini=, stack_end=0x7ffcafd3e638) at ../csu/libc-start.c:392 #52 0x000055c249445ba5 in _start () ```

(NOTE: build bash(source) and add time.sleep(60) somewhere in the daemonize to know every phase you want.)

Q: Why does the strace log show the ERESTARTSYS? A: The bash parent process expects the child process to exit with a SIGINT and bash parent process can read the pipe handle to exit if the peer pipe is closed. If a read operation continues to read again caused by a SIGINT, the strace could show the ERESTARTSYS if the pipe handle is still alive and there is no data in the handle.

bash parent process is blocked at read operation

```shell (gdb) bt #0 0x00007f4d76514992 in __GI___libc_read (fd=fd@entry=3, buf=buf@entry=0x7ffe56c1ba80, nbytes=nbytes@entry=512) at ../sysdeps/unix/sysv/linux/read.c:26 #1 0x00005602d666aa69 in read (__nbytes=512, __buf=0x7ffe56c1ba80, __fd=3) at /usr/x86_64-linux-gnu/include/bits/unistd.h:38 #2 zread (fd=fd@entry=3, buf=buf@entry=0x7ffe56c1ba80 "/parameter_events\v/rosout/\263O~\207\371\362\240\273\301V", len=len@entry=512) at zread.c:61 #3 0x00005602d661a27e in read_comsub (rflag=, flags=0, quoted=0, fd=3) at subst.c:6203 #4 command_substitute ( string=string@entry=0x5602d84ac810 "IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREAKS"..., quoted=quoted@entry=0, flags=flags@entry=0) at subst.c:6547 #5 0x00005602d661c8c9 in param_expand ( string=string@entry=0x5602d81ba610 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., sindex=sindex@entry=0x7ffe56c1be54, quoted=quoted@entry=0, expanded_something=expanded_something@entry=0x7ffe56c1bf18, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1be60, quoted_dollar_at_p=quoted_dollar_at_p@entry=0x7ffe56c1be58, had_quoted_null_p=0x7ffe56c1be5c, pflags=0) at subst.c:9863 #6 0x00005602d661fa8c in expand_word_internal (word=, quoted=quoted@entry=0, isexp=isexp@entry=0, contains_dollar_at=contains_dollar_at@entry=0x7ffe56c1bf1c, expanded_something=expanded_something@entry=0x7ffe56c1bf18) at subst.c:10329 #7 0x00005602d6621e1d in shell_expand_word_list (tlist=, eflags=30) at subst.c:11890 #8 0x00005602d6626af5 in expand_word_list_internal (eflags=30, list=0x5602d8494d50) at subst.c:12014 #9 expand_words_no_vars (list=list@entry=0x5602d8494d50) at subst.c:11366 #10 0x00005602d6631d60 in expand_compound_array_assignment (var=var@entry=0x5602d81b6010, value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:547 #11 0x00005602d66330ef in assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) at arrayfunc.c:820 #12 assign_array_var_from_string (flags=0, value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., var=0x5602d81b6010) at arrayfunc.c:810 #13 assign_array_from_string (name=name@entry=0x5602d84aba10 "COMPREPLY", value=value@entry=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., flags=flags@entry=0) at arrayfunc.c:473 #14 0x00005602d661b70f in do_compound_assignment (flags=0, --Type for more, q to quit, c to continue without paging--c value=0x5602d81ba410 "$(IFS=\"$IFS\"", ' ' , "COMP_LINE=\"$COMP_LINE\"", ' ' , "COMP_POINT=\"$COMP_POINT\"", ' ' , "COMP_TYPE=\"$COMP_TYPE\"", ' ' , "_ARGCOMPLETE_COMP_WORDBREAKS=\"$COMP_WORDBREA"..., name=0x5602d84aba10 "COMPREPLY") at subst.c:3190 #15 do_assignment_internal (word=, expand=expand@entry=1) at subst.c:3298 #16 0x00005602d66269ea in do_word_assignment (flags=0, word=) at subst.c:3345 #17 expand_word_list_internal (eflags=31, list=) at subst.c:11980 #18 expand_words (list=) at subst.c:11357 #19 0x00005602d65f72bc in execute_simple_command (fds_to_close=0x5602d81b71d0, async=0, pipe_out=-1, pipe_in=-1, simple_command=) at execute_cmd.c:4381 #20 execute_command_internal (command=0x5602d849d3d0, asynchronous=, pipe_in=-1, pipe_out=, fds_to_close=0x5602d81b71d0) at execute_cmd.c:846 #21 0x00005602d65f9e88 in execute_connection (command=0x5602d848cb90, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d81b71d0) at execute_cmd.c:2718 #22 0x00005602d65f555a in execute_command_internal (command=0x5602d848cb90, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d81b71d0) at execute_cmd.c:1020 #23 0x00005602d65f8207 in execute_command (command=0x5602d848cb90) at execute_cmd.c:395 #24 0x00005602d65f9e4b in execute_connection (command=0x5602d8499890, asynchronous=0, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:2709 #25 0x00005602d65f555a in execute_command_internal (command=0x5602d8499890, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:1020 #26 0x00005602d65f55a2 in execute_command_internal (command=0x5602d83ec450, asynchronous=, pipe_in=-1, pipe_out=-1, fds_to_close=0x5602d83e9890) at execute_cmd.c:1012 #27 0x00005602d65f957c in execute_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d848fc90, flags=flags@entry=0, fds_to_close=fds_to_close@entry=0x5602d83e9890, async=async@entry=0, subshell=subshell@entry=0) at execute_cmd.c:5103 #28 0x00005602d65f96ec in execute_shell_function (var=var@entry=0x5602d84a9210, words=words@entry=0x5602d848fc90) at execute_cmd.c:5162 #29 0x00005602d66464d5 in gen_shell_function_matches (cs=0x5602d8154590, cs=0x5602d8154590, nw=, foundp=, cw=, lwords=0x5602d849eed0, ind=, line=0x5602d849a110 "ros2 topic echo /", text=0x5602d81bb5f0 "/", cmd=0x5602d81bb5d0 "ros2") at pcomplete.c:1155 #30 gen_compspec_completions (cs=cs@entry=0x5602d8154590, cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998) at pcomplete.c:1422 #31 0x00005602d664702c in gen_progcomp_completions (ocmd=ocmd@entry=0x5602d81bb5d0 "ros2", cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1c998, retryp=0x7ffe56c1c99c, lastcs=0x7ffe56c1c9a0) at pcomplete.c:1594 #32 0x00005602d66471d6 in programmable_completions (cmd=cmd@entry=0x5602d81bb5d0 "ros2", word=word@entry=0x5602d81bb5f0 "/", start=start@entry=0, end=end@entry=17, foundp=foundp@entry=0x7ffe56c1ca34) at pcomplete.c:1649 #33 0x00005602d663f7af in attempt_shell_completion (text=0x5602d81bb5f0 "/", start=, end=17) at bashline.c:1665 #34 0x00005602d667dfbc in gen_completion_matches (text=text@entry=0x5602d81bb5f0 "/", start=start@entry=16, end=end@entry=17, our_func=our_func@entry=0x5602d667b780 , found_quote=, quote_char=) at complete.c:1222 #35 0x00005602d667e1ea in rl_complete_internal (what_to_do=9) at complete.c:2031 #36 0x00005602d66737cb in _rl_dispatch_subseq (key=, map=0x5602d66e22e0 , got_subseq=) at readline.c:887 #37 0x00005602d6673d98 in _rl_dispatch (map=, key=) at readline.c:833 #38 readline_internal_char () at readline.c:645 #39 0x00005602d667471d in readline_internal_charloop () at readline.c:694 #40 readline_internal () at readline.c:706 #41 readline (prompt=) at readline.c:385 #42 0x00005602d65de056 in yy_readline_get () at /Users/chet/src/bash/src/parse.y:1488 #43 0x00005602d65e0ac0 in yy_getc () at /Users/chet/src/bash/src/parse.y:1422 #44 shell_getc (remove_quoted_newline=remove_quoted_newline@entry=1) at /Users/chet/src/bash/src/parse.y:2358 #45 0x00005602d65e407a in read_token (command=) at /Users/chet/src/bash/src/parse.y:3290 #46 0x00005602d65e75e2 in read_token (command=0) at /Users/chet/src/bash/src/parse.y:3254 #47 yylex () at /Users/chet/src/bash/src/parse.y:2797 #48 yyparse () at y.tab.c:1835 #49 0x00005602d65dd56b in parse_command () at eval.c:347 #50 0x00005602d65dd707 in read_command () at eval.c:391 #51 0x00005602d65dd92a in reader_loop () at eval.c:138 #52 0x00005602d65dc583 in main (argc=1, argv=0x7ffe56c1de38, env=0x7ffe56c1de48) at shell.c:811 ```

Q: Why is the pipe handle alive? A: Because the pipe handle from bash parent process is inherited into the daemon. (bash -> bash -> python3(ros2) -> python3(daemon))

pipe handle is not closed (show the final state)

```shell (ros2 NOT SETUP) [09:58:14 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) chenlh bash-5.1 $ ps aux | grep bash chenlh 1256201 0.0 0.0 10792 7652 pts/5 S+ Sep12 0:00 ./bash chenlh 2835661 0.0 0.0 6556 2348 pts/18 S+ 09:58 0:00 grep bash (ros2 NOT SETUP) [09:58:20 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) chenlh bash-5.1 $ ps aux | grep ros2 chenlh 50931 0.2 0.5 38691908 191560 ? SLsl Sep12 3:32 /usr/share/code/code --enable-crashpad ./ros2-master.code-workspace chenlh 2794616 0.3 0.2 1292928 78980 pts/5 Sl+ 09:51 0:01 /usr/bin/python3 -c from ros2cli.daemon.daemonize import main; main() --name ros2-daemon --ros-domain-id 28 --rmw-implementation rmw_fastrtps_cpp chenlh 2836266 0.0 0.0 6556 2444 pts/18 S+ 09:58 0:00 grep ros2 (ros2 NOT SETUP) [09:58:31 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) chenlh bash-5.1 $ ls -l /proc/1256201/fd total 0 lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 0 -> /dev/pts/5 lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 1 -> /dev/pts/5 lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 2 -> /dev/pts/5 lrwx------ 1 chenlh chenlh 64 Sep 12 17:17 255 -> /dev/pts/5 lr-x------ 1 chenlh chenlh 64 Sep 12 17:17 3 -> 'pipe:[13549234]' (ros2 NOT SETUP) [09:58:42 OptiPlex-7080] /home/chenlh/Projects/bash/debian/bash-5.1 (main modified) chenlh bash-5.1 $ ls -l /proc/2794616/fd total 0 l-wx------ 1 chenlh chenlh 64 Sep 13 09:58 0 -> /home/chenlh/.ros/log/python3_2794616_1694569867039.log lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 1 -> /dev/null lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 10 -> 'socket:[13556055]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 11 -> 'socket:[13556056]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 12 -> 'socket:[13556057]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 13 -> 'socket:[13556058]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 14 -> 'anon_inode:[eventfd]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 15 -> 'anon_inode:[eventpoll]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 16 -> 'anon_inode:[timerfd]' lr-x------ 1 chenlh chenlh 64 Sep 13 09:58 17 -> /dev/shm/fastrtps_c777f11058a465d2_el lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 18 -> 'socket:[13556074]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 19 -> 'socket:[13556080]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 2 -> /dev/null lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 20 -> 'socket:[13556083]' lr-x------ 1 chenlh chenlh 64 Sep 13 09:58 21 -> /dev/shm/fastrtps_port14413_el lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 22 -> 'socket:[13556089]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 23 -> 'socket:[13556090]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 24 -> 'socket:[13556091]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 3 -> 'socket:[13549244]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 4 -> 'socket:[13556051]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 5 -> 'socket:[13556052]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 6 -> 'socket:[13556053]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 7 -> 'socket:[13556054]' l-wx------ 1 chenlh chenlh 64 Sep 13 09:58 8 -> 'pipe:[13549234]' lrwx------ 1 chenlh chenlh 64 Sep 13 09:58 9 -> /dev/pts/5 ```

Notice that the pipe handle id (8) is from register-python-argcomplete3 / register-python-argcomplete, which is not a random number.

pipe handle id (8)

```shell COMPREPLY=( $(IFS="$IFS" \ COMP_LINE="$COMP_LINE" \ COMP_POINT="$COMP_POINT" \ COMP_TYPE="$COMP_TYPE" \ _ARGCOMPLETE_COMP_WORDBREAKS="$COMP_WORDBREAKS" \ _ARGCOMPLETE=1 \ _ARGCOMPLETE_SUPPRESS_SPACE=$SUPPRESS_SPACE \ "$1" 8>&1 9>&2 1>/dev/null 2>/dev/null) ) or __python_argcomplete_run_inner() { if [[ -z "${_ARC_DEBUG-}" ]]; then "$@" 8>&1 9>&2 1>/dev/null 2>&1 else "$@" 8>&1 9>&2 1>&9 2>&1 fi } ```

The easiest way to fix this issue is to close the pipe handle (8) in daemon at the beginning.

diff --git a/ros2cli/ros2cli/daemon/daemonize.py b/ros2cli/ros2cli/daemon/daemonize.py
index d8994c4..0b0e632 100644
--- a/ros2cli/ros2cli/daemon/daemonize.py
+++ b/ros2cli/ros2cli/daemon/daemonize.py
@@ -76,6 +76,11 @@ def main():
     callable_ = pickle.load(sys.stdin.buffer)
     sys.stdin.close()
     os.close(0)  # force C stream close
+    try:
+        # force pipe handle(8) to close if available
+        os.close(8)
+    except OSError:
+        pass
     return callable_()

I'd like to hear your opinion.

clalancette commented 11 months ago

I'd like to hear your opinion.

@iuhilnehc-ynos Wow, thanks for the amazing analysis and finding the problem.

I guess my issue with just closing handle 8 is that it seems like this number could move around on different operating systems. For instance, when we switch from Ubuntu 22.04 to 24.04, or when we are on RHEL. It just seems fragile to rely on that.

Now that we understand the problem, it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem. What do you think?

hidmic commented 11 months ago

That was a deep, deep rabbit hole @iuhilnehc-ynos. Impressive archaeology work.

I guess my issue with just closing handle 8 is that it seems like this number could move around on different operating systems.

Yeah, it's brittle.

Now that we understand the problem, it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem.

Well... Spawning the daemon may be slow, and I've seen the daemon end up in weird, inconsistent states that do not match the actual ROS graph, but without it you are exposed to DDS discovery latency.

Looking at subprocess.Popen documentation again, it appears there is a way to close all file descriptors but a few. See pass_fds on Linux and handle_list on Windows. You'll have to explicitly pass the socket around to daemonize, but IMO that'd be the future-proof solution. I missed this completely when I was working on #652, my bad. :facepalm:

fujitatomoya commented 11 months ago

build bash(source) and add time.sleep(60) somewhere in the daemonize to know every phase you want.

wow..., that is deep... amazing analysis @iuhilnehc-ynos 🚀 thanks!

it seems to me it is better to skip spawning the daemon for the completer. It will probably improve completion performance (a bit), and will avoid this problem. What do you think? inconsistent states that do not match the actual ROS graph without it you are exposed to DDS discovery latency.

all these are true. i second the idea to skip spawning the daemon for the auto-completer. besides, after the command is issued, daemon will start running.

iuhilnehc-ynos commented 9 months ago

I tested the tab completer for most ros2cli commands but not all.

fujitatomoya commented 9 months ago

@iuhilnehc-ynos can you start the CI? i think this is really good to have.

iuhilnehc-ynos commented 9 months ago

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status
clalancette commented 9 months ago

After looking at this again, I think my previous comments were wrong. I'm very sorry about this, but I think this needs to change direction. Let me explain why.

When we are trying to do completion for topic names, node names, service names, or action names, we really do want to get the information from the daemon. That's because the daemon will have the most up-to-date information from the graph, and we really want to rely on that. If we never spawn the daemon here, then we can easily get into a situation where tab-completion will never discover some of these graph items. That means that there will be a difference between what tab-completion shows, and what ros2 topic list (for instance) shows, since the latter will actually spawn the daemon.

Let me enumerate a few different ways to fix the problem in the attached issue:

  1. What is currently in this PR, which essentially never uses the daemon for completion. This solves the blocking problem, but since it never uses the daemon it can easily not show the proper information for completion.
  2. The previous solution that @iuhilnehc-ynos had, which would use the daemon if it was around, but wouldn't spawn it if it wasn't. I think this is better than the first solution, since we'll at least use the daemon if it is there. But I don't think this is the best solution, since we never spawn the daemon at all.
  3. The original solution that @iuhilnehc-ynos proposed in https://github.com/ros2/ros2cli/pull/852#issuecomment-1716847919 , where we close the specific handle causing the issue. Conceptually I like that solution, but I don't like the hard-coded file descriptor number; that will be easy to break in the future.
  4. Something similar to 3, where before we spawn the daemon, we close all unnecessary file handles. What I'm thinking about here is that we change https://github.com/ros2/ros2cli/blob/419afea1282ae4daa43ea446a2c0bcc7394f856c/ros2cli/ros2cli/daemon/daemonize.py#L128 to True (https://docs.python.org/3/library/subprocess.html#subprocess.Popen). Since the inheritable flag for the server socket is already set to True, in theory this should work. But there are probably some additional complications I don't know about.

Thoughts about all of this?

iuhilnehc-ynos commented 9 months ago

That's because the daemon will have the most up-to-date information from the graph, and we really want to rely on that. If we never spawn the daemon here, then we can easily get into a situation where tab-completion will never discover some of these graph items. That means that there will be a difference between what tab-completion shows, and what ros2 topic list (for instance) shows, since the latter will actually spawn the daemon.

I totally agree on this.

4. Something similar to 3, where before we spawn the daemon, we close all unnecessary file handles.

In the 3, the patch is to close the pipe handle(8) in the spawned daemon process. I don't think we can close the pipe handle(8) before we spawn the daemon, at least it doesn't work at the first time.

4. What I'm thinking about here is that we change https://github.com/ros2/ros2cli/blob/419afea1282ae4daa43ea446a2c0bcc7394f856c/ros2cli/ros2cli/daemon/daemonize.py#L128 to True (https://docs.python.org/3/library/subprocess.html#subprocess.Popen). Since the inheritable flag for the server socket is already set to True, in theory this should work. But there are probably some additional complications I don't know about.

I am not certain if you just suggested me to set kwargs['close_fds'] with True. It seems to work on the surface, but the daemon is NOT spawned successfully. I think that the inheritable flag should work with close_fds(false).

https://docs.python.org/3/library/subprocess.html#subprocess.Popen

If close_fds is true, all file descriptors except 0, 1 and 2 will be closed before the child process is executed. Otherwise when close_fds is false, file descriptors obey their inheritable flag as described in [Inheritance of File Descriptors](https://docs.python.org/3/library/os.html#fd-inheritance).

https://docs.python.org/3/library/os.html#fd-inheritance

Using the [subprocess](https://docs.python.org/3/library/subprocess.html#module-subprocess) module, all file descriptors except standard streams are closed, and inheritable handles are only inherited if the close_fds parameter is False.
clalancette commented 8 months ago

I am not certain if you just suggested me to set kwargs['close_fds'] with True. It seems to work on the surface, but the daemon is NOT spawned successfully. I think that the inheritable flag should work with close_fds(false).

Yeah, you are totally right about this. I misread the documentation.

In the 3, the patch is to close the pipe handle(8) in the spawned daemon process. I don't think we can close the pipe handle(8) before we spawn the daemon, at least it doesn't work at the first time.

Hm, I see. OK.

So that leaves us with either solution 2 (use the daemon if it is there, but don't spawn), or 3 (close the handle after spawning).

I think we should think about option 3 a little further, and maybe go back to how this whole thing is designed. Starting from https://github.com/ros2/ros2cli/blob/419afea1282ae4daa43ea446a2c0bcc7394f856c/ros2cli/ros2cli/node/daemon.py#L102 , we do something like the following:

  1. During the execution of something like ros2 topic echo, attempt to create the XMLRPC server in this process. If that fails with EADDRINUSE, the daemon already exists and we have no more work to do. If that succeeds, then we'll proceed.
  2. Call daemonize() to create the new daemon process. In turn, this will: a. Setup a bunch of kwargs (including setting close_fds to False) b. Call subprocess.Popen to create a new process c. Pickle the current state of this process. d. Send that pickle over to the new process
  3. Close the server socket in this process. Since the new daemon has inherited the FD, it now has control of the XMLRPC server.

So I'm thinking about a few different ways to tackle this. These are all ideas/questions, nothing concrete:

  • Do we really have to create the XMLRPC server in the ros2 topic echo process? The comments say that it is to prevent a TOCTTOU race, but maybe there is another way we could prevent that race, and still create the XMLRPC server in the daemonized process instead?
  • If we do want to continue to create the XMLRPC server in the ros2 topic echo process, maybe we can reduce the FDs we are passing around. That is, maybe we can set close_fds to True, and somehow only send over the FDs we care about? https://stackoverflow.com/questions/2989823/how-to-pass-file-descriptors-from-parent-to-child-in-python suggests it is possible, but I haven't played with it.
  • The process of daemonizing doesn't seem to be exactly what I remember, at least in Unix. https://stackoverflow.com/questions/17954432/creating-a-daemon-in-linux/17955149#17955149 has a good overview; maybe we need to do more of that here?
  • Finally, one last thought; we could go with the solution of closing the fd(8), if we were certain we would know if it changed via a test or something. Maybe we can somehow get some meta-information about that file descriptor?

Anyway, this is all just food for thought. Let me know if you have additional ideas.

iuhilnehc-ynos commented 8 months ago
  • maybe there is another way we could prevent that race, and still create the XMLRPC server in the daemonized process instead?
  • The process of daemonizing doesn't seem to be exactly what I remember, at least in Unix.

I think what we can do are the following things.

  • use lockfile(e.g. $HOME/.ros/daemon-$ROS_DOMAIN_ID.lock) to avoid the TOC/TOU.
  • make the XMLRPC server created in the daemonized process.
  • use a better way to create a daemon in Unix-like.
  • Finally, one last thought; we could go with the solution of closing the fd(8)

To close the fd(8) is a bad solution, please forget it.

clalancette commented 8 months ago

I had one more thought about this over the weekend.

We are currently setting close_fds to False when we do the subprocess.Popen. That means that all fds that are not marked inheritable should be closed. So before the subprocess.Popen, we could go through and mark every fd except for 0, 1, 2, and the XMLRPC socket as non-inheritable, and they should get closed when the subprocess.Popen happens. There is a slight complication with Windows (see https://docs.python.org/3/library/os.html#fd-inheritance), but we can handle that.

Otherwise, if you don't think that is a good idea or it doesn't work for some reason, we can go with your proposal of a lockfile. If we go that way, the lockfile name should include both the DOMAIN_ID, as well as the RMW_IMPLEMENTATION in use. And we'll also have to see how lockfiles work on Windows.

iuhilnehc-ynos commented 8 months ago

That means that all fds that are not marked inheritable should be closed. So before the subprocess.Popen, we could go through and mark every fd except for 0, 1, 2, and the XMLRPC socket as non-inheritable, and they should get closed when the subprocess.Popen happens.

:+1:

There is a slight complication with Windows (see https://docs.python.org/3/library/os.html#fd-inheritance), but we can handle that.

As I have no windows development environment at company, I'll address it later.

Otherwise, if you don't think that is a good idea or it doesn't work for some reason, we can go with your proposal of a lockfile. If we go that way, the lockfile name should include both the DOMAIN_ID, as well as the RMW_IMPLEMENTATION in use. And we'll also have to see how lockfiles work on Windows.

I'll leave it in the future.

iuhilnehc-ynos commented 8 months ago

It seems a bit rude, but it does work currently.

iuhilnehc-ynos commented 8 months ago

I am not sure if the tab-completion of ros2 in Windows is supported or not. I did the test last night, but it doesn't work, so I decided to remove the set not inheritable actions for windows.

fujitatomoya commented 8 months ago

@clalancette friendly ping, lgtm for me. i guess windows should work?

iuhilnehc-ynos commented 8 months ago

CI:

  • Linux Build Status
  • Linux-aarch64 Build Status
  • Windows Build Status
fujitatomoya commented 8 months ago

@Mergifyio backport iron humble

mergify[bot] commented 8 months ago

backport iron humble

✅ Backports have been created

* [#872 make handles not inheritable to prevent from blocking durning tab-completion (backport #852)](https://github.com/ros2/ros2cli/pull/872) has been created for branch `iron` * [#873 make handles not inheritable to prevent from blocking durning tab-completion (backport #852)](https://github.com/ros2/ros2cli/pull/873) has been created for branch `humble`
akssri-sony commented 6 months ago

Looks like this is causing ros2 topic daemon to hang in arch-linux, but works fine on Ubuntu-20.04. Unclear why.

Edit: Bug report, https://github.com/ros2/ros2/issues/1531

akssri-sony commented 6 months ago

The limit value (soft in L143) is way higher in Arch (1073741816) vs Ubuntu (1048576 - ~1000x).

Edit: Can confirm it's getting stuck in this loop.

Is there a better way to do this and/or limit the bounds of the loop ?

(Arch is at 6.7.8-arch-1; Ubuntu ~ 5.15).

akssri-sony commented 6 months ago

Looks like this is something that is set by (newer ? some ?) versions of systemd to a higher value (~2^30) by default.

  • © Githubissues.
  • Githubissues is a development platform for aggregating issues.