client_main not invoked as dlsym() wasn't found.

[swick]

$ sudo hotpatcher -l /home/sebastian/prog/libglcapture/build/src/libglcapture-client.so -s client_main -vvv 29673 2>&1
Options Given:
Verbose Level: 2
Process PID: 29673
Symbol name: client_main
Library name: /home/sebastian/prog/libglcapture/build/src/libglcapture-client.so
Dry run: false
[exe_load_headers:490] Entry point 0x400750
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x400238
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_load_maps:278] Max number of mappings present: 23
[ld_load_maps:288] Allocated memory to load proc maps.
[ld_find_library:440] Found entry /lib/x86_64-linux-gnu/ld-2.15.so matching /lib64/ld-linux-x86-64.so.2
[ld_find_library:376] Doing best substring search for libc.
[ld_find_library:440] Found entry /lib/x86_64-linux-gnu/libc-2.15.so matching libc
[ld_find_library:376] Doing best substring search for libdl.
[ld_find_library:447] Library libdl not found in procmaps
[hotpatch_gather_functions:102] libdl not mapped.
[ld_find_library:376] Doing best substring search for libpthread.
[ld_find_library:447] Library libpthread not found in procmaps
[hotpatch_gather_functions:104] libpthread not mapped.
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:106] Found malloc at 0x7f952cd2df40 in libc
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:107] Found realloc at 0x7f952cd2e680 in libc
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:108] Found free at 0x7f952cd2e580 in libc
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:126] Found __libc_dlopen_mode at 0x7f952cddb690 in libc
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:127] Found __libc_dlclose at 0x7f952cddb7f0 in libc
[exe_load_headers:490] Entry point 0x21880
[exe_load_program_headers:414] PT_INTERP section found
[exe_load_program_headers:441] Found /lib64/ld-linux-x86-64.so.2 at V-Addr 0x184410
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:450] PT_LOAD section found
[exe_load_program_headers:446] PT_DYNAMIC section found
[ld_find_address:489] 2189 symbols found in /lib/x86_64-linux-gnu/libc-2.15.so
[hotpatch_gather_functions:128] Found __libc_dlsym at 0x7f952cddb740 in libc
[hotpatch_gather_functions:150] Pthread's symbol not found. Will disable pthread usage in injection.
[hotpatch_inject_library:620] Allocating 1024 bytes in the target.
[hotpatch_inject_library:694] Attaching to PID 29673
[hotpatch_inject_library:698] Waiting...
[hotpatch_inject_library:703] Getting original registers.
[hotpatch_inject_library:708] Copying stack out.
[hotpatch_inject_library:717] Copying Null to stack.
[hotpatch_inject_library:719] Setting registers and invoking malloc.
[hotpatch_inject_library:719] Executing...
[hotpatch_inject_library:719] Waiting...
[hotpatch_inject_library:719] Getting registers.
[hotpatch_inject_library:725] Copying 1024 bytes to 0x1843010.
[hotpatch_inject_library:731] Copying Null to stack.
[hotpatch_inject_library:734] Setting registers and invoking dlopen.
[hotpatch_inject_library:734] Executing...
[hotpatch_inject_library:734] Waiting...
[hotpatch_inject_library:734] Getting registers.
[hotpatch_inject_library:737] Dll opened at 0x0
[hotpatch_inject_library:779] client_main not invoked as dlsym() wasn't found.
[hotpatch_inject_library:789] Setting original registers.
[hotpatch_inject_library:797] Copying stack back.
[hotpatch_inject_library:806] Executing...
Dll was injected at (nil)
Invocation of client_main() returned (nil)

I'm not sure why it doesn't work.

client_main not invoked as dlsym() wasn't found

even so it says

Found __libc_dlsym at 0x7f952cddb740 in libc

I think the real problem here is that dlopen returns 0:

Dll opened at 0x0

Or maybe it's just me being dumb.

[vikasnkumar]

Are you using the latest code from the repository ? Hotpatch does not manage library dependencies. libdl manages library dependencies. So if your libglcapture-client.so is injected into a process that does not link with libdl you will have to inject every .so file that libglcapture-client.so is dependent on except for libc. You can use "ldd -r libglcapture-client.so" to give you that information.

Please give me the following info: a) OS type/name/details such as Ubuntu vs debian vs CentoS b) Can you try injecting the hotpatchtest.so library into the process and see if it writes to /tmp/hotpatchtest.log file ? We need to separate out the failure of hotpatch working on your system vs injecting your custom library.

[vikasnkumar]

I think i fixed this problem in the latest checkin of hotpatch a few weeks ago. Please try that first.

[swick]

Thanks for the reply. I'm on Ubuntu 12.10 x86_64. I tried to inject libhotpatchtest.so but the output is still the same. The funny thing is, that it works just fine with other target programs. I guess there is something a program must link to because I made a minimal program and tried to inject libhotpatchtest.so and it failed with the same output

#include <stdio.h>

int main(int argc, char **argv) {
    while(1) {
        printf(".\n");
        sleep(1);
    }
    return 0;
}

[vikasnkumar]

There is a dummy program I provide that you can inject into.

When you build hotpatch from source using "make" in a new shell start the dummy program in the following way:

./Release/test/dummy

Then from another shell run the following:

PID=pgrep dummy ./Release/src/hotpatcher -l $PWD/Release/test/libhotpatchtest.so $PID

cat /tmp/hotpatchtest.log

See if this works.

If this works, then you can compare your program with the dummy program.

This should work without "sudo".

If it doesn't work, we need to investigate further.

--Vikas On 01/30/2013 04:46 PM, Sebastian Wick wrote:

Thanks for the reply. I'm on Ubuntu 12.10 x86_64. I tried to inject libhotpatchtest.so but the output is still the same. The funny thing is, that it works just fine with other target programs. I guess there is something a program must link to because I made a minimal program and tried to inject libhotpatchtest.so and it failed with the same output

|#include

int main(int argc, char **argv) { while(1) { printf(".\n"); sleep(1); } return 0; } |

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-12914364.

[swick]

I need the sudo because Ubuntu needs super user rights to use ptrace. Same problem as always:

symbol is 0
Dll was injected at (nil)
Invocation of _init() returned (nil)

and the file /tmp/hotpatchtest.log doesn't exist.

[vikasnkumar]

Are you saying Ubuntu needs a sudo to run gdb on any program at all times now ? I have not tested Hotpatch on 12.10 so cannot guarantee that it works there. It works on 11.10 and 12.04 of Ubuntu, and the more recent versions of CentOS and Debian.

Btw, are you using the latest code from Github ?

On 01/31/2013 07:54 AM, Sebastian Wick wrote:

I need the sudo because Ubuntu needs super user rights to use ptrace. Same problem as always:

symbol is 0 Dll was injected at (nil) Invocation of _init() returned (nil)

and the file /tmp/hotpatchtest.log doesn't exist.

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-12941044.

[swick]

gdb needs sudo to attach itself to a proccess which is not a child of gdb since Ubuntu 10.10 (https://wiki.ubuntu.com/Security/Features)

And yes, I use the lastest master.

[vikasnkumar]

Ok. Have you tried not using sudo. Just run the test as a root user or turn the Security Features off temporarily to test.

$ sudo su -

$ ./hotpatcher ....

Just want to confirm that sudo is not causing problems because of differences between user id and effective user id. Hotpatch doesn't care about it but still.

I will have to take care of Ubuntu next week.

On 01/31/2013 09:46 AM, Sebastian Wick wrote:

gdb needs sudo to attach itself to a proccess which is not a child of gdb since Ubuntu 10.10 (https://wiki.ubuntu.com/Security/Features)

And yes, I use the lastest master.

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-12945430.

[swick]

Tried sudo su first, then changed the kernel setting. No change at all. Same problem, same output.

[swick]

Any news, yet? I'd like to help but I have to idea how to debug it ;)

[vikasnkumar]

Sorry, I have not found time to debug the problem. I just installed Ubuntu 12.04 today on a laptop for another project. Maybe tonight I can finally get time to look at hotpatch.

Are you trying on 32 or 64 bit systems ?

On 02/04/2013 02:34 PM, Sebastian Wick wrote:

Any news, yet? I'd like to help but I have to idea how to debug it ;)

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-13094667.

[vikasnkumar]

Ok. I tested on Ubuntu 12.04 LTS and hotpatch seems to work in my test case.

I have the dummy programming running in a bash shell.

I run hotpatcher to inject libhotpatchtest.so into dummy and it succeeds and writes to the /tmp/hotpatchtest.log file.

I had to use sudo to do this.

On 02/04/2013 02:34 PM, Sebastian Wick wrote:

Any news, yet? I'd like to help but I have to idea how to debug it ;)

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-13094667.

[swick]

I'm on a 64 bit system and it doesn't work here. Maybe I messed something up, will try to test on another system.

[swick]

Tested on a fresh ubuntu 12.10 64bit system, same error.

$ git clone https://github.com/vikasnkumar/hotpatch.git
$ mkdir build
$ cd build
$ cmake ../hotpatch/
$ make
$ ./test/dummy &
$ sleep 1
$ sudo ./src/hotpatcher -l ./test/libhotpatchtest.so $(pidof dummy)
Dll was injected at (nil)
Invocation of _init() returned (nil)
$ sleep 1
$ killall dummy
$ cat /tmp/hotpatchtest.log
cat: /tmp/hotpatchtest.log: No such file or directory

[vikasnkumar]

Please can you try this without running cmake yourself. The top level makefile calls Cmake itself by setting up the appropriate cmake variables as needed.

In a new terminal, please run the following $ git clone https://github.com/vikasnkumar/hotpatch.git $ make $ ./Release/test/dummy

In a separate terminal $ PID=$(pgrep dummy) $ cd Release $ sudo $PWD/src/hotpatcher -l $PWD/test/libhotpatchtest.so $PID

Thanks Vikas

On 02/04/2013 04:26 PM, Sebastian Wick wrote:

Tested on a fresh ubuntu 12.10 64bit system, same error.

$ git clone https://github.com/vikasnkumar/hotpatch.git $ mkdir build $ cd build $ cmake ../hotpatch/ $ make $ ./test/dummy & $ sleep 1 $ sudo ./src/hotpatcher -l ./test/libhotpatchtest.so $(pidof dummy) Dll was injected at (nil) Invocation of _init() returned (nil) $ sleep 1 $ killall dummy $ cat /tmp/hotpatchtest.log cat: /tmp/hotpatchtest.log: No such file or directory

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-13100400.

[swick]

Same problem.

[vikasnkumar]

What are you really trying to do with hotpatch ? Is using Ubuntu 12.10 necessary ? Can you not use it on an earlier version ? I am not sure what has changed with Ubuntu 12.10 vs the earlier versions. I will have to find time and a system to install Ubuntu 12.10 64-bit on.

On 02/04/2013 05:28 PM, Sebastian Wick wrote:

Same problem.

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-13103522.

[vikasnkumar]

I am downloading Ubuntu 12.10 64-bit to run it on a VM... let's see what the real problem is .

On 02/04/2013 05:28 PM, Sebastian Wick wrote:

Same problem.

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-13103522.

[vikasnkumar]

Ok. I am testing it on 12.10. Yes there is something off with the return value of dlopen() and I will investigate that when I get time this weekend.

However, the library does get added to the memory map as you can see by running

$ grep hotpatchtest /proc/$(pgrep dummy)/maps

Not sure why dlopen() is returning NULL which means somewhere an error occurred in dlopen() itself. So the library is partially loaded and when you try to inject it again into the program it says inconsistency detected.

More debugging is necessary. In the meantime hotpatch works fine on Debian 6.0, CentOS 6.2 and Ubuntu 12.04.

When I find time to fix for 12.10 I will until then you might have to use a system which is supported.

--Vikas

[swick]

Thanks a lot. I just wanted to make sure that this will work with all linux x86 platforms.

[swick]

any news here?

[vikasnkumar]

sorry was out of town. have not had time to fix the bugs in Ubuntu 12.10.

On 03/11/2013 08:45 AM, Sebastian Wick wrote:

any news here?

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-14711233.

[eXeC64]

I'm having the same issue with Mint 13, which is derived from 12.10 iirc.

I've noticed that I'm able to inject into a process within the same tree though, for example: urxvt -> zsh -> hotpatcher I can inject into the zsh instance that hotpatcher is run from, but not another zsh instance, and only when I use sudo.

Have you made any progress @vikasnkumar?

[vikasnkumar]

Sorry have not had time to make any serious progress. Linux 3.2 has become more secure not allowing injection as nicely as 2.6. Hence you need the sudo and hence it is more tough to inject into a non-tree process. I can fix it by using some custom reverse engineering but it will get patched in the kernel and I will keep chasing newer exploits. That I am not willing to do. I will try to find a more workable solution. Could you let me know what you are using hotpatch for ? You can email me privately. I can maybe suggest another route to solve your specific problem.

[swick]

FYI, I currently use GDB for injecting: https://gist.github.com/swick/5470356

[vikasnkumar]

I have tested on Ubuntu 13.04 and yes you have to use SUDO to inject if your /proc/sys/kernel/yama/ptrace_scope has the value 1 and no need of sudo if it has the value 0.

$ sudo su -c "echo 0 > /proc/sys/kernel/yama/ptrace_scope"

You can now inject a library into other processes without using sudo. This works on kernel 3.8 running on Ubuntu 13.04. Cannot say about 12.10 and why it never worked there. Maybe it is your system's security features.

[ntop001]

Same error happened on my computer.

• os : ubuntu 12.04 LTS 64-bit
• kernel : 3.2.0-52-generic

But error only happened when I injected into a specified PID .

such as : bash> ./hotpatcher -l ./libhotpatchtest.so -s mysym -v1 9057

9057 is a simple app with a while loop.

if I use bash> ./hotpatcher -l ./libhotpatchtest.so -s mysym -v1 $$ and inject into current shell pid, it works well.

It really confused me~

[vikasnkumar]

If the /proc/sys/kernel/yama/ptrace_scope is set to 1 on Ubuntu then you can inject only in the same process tree. Hence you are able to inject in your own bash shell and not on some other process started in a separate shell.

Here is the link for that: https://wiki.ubuntu.com/Security/Features#ptrace_scope

So either turn the security feature off or stick to injecting in same process tree.

This is only for Ubuntu and its derivatives. Debian and CentOS do not have this issue for example.

On 08/25/2013 01:30 AM, ntop001 wrote:

Same error happened on my computer.

• os : ubuntu 12.04 LTS 64-bit
• kernel : 3.2.0-52-generic

But error only happened when I injected into a specified PID .

such as : bash> ./hotpatcher -l ./libhotpatchtest.so -s mysym -v1 9057

9057 is a simple app with a while loop.

if I use |bash> ./hotpatcher -l ./libhotpatchtest.so -s mysym -v1 $$| and inject into current shell pid, it works well.

It really confused me~

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-23222259.

[ntop001]

Thanks , I'll test later.

[lengzijian]

Same error happened on my computer.

Linux 2.6.18-238.12.1.el5.centos.plus #1 SMP Wed Jun 1 11:12:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

error only happened when I injected into a specified PID on centos

info：

Dll was injected at (nil) Invocation of func() returned (nil)

[vikasnkumar]

Can you be more specific as to what you launched and what exactly was the error ? Saying "same error" will not help.

Thanks

On 10/24/2013 02:23 AM, lengzijian wrote:

Same error happened on my computer.

Linux 2.6.18-238.12.1.el5.centos.plus #1 https://github.com/vikasnkumar/hotpatch/issues/1 SMP Wed Jun 1 11:12:25 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

error only happened when I injected into a specified PID on centos

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-26969216.

[lengzijian]

if I inject into current shell pid, it works well. you say that CentOS do not have this issue . error happended when i injected into others pid on centos

[vikasnkumar]

CentOS 6.2 is what I had tested on. I have not tested on other versions. What kernel are you running ? What version of CentOS are you running ? Are you running SELinux ? There might be other security things that might be in place such as ptrace limitations.

All these things need to be looked at.

On 10/28/2013 12:40 AM, lengzijian wrote:

if I inject into current shell pid, it works well. you say that CentOS do not have this issue . error happended when i injected into others pid on centos

— Reply to this email directly or view it on GitHub https://github.com/vikasnkumar/hotpatch/issues/2#issuecomment-27189156.