search

apptainer / singularity

Singularity has been renamed to Apptainer as part of us moving the project to the Linux Foundation. This repo has been persisted as a snapshot right before the changes.
https://github.com/apptainer/apptainer
Other
2.53k stars 424 forks source link

curl and dns #16

Closed kforner closed 8 years ago

kforner commented 8 years ago

Trying to investigate dns related issues, I made a curl sapp.

I can not pinpoint what's needed to make it work, but here's my findings:

Name: curl
Exec: /usr/bin/curl

does not work: ./curl.sapp google.fr curl: (6) Could not resolve host: google.fr

Name: curl
Exec: /usr/bin/curl
%files:
/lib/x86_64-linux-gnu/

this works !

But from here, I used singularity strace to get the exact list of used libs, I pasted them in the %files section, but never managed to get it to work:

singularity strace 2>curl.err curl google.fr parsing it gives me the following list of files:

/lib/i386-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libc.so.6
/lib/x86_64-linux-gnu/libcom_err.so.2
/lib/x86_64-linux-gnu/libcrypt.so.1
/lib/x86_64-linux-gnu/libcrypto.so.1.0.0
/lib/x86_64-linux-gnu/libdl.so.2
/lib/x86_64-linux-gnu/libgcrypt.so.11
/lib/x86_64-linux-gnu/libgpg-error.so.0
/lib/x86_64-linux-gnu/libkeyutils.so.1
/lib/x86_64-linux-gnu/libpthread.so.0
/lib/x86_64-linux-gnu/libresolv.so.2
/lib/x86_64-linux-gnu/libssl.so.1.0.0
/lib/x86_64-linux-gnu/libz.so.1
/usr/lib/x86_64-linux-gnu/libasn1.so.8
/usr/lib/x86_64-linux-gnu/libcurl.so.4
/usr/lib/x86_64-linux-gnu/libffi.so.6
/usr/lib/x86_64-linux-gnu/libgnutls.so.26
/usr/lib/x86_64-linux-gnu/libgssapi.so.3
/usr/lib/x86_64-linux-gnu/libgssapi_krb5.so.2
/usr/lib/x86_64-linux-gnu/libhcrypto.so.4
/usr/lib/x86_64-linux-gnu/libheimbase.so.1
/usr/lib/x86_64-linux-gnu/libheimntlm.so.0
/usr/lib/x86_64-linux-gnu/libhx509.so.5
/usr/lib/x86_64-linux-gnu/libidn.so.11
/usr/lib/x86_64-linux-gnu/libk5crypto.so.3
/usr/lib/x86_64-linux-gnu/libkrb5.so.26
/usr/lib/x86_64-linux-gnu/libkrb5.so.3
/usr/lib/x86_64-linux-gnu/libkrb5support.so.0
/usr/lib/x86_64-linux-gnu/liblber-2.4.so.2
/usr/lib/x86_64-linux-gnu/libldap_r-2.4.so.2
/usr/lib/x86_64-linux-gnu/libp11-kit.so.0
/usr/lib/x86_64-linux-gnu/libroken.so.18
/usr/lib/x86_64-linux-gnu/librtmp.so.0
/usr/lib/x86_64-linux-gnu/libsasl2.so.2
/usr/lib/x86_64-linux-gnu/libsqlite3.so.0
/usr/lib/x86_64-linux-gnu/libtasn1.so.6
/usr/lib/x86_64-linux-gnu/libwind.so.0

What could be the problem ?

Thanks.

truatpasteurdotfr commented 8 years ago

reducing the scope: the minimal ping.sspec shows the same issue (CentOS-6 x86_64)

[tru@visu1 singularity]$ cat ping.sspec 
Name: ping
Exec: /bin/ping
[tru@visu1 singularity]$ singularity build ping.sspec 
...
[tru@visu1 singularity]$ ./ping.sapp -c 1 google.com
ping: unknown host google.com
[tru@visu1 singularity]$ ping -c 1 google.com
PING google.com (172.217.19.142) 56(84) bytes of data.
64 bytes from par03s12-in-f14.1e100.net (172.217.19.142): icmp_seq=1 ttl=53 time=2.53 ms

--- google.com ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 4ms
rtt min/avg/max/mdev = 2.533/2.533/2.533/0.000 ms
gmkurtzer commented 8 years ago

So, I'm able to build a curl sapp and run it without issue:

`[gmk@centos7-x64 demo]$ cat example.sspec Name: curl Exec: /bin/curl [gmk@centos7-x64 demo]$ singularity --quiet build example.sspec Running test... Hello from within the container... (no test code defined) WROTE: curl.sapp [gmk@centos7-x64 demo]$ ./curl.sapp google.com

301 Moved

301 Moved

The document has moved here.

[gmk@centos7-x64 demo]$ `

Ping is an interesting example, because technically to send an ICMP you need to special system privileges. This is what I'm getting on Centos 7 x64:

$ ./ping.sapp -c 1 google.com ping: icmp open socket: Operation not permitted [gmk@centos7-x64 demo]$

So, I'm guessing something is different on my system because it is resolving properly. DNS resolution usually happens by libc dlopen'ing name resolution libraries. Do this please:

$ singularity specgen curl google.com

and then paste the resulting curl.sspec (if both of you can do that, it would be great). Singularity always needs to included these libraries into a container.

Thanks!

truatpasteurdotfr commented 8 years ago 
[tru@visu1 singularity]$ \rm -rf ~/.singularity-cache/
[tru@visu1 singularity]$ singularity list
CONTAINER NAME       UUID                                          SIZE SUMMARY
[tru@visu1 singularity]$  singularity specgen curl google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.fr/?gfe_rd=cr&amp;ei=HiT8VuLZOcHI8gfj0K-ABQ">here</A>.
</BODY></HTML>
WROTE: curl.sspec
[tru@visu1 singularity]$ singularity build curl.sspec 
...
WROTE: curl.sapp
Cleaning up temporary files...
[tru@visu1 singularity]$ ./curl.sapp google.com
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<TITLE>302 Moved</TITLE></HEAD><BODY>
<H1>302 Moved</H1>
The document has moved
<A HREF="http://www.google.fr/?gfe_rd=cr&amp;ei=XiT8VqWmOcHI8gfj0K-ABQ">here</A>.
</BODY></HTML>

so it works with the "specgen"erated file :)

truatpasteurdotfr commented 8 years ago

attaching the generated file on CentOS-6 curl.sspec.txt

gmkurtzer commented 8 years ago

I betcha it is the libnss3 and libnssutil3. I don't think I'm adding those automatically... That should catch it for Centos6. ;)

Thanks Tru!

gmkurtzer commented 8 years ago

If you retest from trunk, it should now grab those two libraries when building a SAPP. Hopefully that will fix it for both cases! Karl, can you retest current master?

Thanks!

truatpasteurdotfr commented 8 years ago

:( 

[tru@visu1 singularity]$ singularity   build curl.sspec 2>&1 |grep nss
Installing file: /lib64/libnss_files.so.2
Installing file: /lib64/libnss_dns.so.2
Installing file: /lib64/libnss3.so
Installing file: /usr/lib64/libnssutil3.so
Installing file: /lib64/libnssutil3.so
Installing file: /usr/lib64/libnss3.so

but

[tru@visu1 singularity]$ ./curl.sapp google.com
curl: (6) Couldn't resolve host 'google.com'
gmkurtzer commented 8 years ago

Hi Tru,

Can you run another test for me? Please capture all of the out from this command in /tmp/debug, and send me that file please:

$ singularity strace curl google.com >/tmp/debug 2>&1

Thanks!

kforner commented 8 years ago

my curl.sspec:

curl.sspec.txt

I tried the ping sapp: same problem:

./ping.sapp google.fr
ping: unknown host google.fr

the curl strace: curl.strace.txt

gmkurtzer commented 8 years ago

Hi Karl, I'm no at computer yet (so I'm not sure what the attachments are) but can you also do the strace against curl? Ping is an odditity because it always requires additional permission to run. Up until just recently ping was setuid root. Now it generally uses Linux capabilities to achieve similar results which is why I'm not counting on ping to ever work properly in a singularity container.

Probably the smarter thing would be for singularity to check on files with non standard permissions and error if one is found.

Thanks again!

kforner commented 8 years ago

can you also do the strace against curl? look at the previous post !

gmkurtzer commented 8 years ago

Oh, Tru's response? I wasn't making the assumption that you are running on the same host OS. If you are that will indeed work fine and will use what he provided.

Will investigate more soon as I'm at a real keyboard.

Thanks!

gmkurtzer commented 8 years ago

Heya Karl,

Sorry for the confusion about not seeing your attachments. Looking at them now, and I'm noticing that libnss_myhostname.so.* was not installed into your container. I am debugging why that might have happened as that file should always be installed (as of several days ago!).

Thanks!

gmkurtzer commented 8 years ago

ok, I think Karl's issue is now fixed now in master.

The problem was that I made an assumption that the libraries I am statically including (e.g. libnssmyhostname.so.) exist right off of the /lib_ directories. But on Debian like systems they are actually 2 levels deep. Please test. :)

Tru, can you also test against the git master?

Thanks!

kforner commented 8 years ago

Still does not work for me. I do not know if it is relevant, but for nslookup.spec, I had to include /usr/lib/x86_64-linux-gnu/openssl-1.0.0

gmkurtzer commented 8 years ago

Can you try that for curl too, and then if it works send me an strace of it?

Can you remind me, when you used specgen to create the spec for curl, did it work? If so can you show me that spec?

Thanks!

kforner commented 8 years ago

Can you try that for curl too, It did not work for curl.

Can you remind me, when you used specgen to create the spec for curl, did it work?

nope

same spec as above

gmkurtzer commented 8 years ago

What distribution are you using?

Can you send me your curl SAPP? (Either attach to the ticket or email direct to me please).

Thanks!

kforner commented 8 years ago

ubuntu 14.04

Can you send me your curl SAPP? (Either attach to the ticket or email direct to me please). which curl sapp ? and which email ? the berkeley one ?

gmkurtzer commented 8 years ago

I have it running on another Debian/Ubuntu based distribution (Lubuntu) without a problem.

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but I am creating a generic /etc/nsswitch.conf. I wonder if that is the problem... Might be worth copying your hosts's /etc/nsswitch.conf into ~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for testing.

kforner commented 8 years ago

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but I am creating a generic >/etc/nsswitch.conf. I wonder if that is the problem... Might be worth copying your hosts's >/etc/nsswitch.conf into ~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for testing.

I tried copying both the nsswitch.conf then the resolv.conf without success.

But remember, if I add /lib/x86_64-linux-gnu/ in %files it works. Could it be related to pam, sssd, etc... ?

kforner commented 8 years ago

On Fri, Apr 1, 2016 at 4:50 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

I have it running on another Debian/Ubuntu based distribution (Lubuntu) without a problem.

At the moment, I am using the host's /etc/resolv.conf and /etc/hosts, but I am creating a generic /etc/nsswitch.conf. I wonder if that is the problem... Might be worth copying your hosts's /etc/nsswitch.conf into ~/.singulairty-cache/containers/[container UUID]/c/etc/nsswitch.conf for testing.

in any case, I think that files explicitly specified in the %files should override your templates.

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/gmkurtzer/singularity/issues/16#issuecomment-204425317

gmkurtzer commented 8 years ago

But remember, if I add /lib/x86_64-linux-gnu/ in %files it works. Could it be related to pam, sssd, etc... ?

Yes, we are missing some library that is getting dlopen'ed probably by libc itself would be my guess. But why we aren't able to detect it is worrysome. I went over your specgen output and your strace output of library open(), and there were no differences.

I didn't get the SAPP file though, did you send it? Can you also send me the SAPP file where it worked when you included all of /lib/x86_64-linux-gnu/? It will be bigger, so maybe a dropbox or other URL link if it doesn't fit in the ticket?

Thanks

gmkurtzer commented 8 years ago

in any case, I think that files explicitly specified in the %files should override your templates.

Oh, this is easy enough to do, but.... It does inherently break portability and possibly expose hosts (targets) to people who are distributing the SAPP file. Let me think about that.

kforner commented 8 years ago

I didn't get the SAPP file though, did you send it? Can you also send me the SAPP file where it worked when you included all of /lib/x86_64-linux-gnu/?

Here they are:

Oh, this is easy enough to do, but.... It does inherently break portability and possibly expose hosts >(targets) to people who are distributing the SAPP file. Let me think about that.

My point is that if you explicitly add them to %files, then you (should) know what you're doing. You could maintain a list of sensitive files, and emit warnings if they are in %files...

gmkurtzer commented 8 years ago

I'm not sure if this is related or not, but both are picking up a library in a non-standard location:

/home/karl/bin/libnss_dns.so.2

Can you confirm that /bin/ldd is finding that, and that it is necessary?

Thanks!

kforner commented 8 years ago

there is no /home/karl/bin/libnss_dns.so.2

but there is a /home/karl/bin/libstderred.so which is loaded via LD_PRELOAD=/home/karl/bin/libstderred.so Could it be related ?

On Mon, Apr 4, 2016 at 4:46 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

I'm not sure if this is related or not, but both are picking up a library in a non-standard location:

/home/karl/bin/libnss_dns.so.2

Can you confirm that /bin/ldd is finding that, and that it is necessary?

Thanks!

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/gmkurtzer/singularity/issues/16#issuecomment-205328751

gmkurtzer commented 8 years ago

No, I don't think that is related. Is your LD_LIBRARY_PATH set?

I found another thing very interesting:

The container that works has multiple copies of libnss_files and libnss_dns and they aren't the same.

$ md5sum lib64/libnss_files.so.2 lib/x86_64-linux-gnu/libnss_files.so.2 5b3b1eb6d2be03550623921a55c1b41f lib64/libnss_files.so.2 67f25cd8773539eda122b6b44c117048 lib/x86_64-linux-gnu/libnss_files.so.2 $ md5sum lib64/libnss_dns.so.2 lib/x86_64-linux-gnu/libnss_dns.so.2 36286bd509a1cf64ad086e3ca33255df lib64/libnss_dns.so.2 4c611c519cbd06d59ec35f8d210befdb lib/x86_64-linux-gnu/libnss_dns.so.2

To that end I just changed some of the code in master, hoping to fix that. Can you try again?

Thanks!

kforner commented 8 years ago

It works !!!

On Mon, Apr 4, 2016 at 4:57 PM, Gregory M. Kurtzer <notifications@github.com

wrote:

No, I don't think that is related. Is your LD_LIBRARY_PATH set?

I found another thing very interesting:

The container that works has multiple copies of libnss_files and libnss_dns and they aren't the same.

$ md5sum lib64/libnss_files.so.2 lib/x86_64-linux-gnu/libnss_files.so.2 5b3b1eb6d2be03550623921a55c1b41f lib64/libnss_files.so.2 67f25cd8773539eda122b6b44c117048 lib/x86_64-linux-gnu/libnss_files.so.2 $ md5sum lib64/libnss_dns.so.2 lib/x86_64-linux-gnu/libnss_dns.so.2 36286bd509a1cf64ad086e3ca33255df lib64/libnss_dns.so.2 4c611c519cbd06d59ec35f8d210befdb lib/x86_64-linux-gnu/libnss_dns.so.2

To that end I just changed some of the code in master, hoping to fix that. Can you try again?

Thanks!

— You are receiving this because you authored the thread. Reply to this email directly or view it on GitHub https://github.com/gmkurtzer/singularity/issues/16#issuecomment-205335807

gmkurtzer commented 8 years ago

Fantastic! Thank you so much for your patience while I wrap my head around it. I also wrote a note in the TODO to think about how to handle the issue with sensitive files in %files and how to include or overwrite them.

BTW, It was very cool getting your SAPPs and being able to replicate your exact problem. lol

kforner commented 8 years ago

BTW, It was very cool getting your SAPPs and being able to replicate your exact problem. yes we'll know for the next time ;)

thanks for the fix.