search

3ofcoins / jetpack

**PROTOTYPE** FreeBSD Jail/ZFS based implementation of the Application Container Specification
Other
529 stars 38 forks source link

interestingly, following your documentation... failures! #3

Closed ghost closed 9 years ago

ghost commented 9 years ago

jetpack image freebsd-base/release build -cp=/root/jetpack/share/jetpack.image.mk /usr/bin/make .jetpack.build. sed -i '' 's|^Components.*|Components world/base|' /etc/freebsd-update.conf install -v -m 0644 rc.conf /etc/rc.conf install: rc.conf -> /etc/rc.conf install -v -m 0600 entropy /entropy install: entropy -> /entropy patch /usr/sbin/freebsd-update < freebsd-update.patch Hmm... Looks like a unified diff to me...

The text leading up to this was:

|--- /usr/sbin/freebsd-update 2015-02-08 22:15:58.178818000 +0100

|+++ freebsd-update 2015-02-09 13:45:42.202917000 +0100

Patching file /usr/sbin/freebsd-update using Plan A... Hunk #1 succeeded at 610 (offset -8 lines). done env PAGER=cat freebsd-update -s update6.freebsd.org fetch install Looking up update6.freebsd.org mirrors... none found. Fetching public key from update6.freebsd.org... failed. No mirrors remaining, giving up. *\ Error code 1

Stop. make: stopped in /.jetpack.build. run.Command[/root/jetpack/bin/stage2 -jid 1 -chroot /0 -user 0 -group 0 -name jetpack/build -cwd /.jetpack.build. /usr/bin/make .jetpack.build.]: exit status 1 /root/jetpack/jetpack/image.go:331: *\ Error code 1

Stop. make: stopped in /root/jetpack/images/freebsd-base ERROR: run.Command[make -C /root/jetpack/images/freebsd-base]: exit status 1 cannot unmount '/var/jetpack/test.462087242/pods/0a5966da-e84c-4cd5-a14b-2c78c0a6fdd9/rootfs/0': Device busy ERROR: run.Command[/sbin/zfs destroy -r zroot/jetpack/test.462087242]: exit status 1 run.Command[/root/jetpack/bin/test.integration dataset=zroot/jetpack]: exit status 2 root@xoa:~/jetpack # ping update6.freebsd.org PING update6.freebsd.org (198.148.79.66): 56 data bytes 64 bytes from 198.148.79.66: icmp_seq=0 ttl=49 time=270.566 ms 64 bytes from 198.148.79.66: icmp_seq=1 ttl=49 time=270.299 ms ^C --- update6.freebsd.org ping statistics --- 2 packets transmitted, 2 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 270.299/270.433/270.566/0.133 ms root@xoa:~/jetpack # freebsd-update -s update6.freebsd.org fetch install Looking up update6.freebsd.org mirrors... none found. Fetching public key from update6.freebsd.org... failed. No mirrors remaining, giving up.

mpasternacki commented 9 years ago

Hi,

Looks like inside of the pod doesn't have access to the network, or functional DNS.

Did you follow the pf setup to NAT traffic from within the pod/jail to the outside world?

Does your /etc/resolv.conf use DNS servers that may not be reachable from within the container? The DNS setup for the pod jails currently consists of copying host's /etc/resolv.conf.

I see that you have successfully built the freebsd-base/release image (the unpatched 10.1 base system). To help diagnose the problem, you can start a pod with interactive console of that image, and look for the reason. I'll paste some samples, assuming you use the default settings (in particular, the default IP range where host uses interface lo1 configured as 127.23.0.1. I'll set allow.raw_sockets=true in pod's jail.conf to be able to ping from the container:

$ jetpack pod create -run -destroy -a=jetpack/jail.conf/allow.raw-sockets=true freebsd-base/release

This should drop you into a root console. First, let's try to ping the host. Next, we'll try to ping the IP for example.com:

root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 172.23.0.1
PING 172.23.0.1 (172.23.0.1): 56 data bytes
64 bytes from 172.23.0.1: icmp_seq=0 ttl=64 time=0.035 ms
64 bytes from 172.23.0.1: icmp_seq=1 ttl=64 time=0.123 ms
64 bytes from 172.23.0.1: icmp_seq=2 ttl=64 time=0.122 ms

--- 172.23.0.1 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 0.035/0.093/0.123/0.041 ms
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 93.184.216.34
PING 93.184.216.34 (93.184.216.34): 56 data bytes
64 bytes from 93.184.216.34: icmp_seq=0 ttl=52 time=109.691 ms
64 bytes from 93.184.216.34: icmp_seq=1 ttl=52 time=112.540 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=52 time=112.311 ms

--- 93.184.216.34 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 109.691/111.514/112.540/1.292 ms

If first one fails, there's a general problem with networking. If the second one fails, there's a routing or NAT issue. If you configured PF correctly, please check sysctl net.inet.ip.forwarding on the host. If it's set to 0, try whether setting it to 1 helps. It might be necessary; if it is, I may have missed it from README, because I have that enabled for other reasons. Please let me know if that is the issue.

If IP works correctly, please check that /etc/resolv.conf looks reasonable, and that hostnames resolve correctly. Ping your name servers. Use the host command to run DNS queries. Use fetch command to check TCP:

root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # cat /etc/resolv.conf
…
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 8.8.8.8
PING 8.8.8.8 (8.8.8.8): 56 data bytes
64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=16.912 ms
64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=18.406 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=22.436 ms

--- 8.8.8.8 ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 16.912/19.251/22.436/2.333 ms
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # host example.com 127.0.0.1
Using domain server:
Name: 127.0.0.1
Address: 127.0.0.1#53
Aliases: 

example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # host example.com 8.8.8.8
Using domain server:
Name: 8.8.8.8
Address: 8.8.8.8#53
Aliases: 

example.com has address 93.184.216.34
example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 example.com
PING example.com (93.184.216.34): 56 data bytes
64 bytes from 93.184.216.34: icmp_seq=0 ttl=52 time=113.945 ms
64 bytes from 93.184.216.34: icmp_seq=1 ttl=52 time=111.940 ms
64 bytes from 93.184.216.34: icmp_seq=2 ttl=52 time=109.855 ms

--- example.com ping statistics ---
3 packets transmitted, 3 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 109.855/111.913/113.945/1.670 ms
root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # fetch http://example.com
example.com                                   100% of 1270  B 2932 kBps 00m00s

Please let me know what you found out. If you need to set custom DNS servers for the pods (e.g. to avoid IPs unreachable from jails), this should be reasonably easy to implement. If all of the above works, and the make in images/freebsd-base still fails, let me know; I will try to figure out what might be the cause.

Thanks for trying out Jetpack, BTW! I hope you'll be able to get it up and running soon.

ghost commented 9 years ago

Thanks for that, though now im curious if there would be a way to utilize vimage for the network portion, i would prefer to have hosts talk on xn0 which is the wan side with local dhcp addresses, or static assigned on the same network, without utilizing pf

In the interim I have added pf and a lo1 I have gotten further, thanks for the heads up, i reread the document

the line ifconfig lo1 create inet 172.23.0.1/16 kind of through me until i read the pf section :)

now, it seems it fails later with open /var/jetpack/test.240127459/pods/44104a4d-6cc8-451f-8fa1-bb97808341ad/rootfs/0/etc/resolv.conf: no such file or directory

which is odd because the main system does have a proper /etc/resolv.conf bin/jetpack test cannot open 'zroot/jetpack/test.240127459': dataset does not exist JetPack 0.0.1 (v0.0.1-82-g5330a94), compiled on 2015-04-29T00:23:03Z (development) ZFS Dataset zroot/jetpack/test.240127459 Mountpoint /var/jetpack/test.240127459 Configuration: root.zfs zroot/jetpack/test.240127459 root.zfs.mountpoint /var/jetpack/test.240127459 images.ami.store no images.ami.compression xz images.zfs.atime off images.zfs.compress lz4 jail.interface lo1 jail.namePrefix jetpack/ debug off sha256 -c 2b028a894d25711ad496762622a52d74b1e32ee04693ad1cf056e3ddcdc23975 base.txz SHA256 (base.txz) = 2b028a894d25711ad496762622a52d74b1e32ee04693ad1cf056e3ddcdc23975 jetpack image import base.txz manifest.json

The following files will be updated as part of updating to 10.1-RELEASE-p9: /bin/freebsd-version /boot/boot1.efi /boot/boot1.efifat ____SNIP____ Installing updates... done. rm -rf /var/db/freebsd-update/* ./manifest.json.sh > manifest.json Pod dying since 175.745832ms, waiting... Pod dying since 2.678158792s, waiting... Pod dying since 5.184116285s, waiting... Pod dying since 7.690087916s, waiting... Pod dying since 10.222114569s, waiting... Pod dying since 12.760005399s, waiting... Pod dying since 15.288094987s, waiting... Pod dying since 17.803733529s, waiting... Pod dying since 20.311760549s, waiting... Pod dying since 22.841798152s, waiting... Pod dying since 25.357716796s, waiting... Pod dying since 27.871071136s, waiting... Pod dying since 30.404391419s, waiting... Pod dying since 32.939016854s, waiting... Pod dying since 35.470003973s, waiting... Image 25d58c5f-18e7-4e75-aeb7-8f6d3cacc375 Hash sha512-4fe0a5dc4e2eaba16ae40a6c280130e86b1aa08230e2d7c9afa47e512ec0c4f5695c5bf2337b12b09c80dc18e47d18fc5ebb6308b527bb8207edb2434c4d2338

Origin 72f2b72f-1467-4112-8f9e-45d58f4f1d58 Timestamp 2015-04-30 14:51:56.433940171 +1000 AEST Manifest freebsd-base Labels version 10.1.9 os freebsd arch amd64 Annotations timestamp 2015-04-30T14:51:56.433709088+10:00 jetpack image freebsd-base build -cp=/root/jetpack/share/jetpack.image.mk /usr/bin/make .jetpack.build. open /var/jetpack/test.240127459/pods/44104a4d-6cc8-451f-8fa1-bb97808341ad/rootfs/0/etc/resolv.conf: no such file or directory /root/jetpack/jetpack/pod.go:281: /root/jetpack/jetpack/pod.go:500: /root/jetpack/jetpack/pod.go:501: /root/jetpack/jetpack/image.go:331: *\ Error code 1

Stop. make: stopped in /root/jetpack/images/example.showenv ERROR: run.Command[make -C /root/jetpack/images/example.showenv]: exit status 1 run.Command[/root/jetpack/bin/test.integration dataset=zroot/jetpack]: exit status 2

On Thu, Apr 30, 2015 at 11:19 PM, Maciej Pasternacki < notifications@github.com> wrote:

Hi,

Looks like inside of the pod doesn't have access to the network, or functional DNS.

Did you follow the pf setup to NAT traffic from within the pod/jail to the outside world?

Does your /etc/resolv.conf use DNS servers that may not be reachable from within the container? The DNS setup for the pod jails currently consists of copying host's /etc/resolv.conf.

I see that you have successfully built the freebsd-base/release image (the unpatched 10.1 base system). To help diagnose the problem, you can start a pod with interactive console of that image, and look for the reason. I'll paste some samples, assuming you use the default settings (in particular, the default IP range where host uses interface lo1 configured as 127.23.0.1. I'll set allow.raw_sockets=true in pod's jail.conf to be able to ping from the container:

$ jetpack pod create -run -destroy -a=jetpack/jail.conf/allow.raw-sockets=true freebsd-base/release

This should drop you into a root console. First, let's try to ping the host. Next, we'll try to ping the IP for example.com:

root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 172.23.0.1 PING 172.23.0.1 (172.23.0.1): 56 data bytes 64 bytes from 172.23.0.1: icmp_seq=0 ttl=64 time=0.035 ms 64 bytes from 172.23.0.1: icmp_seq=1 ttl=64 time=0.123 ms 64 bytes from 172.23.0.1: icmp_seq=2 ttl=64 time=0.122 ms

--- 172.23.0.1 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 0.035/0.093/0.123/0.041 ms root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 93.184.216.34 PING 93.184.216.34 (93.184.216.34): 56 data bytes 64 bytes from 93.184.216.34: icmp_seq=0 ttl=52 time=109.691 ms 64 bytes from 93.184.216.34: icmp_seq=1 ttl=52 time=112.540 ms 64 bytes from 93.184.216.34: icmp_seq=2 ttl=52 time=112.311 ms

--- 93.184.216.34 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 109.691/111.514/112.540/1.292 ms

If first one fails, there's a general problem with networking. If the second one fails, there's a routing or NAT issue. If you configured PF correctly, please check sysctl net.inet.ip.forwarding on the host. If it's set to 0, try whether setting it to 1 helps. It might be necessary; if it is, I may have missed it from README, because I have that enabled for other reasons. Please let me know if that is the issue.

If IP works correctly, please check that /etc/resolv.conf looks reasonable, and that hostnames resolve correctly. Ping your name servers. Use the host command to run DNS queries. Use fetch command to check TCP:

root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # cat /etc/resolv.conf … root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 8.8.8.8 PING 8.8.8.8 (8.8.8.8): 56 data bytes 64 bytes from 8.8.8.8: icmp_seq=0 ttl=54 time=16.912 ms 64 bytes from 8.8.8.8: icmp_seq=1 ttl=54 time=18.406 ms 64 bytes from 8.8.8.8: icmp_seq=2 ttl=54 time=22.436 ms

--- 8.8.8.8 ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 16.912/19.251/22.436/2.333 ms root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # host example.com 127.0.0.1 Using domain server: Name: 127.0.0.1 Address: 127.0.0.1#53 Aliases: example.com has address 93.184.216.34example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946 root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # host example.com 8.8.8.8 Using domain server: Name: 8.8.8.8 Address: 8.8.8.8#53 Aliases: example.com has address 93.184.216.34example.com has IPv6 address 2606:2800:220:1:248:1893:25c8:1946 root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # ping -c 3 example.com PING example.com (93.184.216.34): 56 data bytes 64 bytes from 93.184.216.34: icmp_seq=0 ttl=52 time=113.945 ms 64 bytes from 93.184.216.34: icmp_seq=1 ttl=52 time=111.940 ms 64 bytes from 93.184.216.34: icmp_seq=2 ttl=52 time=109.855 ms

--- example.com ping statistics --- 3 packets transmitted, 3 packets received, 0.0% packet loss round-trip min/avg/max/stddev = 109.855/111.913/113.945/1.670 ms root@99c71b82-bdd7-4b63-82b0-ebd4dcebc2e3:~ # fetch http://example.comexample.com 100% of 1270 B 2932 kBps 00m00s

Please let me know what you found out. If you need to set custom DNS servers for the pods (e.g. to avoid IPs unreachable from jails), this should be reasonably easy to implement. If all of the above works, and the make in images/freebsd-base still fails, let me know; I will try to figure out what might be the cause.

Thanks for trying out Jetpack, BTW! I hope you'll be able to get it up and running soon.

— Reply to this email directly or view it on GitHub https://github.com/3ofcoins/jetpack/issues/3#issuecomment-97777930.

mpasternacki commented 9 years ago

The "no such file or directory" occured when opening pod's /etc/resolv.conf for writing. I found the bug there: during build, wrong dataset has been saved as rootfs (missed a spot when refactoring for multi-app pods). Please try now, removing images you've built and rebuilding them after the update.

mpasternacki commented 9 years ago

Ah, regarding VIMAGE: this is on the roadmap, but I want to have Jetpack run on GENERIC kernel as well, so it's going to be optional (if I manage to get VIMAGE to run without panicking; on 10.1 I experience panic on boot with VIMAGE enabled; -CURRENT boots, but I wasn't courageous enough to try it yet).

DHCP would move a lot of responsibility and trust into the containers' inside (having to run DHCP client; trusting that DHCP client behaves properly), and would have to be explicitly supported by the containers. OTOH, this can be a stage1 responsibility now (or maybe stage1.5? Before the apps start, outside any of the pod's apps, but inside the jail), so it might work. Still, the runtime (metadata service in particular, probably any port forwarding mechanisms) would need to know pods' IP addresses. I think that support for routable IPs not chosen by the runtime, not hardcoded into pod manifest, should be post-MVP.

ghost commented 9 years ago

This seems to have resolved it, Thanks, and VIMAGE appears stable on CURRENT

Pod dying since 30.284807085s, waiting... Pod dying since 32.803964719s, waiting... Pod dying since 35.328886786s, waiting... Image 100e8704-a5f1-4371-85d1-c9f8518c1159 Hash sha512-41415996766b2ec79ff4ba895b28ab626e2e5f53bab158fc1d10ff5431ac6fd104ed77488e1e701339b903121c690bccee48f5966c5530a9ba2e283140734d2e

Origin 82bcd949-c237-4d1a-a761-7b3f53c6b0ee Timestamp 2015-04-30 21:38:01.787954234 +1000 AEST Manifest freebsd-base Labels version 10.1.9 os freebsd arch amd64 Annotations timestamp 2015-04-30T21:38:01.787693871+10:00 jetpack image freebsd-base build -cp=/root/jetpack/share/jetpack.image.mk /usr/bin/make .jetpack.build. install -d /usr/local/bin install -d -o root -g guest -m 0775 /opt/data install -m 0755 showenv.sh /usr/local/bin/showenv Image e0b1752a-4fc7-48cb-811d-2ad7d966fd13 Hash sha512-40109cade842e10b99170af9287e86b2ee4eda23469fce31bcf970f89b09133283d85f4dcc248f0bfc52a4922aed59493c8b3496713bac57942c30accb17cb01

Origin 100e8704-a5f1-4371-85d1-c9f8518c1159 Timestamp 2015-04-30 21:38:05.728605614 +1000 AEST Manifest example/showenv Labels version 1.1.1 os freebsd arch amd64 App Exec "/usr/local/bin/showenv" "foo" "bar" "baz" "quux" User games Group guest Environment: FOO bar BAZ quux Mount Points data /opt/data [types.Isolators types.Isolators(nil)] Annotations timestamp 2015-04-30T21:38:05.728308645+10:00 PASS

On Fri, May 1, 2015 at 2:31 AM, Maciej Pasternacki <notifications@github.com

wrote:

Ah, regarding VIMAGE: this is on the roadmap, but I want to have Jetpack run on GENERIC kernel as well, so it's going to be optional (if I manage to get VIMAGE to run without panicking; on 10.1 I experience panic on boot with VIMAGE enabled; -CURRENT boots, but I wasn't courageous enough to try it yet).

DHCP would move a lot of responsibility and trust into the containers' inside (having to run DHCP client; trusting that DHCP client behaves properly), and would have to be explicitly supported by the containers. OTOH, this can be a stage1 responsibility now (or maybe stage1.5? Before the apps start, outside any of the pod's apps, but inside the jail), so it might work. Still, the runtime (metadata service in particular, probably any port forwarding mechanisms) would need to know pods' IP addresses. I think that support for routable IPs not chosen by the runtime, not hardcoded into pod manifest, should be post-MVP.

— Reply to this email directly or view it on GitHub https://github.com/3ofcoins/jetpack/issues/3#issuecomment-97870140.