Unable to run base image on EC2 instance

3ofcoins / jetpack

**PROTOTYPE** FreeBSD Jail/ZFS based implementation of the Application Container Specification

Other

529 stars 38 forks source link

Unable to run base image on EC2 instance #11

Closed JayBusch closed 8 years ago

JayBusch commented 8 years ago

I have attempted to compile and run jetpack on a FreeBSD 10.2-RELEASE instance on Amazon/EC2.

Any help that can be provided in identifying what I may be doing wrong or if there is a bug of some sort would be GREATLY appreciated.

The instance was built from the ami-bbc43aff AMI provided by Colin Percival (http://www.daemonology.net/). All commands were executed as root (logging in as ec2-user then issuing "su").

I created a zpool named zroot of 10GB (from 5 EBS blocks of 2GB each).

I was able to install go and compile the source using "make", and install using "make install".

I added a user using "pw useradd _jetpack -d /var/jetpack -s /usr/sbin/nologin"

I added a network interface using "ifconfig lo1 create inet 172.23.0.1/16" and by adding the neccesary information to /etc/rc.conf.

I enabled PF and added the following to /etc/pf.conf "set skip on lo1 nat pass on $ext_if from lo1:network to any -> $ext_if".

I edited PF to define ext_if as lo0 (this is the public interface on the EC2 image).

I started PF.

I copied the sample jetpack.conf file to /usr/local/etc/jetpack.conf. I edited /usr/local/etc/jetpack.conf to include the mds.signing-key and mds.token-key with random hex keys generated using "openssl rand -hex 32".

I ran "jetpack init" and got the following output:

cannot open 'zroot/jetpack': dataset does not exist 2015-11-13T19:25:04Z jetpack Creating ZFS dataset zroot/jetpack [-p -omountpoint=/var/jetpack] 2015-11-13T19:25:04Z jetpack Creating ZFS dataset zroot/jetpack/images [-oatime=off -ocompress=lz4] 2015-11-13T19:25:04Z jetpack Creating ZFS dataset zroot/jetpack/pods []

I then ran "jetpack run 3ofcoins.net/freebsd.base" and got the following output:

3ofcoins.net/freebsd.base 2015-11-13T19:27:57Z import:b62fb668-bf77-4e17-a2f9-cae6d5e942b7 Starting import of 3ofcoins.net/freebsd.base tar: Error opening archive: Can't stat '' 2015-11-13T19:27:57Z image:b62fb668-bf77-4e17-a2f9-cae6d5e942b7 Destroying cannot open 'zroot/jetpack/images/b62fb668-bf77-4e17-a2f9-cae6d5e942b7': dataset does not exist panic: Not found

goroutine 1 [running]: lib/jetpack.(_Image).getRootfs(0xc820497440, 0xc8200dcb58) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/image.go:102 +0x16f lib/jetpack.(_Image).Destroy(0xc820497440, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/image.go:170 +0x975 lib/jetpack.(_Host).ImportImage.func1(0xc8204bab20, 0xc820497440) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:475 +0x2f lib/jetpack.(_Host).ImportImage(0xc8200fe480, 0xc820100560, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:501 +0x12a4 lib/jetpack.(_Host).fetchImage(0xc8200fe480, 0xc820100560, 0x19, 0xd5e5a0, 0x0, 0x0, 0x19, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:401 +0x2c8 lib/jetpack.(_Host).getImage(0xc8200fe480, 0x0, 0x0, 0x0, 0x0, 0xc820100560, 0x19, 0xd5e5a0, 0x0, 0x0, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:355 +0x220 lib/jetpack.(_Host).GetImage(0xc8200fe480, 0x0, 0x0, 0x0, 0x0, 0xc820100560, 0x19, 0xd5e5a0, 0x0, 0x0, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:304 +0xb7 lib/jetpack.(_Host).getRuntimeImage(0xc8200fe480, 0xc820100580, 0x0, 0x0, 0x0, 0x0, 0xd5e5a0, 0x0, 0x0, 0x40, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:291 +0xb0 lib/jetpack.(_Host).ReifyPodManifest(0xc8200fe480, 0xc8200fc000, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:206 +0x15b main.getPodManifest(0xc82000a200, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:259 +0x148 main.getOrPreparePod(0xc82000a200, 0x1, 0x1, 0xc8200108a0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:277 +0x148 main.cmdWrapPodPrepare0.func1(0xc82000a200, 0x1, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:186 +0x4e main.(_Command).Run(0xc8200e72c0, 0xc82000a200, 0x1, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:80 +0xda main.main() /usr/home/ec2-user/src/jetpack/src/bin/jetpack/main.go:37 +0x156

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]: os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1 /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 24 [IO wait]: net.runtime_pollWait(0x80150cc98, 0x72, 0xc82000c1e0) /usr/local/go/src/runtime/netpoll.go:157 +0x60 net.(_pollDesc).Wait(0xc820358300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(_pollDesc).WaitRead(0xc820358300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(_netFD).Read(0xc8203582a0, 0xc820480000, 0x2000, 0x2000, 0x0, 0x801507050, 0xc82000c1e0) /usr/local/go/src/net/fd_unix.go:232 +0x23a net.(_conn).Read(0xc820022040, 0xc820480000, 0x2000, 0x2000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:172 +0xe4 crypto/tls.(_block).readFromUntil(0xc8204c6a80, 0x80150d100, 0xc820022040, 0x5, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:455 +0xcc crypto/tls.(_Conn).readRecord(0xc820084000, 0x9b1517, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1 crypto/tls.(_Conn).Read(0xc820084000, 0xc8204cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:901 +0x167 net/http.noteEOFReader.Read(0x801517bb0, 0xc820084000, 0xc8203f63c8, 0xc8204cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1370 +0x67 net/http.(_noteEOFReader).Read(0xc8204c2200, 0xc8204cd000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

:126 +0xd0 bufio.(_Reader).fill(0xc8203d9380) /usr/local/go/src/bufio/bufio.go:97 +0x1e9 bufio.(_Reader).Peek(0xc8203d9380, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xcc net/http.(_persistConn).readLoop(0xc8203f6370) /usr/local/go/src/net/http/transport.go:876 +0xf7 created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:685 +0xc78 goroutine 25 [select]: net/http.(_persistConn).writeLoop(0xc8203f6370) /usr/local/go/src/net/http/transport.go:1009 +0x40c created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:686 +0xc9d

mpasternacki commented 8 years ago

Could you please try running the jetpack run line with debug = on in your jetpack.conf file? I'll try to reproduce the issue on my side as well.

mpasternacki commented 8 years ago

Ah, the diagnostics are a problem, but the issue seems to be a typo in README: the image name is 3ofcoins.net/freebsd-base (with a dash, not a dot). To get a console, please do jetpack run -t 3ofcoins.net/freebsd-base, otherwise the app won't have a terminal or stdin attached.

JayBusch commented 8 years ago

I corrected the "." to a "-" and turned on debugging. Here are the results of "jetpack run -t 3ofcoins.net/freebsd-base":

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" 3ofcoins.net/freebsd-base 2015-11-13T19:49:49Z import:fdb9291c-26b1-4eff-a55f-ab71f4021c1b Starting import of 3ofcoins.net/freebsd-base [lib/jetpack/host.go:442] 2015-11-13T19:49:49Z import:fdb9291c-26b1-4eff-a55f-ab71f4021c1b DEBUG: No signature to check [lib/jetpack/host.go:468] 2015-11-13T19:49:49Z import:fdb9291c-26b1-4eff-a55f-ab71f4021c1b DEBUG: Loading manifest [lib/jetpack/host.go:498]
tar -xOqf - manifest |tar: Error opening archive: Can't stat '' "" 2015-11-13T19:49:49Z image:fdb9291c-26b1-4eff-a55f-ab71f4021c1b Destroying [lib/jetpack/image.go:169]
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/fdb9291c-26b1-4eff-a55f-ab71f4021c1b |cannot open 'zroot/jetpack/images/fdb9291c-26b1-4eff-a55f-ab71f4021c1b': dataset does not exist ""
/sbin/zfs list -H -p -tall -oname | "zroot\nzroot/jetpack\nzroot/jetpack/images\nzroot/jetpack/pods\n" panic: Not found

goroutine 1 [running]: lib/jetpack.(_Image).getRootfs(0xc8204ad680, 0xc8200dcb58) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/image.go:102 +0x16f lib/jetpack.(_Image).Destroy(0xc8204ad680, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/image.go:170 +0x975 lib/jetpack.(_Host).ImportImage.func1(0xc8204c8b20, 0xc8204ad680) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:475 +0x2f lib/jetpack.(_Host).ImportImage(0xc8200fe4c0, 0xc8201005a0, 0x19, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:501 +0x12a4 lib/jetpack.(_Host).fetchImage(0xc8200fe4c0, 0xc8201005a0, 0x19, 0xd5e5a0, 0x0, 0x0, 0x19, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:401 +0x2c8 lib/jetpack.(_Host).getImage(0xc8200fe4c0, 0x0, 0x0, 0x0, 0x0, 0xc8201005a0, 0x19, 0xd5e5a0, 0x0, 0x0, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:355 +0x220 lib/jetpack.(_Host).GetImage(0xc8200fe4c0, 0x0, 0x0, 0x0, 0x0, 0xc8201005a0, 0x19, 0xd5e5a0, 0x0, 0x0, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:304 +0xb7 lib/jetpack.(_Host).getRuntimeImage(0xc8200fe4c0, 0xc8201005c0, 0x0, 0x0, 0x0, 0x0, 0xd5e5a0, 0x0, 0x0, 0x40, ...) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:291 +0xb0 lib/jetpack.(_Host).ReifyPodManifest(0xc8200fe4c0, 0xc8200fc000, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/lib/jetpack/host.go:206 +0x15b main.getPodManifest(0xc82000a1f0, 0x1, 0x1, 0x0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:259 +0x148 main.getOrPreparePod(0xc82000a1f0, 0x1, 0x1, 0xc8200128a0, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:277 +0x148 main.cmdWrapPodPrepare0.func1(0xc82000a1f0, 0x1, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:186 +0x4e main.(_Command).Run(0xc8200e7260, 0xc82000a1f0, 0x1, 0x1, 0x0, 0x0) /usr/home/ec2-user/src/jetpack/src/bin/jetpack/command.go:80 +0xda main.main() /usr/home/ec2-user/src/jetpack/src/bin/jetpack/main.go:37 +0x156

goroutine 17 [syscall, locked to thread]: runtime.goexit() /usr/local/go/src/runtime/asm_amd64.s:1696 +0x1

goroutine 5 [syscall]: os/signal.loop() /usr/local/go/src/os/signal/signal_unix.go:22 +0x18 created by os/signal.init.1 /usr/local/go/src/os/signal/signal_unix.go:28 +0x37

goroutine 24 [IO wait]: net.runtime_pollWait(0x80150cd68, 0x72, 0xc82000c1e0) /usr/local/go/src/runtime/netpoll.go:157 +0x60 net.(_pollDesc).Wait(0xc820364300, 0x72, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:73 +0x3a net.(_pollDesc).WaitRead(0xc820364300, 0x0, 0x0) /usr/local/go/src/net/fd_poll_runtime.go:78 +0x36 net.(_netFD).Read(0xc8203642a0, 0xc82048a000, 0x2000, 0x2000, 0x0, 0x801507050, 0xc82000c1e0) /usr/local/go/src/net/fd_unix.go:232 +0x23a net.(_conn).Read(0xc820022040, 0xc82048a000, 0x2000, 0x2000, 0x0, 0x0, 0x0) /usr/local/go/src/net/net.go:172 +0xe4 crypto/tls.(_block).readFromUntil(0xc8204d2a80, 0x80150d110, 0xc820022040, 0x5, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:455 +0xcc crypto/tls.(_Conn).readRecord(0xc820084000, 0x9b1517, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:540 +0x2d1 crypto/tls.(_Conn).Read(0xc820084000, 0xc8204db000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/crypto/tls/conn.go:901 +0x167 net/http.noteEOFReader.Read(0x801517bb0, 0xc820084000, 0xc8203c8318, 0xc8204db000, 0x1000, 0x1000, 0x0, 0x0, 0x0) /usr/local/go/src/net/http/transport.go:1370 +0x67 net/http.(_noteEOFReader).Read(0xc8204cc200, 0xc8204db000, 0x1000, 0x1000, 0x0, 0x0, 0x0)

:126 +0xd0 bufio.(_Reader).fill(0xc8203cd3e0) /usr/local/go/src/bufio/bufio.go:97 +0x1e9 bufio.(_Reader).Peek(0xc8203cd3e0, 0x1, 0x0, 0x0, 0x0, 0x0, 0x0) /usr/local/go/src/bufio/bufio.go:132 +0xcc net/http.(_persistConn).readLoop(0xc8203c82c0) /usr/local/go/src/net/http/transport.go:876 +0xf7 created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:685 +0xc78 goroutine 25 [select]: net/http.(_persistConn).writeLoop(0xc8203c82c0) /usr/local/go/src/net/http/transport.go:1009 +0x40c created by net/http.(_Transport).dialConn /usr/local/go/src/net/http/transport.go:686 +0xc9d

mpasternacki commented 8 years ago

I'm out of ideas right now, will try and reproduce the issue tomorrow (it's 9PM here). It would be helpful if you could try jetpack fetch 3ofcoins.net/freebsd-base (will likely throw the same error, but won't hurt to check), and make sure your instance is at up-to-date patchlevel (use freebsd-update for that, unless the AMI is already patched).

One more random hunch: do you have ca_root_nss package installed? If not, can you check if installing it helps?

JayBusch commented 8 years ago

Actually that seemed to run fine; "jetpack fetch 3ofcoins.net/freebsd-base" produced:

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" Downloading https://3ofcoins-aci.s3.eu-central-1.amazonaws.com/3ofcoins.net/freebsd-base-latest-freebsd-amd64.aci.asc ... Downloading https://3ofcoins-aci.s3.eu-central-1.amazonaws.com/3ofcoins.net/freebsd-base-latest-freebsd-amd64.aci ... Progress: [======================================================] 70.1 MB/70.1 MB 3ofcoins.net/freebsd-base &{0xc820418ba0} &{0xc820454a80} 2015-11-13T20:08:54Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 Starting import of 3ofcoins.net/freebsd-base [lib/jetpack/host.go:442] 2015-11-13T20:08:54Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Checking signature [lib/jetpack/host.go:445] 2015-11-13T20:08:54Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 Image signed by an unknown issuer, attempting to discover public key... [lib/jetpack/host.go:450] Downloading https://3ofcoins-aci.s3.eu-central-1.amazonaws.com/aci-pubkeys.asc ... Prefix: 3ofcoins.net/freebsd-base GPG key fingerprint: 4706 DC5D 5C21 4BC3 AD12 7C6D 53CC C2D6 3A16 2664 Subkey fingerprint: A11F 0E36 5147 EFCD 8D2D 8EC6 68EF 14A2 3881 4C36
- ACI 3ofcoins.net (provisional) info@3ofcoins.net

Are you sure you want to trust this key (yes/no)? yes 2015-11-13T20:11:56Z jetpack Key accepted and saved as /var/jetpack/keys/3ofcoins.net,freebsd-base/4706dc5d5c214bc3ad127c6d53ccc2d63a162664 [lib/jetpack/host.go:635] 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 Valid signature for 3ofcoins.net/freebsd-base by: [lib/jetpack/host.go:461] 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 GPG key fingerprint: 4706 DC5D 5C21 4BC3 AD12 7C6D 53CC C2D6 3A16 2664 Subkey fingerprint: A11F 0E36 5147 EFCD 8D2D 8EC6 68EF 14A2 3881 4C36

ACI 3ofcoins.net (provisional) info@3ofcoins.net [lib/jetpack/host.go:462] 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Saving signature copy [lib/jetpack/host.go:485] 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Loading manifest [lib/jetpack/host.go:498]
- tar -xOqf - manifest | "{\"acKind\":\"ImageManifest\",\"acVersion\":\"0.7.1\",\"name\":\"3ofcoins.net/freebsd-base\",\"labels\":[{\"name\":\"version\",\"value\":\"10.2.5\"},{\"name\":\"os\",\"value\":\"freebsd\"},{\"name\":\"arch\",\"value\":\"amd64\"}],\"annotations\":[{\"name\":\"timestamp\",\"value\":\"2015-10-13T18:10:06+02:00\"}]}" 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: No dependencies to fetch [lib/jetpack/host.go:514]
- /sbin/zfs create -o mountpoint=/var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8/rootfs zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8
- /sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8/rootfs\norigin\t-\n" 2015-11-13T20:11:56Z image:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Saving manifest [lib/jetpack/image.go:221] 2015-11-13T20:11:56Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 Unpacking rootfs [lib/jetpack/host.go:546]
- tar -C /var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 -xf - rootfs & [97743] Progress: [======================================================] 70.1 MB/70.1 MB 2015-11-13T20:12:12Z import:8adc4423-aeb0-43ac-8f58-4395275d8bb8 Successfully imported sha512-8cc63935c53129dcdd27bf774833747296520c652143e124a69819616ee56d584ae5bf792ebbf5a0428d7eb6a30a16bb22326df48ad150a5459f981216a1abc0 [lib/jetpack/host.go:592] 2015-11-13T20:12:12Z image:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Sealing [lib/jetpack/image.go:240]
- /sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8/rootfs\norigin\t-\n"
- /sbin/zfs snapshot zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal
- /sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal | "type\tsnapshot\nmounted\t-\nmountpoint\t-\norigin\t-\n"
- /sbin/zfs set readonly=on zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 ID sha512-8cc63935c53129dcdd27bf774833747296520c652143e124a69819616ee56d584ae5bf792ebbf5a0428d7eb6a30a16bb22326df48ad150a5459f981216a1abc0 Name 3ofcoins.net/freebsd-base:10.2.5,arch="amd64",os="freebsd" Timestamp 2015-11-13T20:12:12Z

JayBusch commented 8 years ago

However a subsequent run of "jetpack run -t 3ofcoins.net/freebsd-base" produced:

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" 2015-11-13T20:13:05Z pod:d0c2657b-56fd-41e1-8eb7-e35a1baa1fae DEBUG: Initializing dataset [lib/jetpack/pod.go:84]
/sbin/zfs create zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae\norigin\t-\n" 2015-11-13T20:13:05Z pod:d0c2657b-56fd-41e1-8eb7-e35a1baa1fae DEBUG: Cloning rootfs.0 for app freebsd-base [lib/jetpack/pod.go:144] 2015-11-13T20:13:05Z image:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Cloning rootfs as zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0 at /var/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs/0 [lib/jetpack/image.go:183]
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8/rootfs\norigin\t-\n"
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal | "type\tsnapshot\nmounted\t-\nmountpoint\t-\norigin\t-\n"
/sbin/zfs clone -o mountpoint=/var/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs/0 zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs/0\norigin\tzroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal\n"
/sbin/zfs set jetpack:name=freebsd-base zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0
/sbin/zfs snapshot zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0@parent
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/d0c2657b-56fd-41e1-8eb7-e35a1baa1fae/rootfs.0@parent | "type\tsnapshot\nmounted\t-\nmountpoint\t-\norigin\t-\n" 2015-11-13T20:13:05Z pod:d0c2657b-56fd-41e1-8eb7-e35a1baa1fae DEBUG: Using IP 172.23.0.2 [lib/jetpack/pod.go:248] 2015-11-13T20:13:05Z pod:d0c2657b-56fd-41e1-8eb7-e35a1baa1fae DEBUG: Saving manifest [lib/jetpack/pod.go:256] Get http://172.23.0.1:1104/~6a065b993055d663670d630934db3222d22a1efa32f00ad65881ac81778e3b4f4b0f54237bce4e46c0f5d5f65d5d3be2585b48b4c8c71b89a0d987d9161480e7/_info: dial tcp 172.23.0.1:1104: connect: connection refused /usr/home/ec2-user/src/jetpack/src/lib/jetpack/mds.go:137: /usr/home/ec2-user/src/jetpack/src/lib/jetpack/mds.go:192: /usr/home/ec2-user/src/jetpack/src/lib/jetpack/app.go:68: /usr/home/ec2-user/src/jetpack/src/bin/jetpack/pod.go:102:

JayBusch commented 8 years ago

I updated the system such that freebsd-update outputs the following (last line only): "No updates needed to update system to 10.2-RELEASE-p7."

I then re-ran the jetpack run command, with the same output as above.

I will get started on the test scripts tonight; perhaps they will shed some light on the situation or provide a good environment to track this sort of issue down.

mpasternacki commented 8 years ago

You need to start the metadata service. Please run $(jetpack config path.libexecdir)/mds as _jetpack, or extract the rc.d script from the contrib/port/ directory and install it manually (may need some editing normally done by ports scripting) – it's not installed by make install, since it's managed by ports in most other software.

mpasternacki commented 8 years ago

(and thanks for trying jetpack fetch – this narrows down the debugging for me)

JayBusch commented 8 years ago

No problem.

I ran the following as root "su -m _jetpack -c /usr/local/libexec/jetpack/mds" (the shell variable wasn't working but I manually copied the output of "jetpack config path.libexec" instead). The result was:

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" 2015/11/13 21:19:03 Cannot clear groups:operation not permitted

Is this due to _jetpack not having a shell? I thought the -m option to su would resolve that but I'm not really sure.

mpasternacki commented 8 years ago

Aah, I forgot that mds drops UID on its own - please run it as root. Apologies for the misinformation.

JayBusch commented 8 years ago

Success!!! Thank you so much for your help!

after "/usr/local/libexec/jetpack/mds &" (as root) produced:

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" 2015/11/13 21:34:03 Listening on: 172.23.0.1:1104

--------THEN-------

"jetpack run -t 3ofcoins.net/freebsd-base" produced:

/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack\norigin\t-\n" 2015-11-13T21:35:22Z pod:4c3880b0-f90d-4751-845d-551c6a43b1b3 DEBUG: Initializing dataset [lib/jetpack/pod.go:84]
/sbin/zfs create zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3\norigin\t-\n" 2015-11-13T21:35:22Z pod:4c3880b0-f90d-4751-845d-551c6a43b1b3 DEBUG: Cloning rootfs.0 for app freebsd-base [lib/jetpack/pod.go:144] 2015-11-13T21:35:22Z image:8adc4423-aeb0-43ac-8f58-4395275d8bb8 DEBUG: Cloning rootfs as zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0 at /var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs/0 [lib/jetpack/image.go:183]
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8/rootfs\norigin\t-\n"
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal | "type\tsnapshot\nmounted\t-\nmountpoint\t-\norigin\t-\n"
/sbin/zfs clone -o mountpoint=/var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs/0 zroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0 | "type\tfilesystem\nmounted\tyes\nmountpoint\t/var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs/0\norigin\tzroot/jetpack/images/8adc4423-aeb0-43ac-8f58-4395275d8bb8@seal\n"
/sbin/zfs set jetpack:name=freebsd-base zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0
/sbin/zfs snapshot zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0@parent
/sbin/zfs get -H -p -oproperty,value type,mounted,mountpoint,origin zroot/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs.0@parent | "type\tsnapshot\nmounted\t-\nmountpoint\t-\norigin\t-\n" 2015-11-13T21:35:22Z pod:d0c2657b-56fd-41e1-8eb7-e35a1baa1fae DEBUG: Loading manifest [lib/jetpack/pod.go:301] 2015-11-13T21:35:22Z pod:ecc588ec-37b8-484a-95df-1c7e22942299 DEBUG: Loading manifest [lib/jetpack/pod.go:301] 2015-11-13T21:35:22Z pod:4c3880b0-f90d-4751-845d-551c6a43b1b3 DEBUG: Using IP 172.23.0.4 [lib/jetpack/pod.go:248] 2015-11-13T21:35:22Z pod:4c3880b0-f90d-4751-845d-551c6a43b1b3 DEBUG: Saving manifest [lib/jetpack/pod.go:256] 172.23.0.1 - host [2015-11-13 21:35:22.550382809 +0000 UTC] "GET /_info" 200 82 "-" "-"
/usr/sbin/jls -d jid dying name | "" 2015-11-13T21:35:22Z pod:4c3880b0-f90d-4751-845d-551c6a43b1b3 DEBUG: Running: jail -c [lib/jetpack/pod.go:429]
jail -f /var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/jail.conf -v -c jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3 jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3: run command: /sbin/ifconfig lo1 inet 172.23.0.4 netmask 255.255.255.255 alias jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3: run command: /sbin/mount -t devfs -o ruleset=4 . /var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs/0/dev jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3: jail_set(JAIL_CREATE) name=jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3 host.hostname=4c3880b0-f90d-4751-845d-551c6a43b1b3 host.hostuuid=4c3880b0-f90d-4751-845d-551c6a43b1b3 ip4.addr=172.23.0.4 osrelease=10.2-RELEASE path=/var/jetpack/pods/4c3880b0-f90d-4751-845d-551c6a43b1b3/rootfs persist=true securelevel=2 jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3: created
/usr/sbin/jls -d jid dying name | "1 0 jetpack/4c3880b0-f90d-4751-845d-551c6a43b1b3\n"
/usr/local/libexec/jetpack/stage2 1:0:0:freebsd-base:/ 'AC_METADATA_URL=http://172.23.0.1:1104/~dc6d74b1a38ce1b16d5b881d1c3f28895d827267f71b68829830d5960ac9f0da7ad100c71fe8d623c4e8ad6c97843443b4f8926f2dc68406d08e82f535c32acc' USER=root LOGNAME=root HOME=/root SHELL=/bin/csh PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin TERM=xterm-256color /usr/bin/login -fp root FreeBSD ?.?.? (UNKNOWN)

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/ Security Advisories: https://www.FreeBSD.org/security/ FreeBSD Handbook: https://www.FreeBSD.org/handbook/ FreeBSD FAQ: https://www.FreeBSD.org/faq/ Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/ FreeBSD Forums: https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/ directory, or can be installed later with: pkg install en-freebsd-doc For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed: freebsd-version ; uname -a Please include that output and any error messages when posting questions. Introduction to manual pages: man man FreeBSD directory layout: man hier

Edit /etc/motd to change this login announcement. root@4c3880b0-f90d-4751-845d-551c6a43b1b3:~ #

mpasternacki commented 8 years ago

:+1: Yay! Now I should go and work to make setting it up less confusing.

JayBusch commented 8 years ago

I'll edit the documentation on my fork a bit and if you like my changes you can pull them of course.

mpasternacki commented 8 years ago

When you have a moment, please confirm that jetpack run -t 3ofcoins.net/freebsd-base when no images are present doesn't fail anymore for you (use jetpack destroy to remove all leftover pods, and jetpack image-destroy to destroy the already downloaded image).

JayBusch commented 8 years ago

Seems to work great now! See the following shell output:

root@ip-XXX-XX-XX-XXX:~ # jetpack images ID NAME root@ip-XXX-XX-XX-XXX:~ # jetpack list ID STATUS IP APPS
root@ip-XXX-XX-XX-XXX:~ # jetpack run -t 3ofcoins.net/freebsd-base Downloading https://3ofcoins-aci.s3.eu-central-1.amazonaws.com/3ofcoins.net/freebsd-base-latest-freebsd-amd64.aci.asc ... Downloading https://3ofcoins-aci.s3.eu-central-1.amazonaws.com/3ofcoins.net/freebsd-base-latest-freebsd-amd64.aci ... Progress: [======================================================] 70.5 MB/70.5 MB 2015-11-14T20:41:45Z import:d90dd0d6-7323-454b-b6c1-afb59d3eb480 Starting import of 3ofcoins.net/freebsd-base 2015-11-14T20:41:46Z import:d90dd0d6-7323-454b-b6c1-afb59d3eb480 Valid signature for 3ofcoins.net/freebsd-base by: 2015-11-14T20:41:46Z import:d90dd0d6-7323-454b-b6c1-afb59d3eb480 GPG key fingerprint: 4706 DC5D 5C21 4BC3 AD12 7C6D 53CC C2D6 3A16 2664 Subkey fingerprint: A11F 0E36 5147 EFCD 8D2D 8EC6 68EF 14A2 3881 4C36

ACI 3ofcoins.net (provisional) info@3ofcoins.net

2015-11-14T20:41:46Z import:d90dd0d6-7323-454b-b6c1-afb59d3eb480 Unpacking rootfs Progress: [======================================================] 70.5 MB/70.5 MB 2015-11-14T20:41:59Z import:d90dd0d6-7323-454b-b6c1-afb59d3eb480 Successfully imported sha512-037411fd3cdc2fef575165c78cbd949f0817b49495e41fcc72c0910ccc28f78bcbee3e38735158ef110f3a5cfb99b0abd5e13dcfec34c93a0248e6cb754e27f4 172.23.0.1 - host [2015-11-14 20:42:00.719689823 +0000 UTC] "GET /_info" 200 82 "-" "-" FreeBSD ?.?.? (UNKNOWN)

Welcome to FreeBSD!

Edit /etc/motd to change this login announcement. root@ed0933f3-268f-4043-b469-cf0ed0779014:~ #