search

nanovms / ops

ops - build and run nanos unikernels
https://ops.city
MIT License
1.27k stars 132 forks source link

tfs: support new tag types #1499

Closed wjhun closed 1 year ago

wjhun commented 1 year ago

This change adds support for vector, immediate integer and string types within the tfs implementation, advancing tfsVersion to 5. This should only be merged once TFS v5 support has been merged into Nanos and released.

See https://github.com/nanovms/nanos/pull/1903

rinor commented 1 year ago

I'm testing this pr with https://github.com/nanovms/nanos/pull/1903 and while trying to read the image data from ops I'm getting

Cannot load image gosed: cannot read filesystem log extension: unexpected name type for symbol

config.json ops ```json { "BaseVolumeSz": "20m", "Files": [ "gosede.main" ], "Klibs": [ "ntp" ], "RunConfig": { "Verbose": true, "AtExit": "echo AtExit - done $?" }, "ManifestPassthrough": { "readonly": {}, "exec_protection": "t", "exec_wait_for_ip4_secs": "5", "ntp_servers": [ "0.europe.pool.ntp.org", "1.europe.pool.ntp.org", "2.europe.pool.ntp.org", "3.europe.pool.ntp.org" ], "ntp_poll_min": "5", "ntp_poll_max": "10" }, "Boot": "/usr/local/src/nanos/output/platform/pc/boot/boot.img", "Kernel": "/usr/local/src/nanos/output/platform/pc/bin/kernel.img", "KlibDir": "/usr/local/src/nanos/output/klib/bin", "Debugflags": [ "reboot_on_exit" ] } ```
ops image tree gosede error ```sh Cannot load image gosed: cannot read filesystem log extension: unexpected name type for symbol ```
ops image tree gosede custom build with debug ```sh getHeader: offset(21) - nameEntry(1) - nameType(4) -> symbol(exec_protection) = value("t") getHeader: offset(47) - nameEntry(1) - nameType(4) -> symbol(exec_wait_for_ip4_secs) = value("5") getHeader: offset(66) - nameEntry(1) - nameType(4) -> symbol(arguments) = value([ "gosede" ]) getHeader: offset(82) - nameEntry(1) - nameType(4) -> symbol(program) = value("/gosede") getHeader: offset(95) - nameEntry(1) - nameType(4) -> symbol(klibs) = value("bootfs") getHeader: offset(201) - nameEntry(1) - nameType(4) -> symbol(ntp_servers) = value([ "0.europe.pool.ntp.org", "1.europe.pool.ntp.org", "2.europe.pool.ntp.org", "3.europe.pool.ntp.org" ]) getHeader: offset(212) - nameEntry(1) - nameType(4) -> symbol(readonly) = value({}) getHeader: offset(276) - nameEntry(1) - nameType(4) -> symbol(filelength) = value("7") getHeader: offset(297) - nameEntry(1) - nameType(4) -> symbol(length) = value("1") getHeader: offset(309) - nameEntry(1) - nameType(4) -> symbol(offset) = value("1025") getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(allocated) = value("1") getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(0) = value({ "allocated": "1", "length": "1", "offset": "1025" }) getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(extents) = value({ "0": { "allocated": "1", "length": "1", "offset": "1025" } }) getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(hostname) = value({ "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" }) getHeader: offset(322) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } }) getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(kernel) = value({ "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } }) getHeader: offset(322) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "kernel": { "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } } }) getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(sys) = value({ "children": { "kernel": { "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } } } }) getHeader: offset(322) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "sys": { "children": { "kernel": { "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } } } } }) getHeader: offset(322) - nameEntry(1) - nameType(4) -> symbol(proc) = value({ "children": { "sys": { "children": { "kernel": { "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } } } } } }) getHeader: offset(346) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("1630208") getHeader: offset(359) - nameEntry(0) - nameType(0) -> symbol(length) = value("3184") getHeader: offset(366) - nameEntry(0) - nameType(0) -> symbol(offset) = value("1026") getHeader: offset(373) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("3184") getHeader: offset(373) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "3184", "length": "3184", "offset": "1026" }) getHeader: offset(373) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "3184", "length": "3184", "offset": "1026" } }) getHeader: offset(373) - nameEntry(1) - nameType(4) -> symbol(gosede.main) = value({ "extents": { "0": { "allocated": "3184", "length": "3184", "offset": "1026" } }, "filelength": "1630208" }) getHeader: offset(429) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("26936") getHeader: offset(440) - nameEntry(0) - nameType(0) -> symbol(length) = value("53") getHeader: offset(447) - nameEntry(0) - nameType(0) -> symbol(offset) = value("4210") getHeader: offset(452) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("53") getHeader: offset(452) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "53", "length": "53", "offset": "4210" }) getHeader: offset(452) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "53", "length": "53", "offset": "4210" } }) getHeader: offset(452) - nameEntry(1) - nameType(4) -> symbol(libnss_dns.so.2) = value({ "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" }) getHeader: offset(452) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "libnss_dns.so.2": { "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" } }) getHeader: offset(452) - nameEntry(1) - nameType(4) -> symbol(x86_64-linux-gnu) = value({ "children": { "libnss_dns.so.2": { "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" } } }) getHeader: offset(452) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "x86_64-linux-gnu": { "children": { "libnss_dns.so.2": { "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" } } } }) getHeader: offset(452) - nameEntry(1) - nameType(4) -> symbol(lib) = value({ "children": { "x86_64-linux-gnu": { "children": { "libnss_dns.so.2": { "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" } } } } }) getHeader: offset(479) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("20") getHeader: offset(489) - nameEntry(0) - nameType(0) -> symbol(length) = value("1") getHeader: offset(496) - nameEntry(0) - nameType(0) -> symbol(offset) = value("4263") getHeader: offset(500) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("1") getHeader: offset(500) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "1", "length": "1", "offset": "4263" }) getHeader: offset(500) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "1", "length": "1", "offset": "4263" } }) getHeader: offset(500) - nameEntry(1) - nameType(4) -> symbol(resolv.conf) = value({ "extents": { "0": { "allocated": "1", "length": "1", "offset": "4263" } }, "filelength": "20" }) getHeader: offset(513) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("33") getHeader: offset(523) - nameEntry(0) - nameType(0) -> symbol(length) = value("1") getHeader: offset(530) - nameEntry(0) - nameType(0) -> symbol(offset) = value("4264") getHeader: offset(534) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("1") getHeader: offset(534) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "1", "length": "1", "offset": "4264" }) getHeader: offset(534) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "1", "length": "1", "offset": "4264" } }) getHeader: offset(534) - nameEntry(1) - nameType(4) -> symbol(passwd) = value({ "extents": { "0": { "allocated": "1", "length": "1", "offset": "4264" } }, "filelength": "33" }) getHeader: offset(586) - nameEntry(0) - nameType(0) -> symbol(length) = value("406") getHeader: offset(593) - nameEntry(0) - nameType(0) -> symbol(offset) = value("4265") getHeader: offset(599) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("406") getHeader: offset(599) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "406", "length": "406", "offset": "4265" }) getHeader: offset(599) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "406", "length": "406", "offset": "4265" } }) getHeader: offset(608) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("207436") getHeader: offset(608) - nameEntry(1) - nameType(4) -> symbol(ca-certificates.crt) = value({ "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" }) getHeader: offset(608) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } }) getHeader: offset(608) - nameEntry(1) - nameType(4) -> symbol(certs) = value({ "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } }) getHeader: offset(608) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "certs": { "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } } }) getHeader: offset(608) - nameEntry(1) - nameType(4) -> symbol(ssl) = value({ "children": { "certs": { "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } } } }) getHeader: offset(608) - nameEntry(0) - nameType(0) -> symbol(children) = value({ "passwd": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4264" } }, "filelength": "33" }, "resolv.conf": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4263" } }, "filelength": "20" }, "ssl": { "children": { "certs": { "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } } } } }) getHeader: offset(608) - nameEntry(1) - nameType(4) -> symbol(etc) = value({ "children": { "passwd": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4264" } }, "filelength": "33" }, "resolv.conf": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4263" } }, "filelength": "20" }, "ssl": { "children": { "certs": { "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } } } } } }) getHeader: offset(626) - nameEntry(0) - nameType(0) -> symbol(filelength) = value("1630208") getHeader: offset(639) - nameEntry(0) - nameType(0) -> symbol(length) = value("3184") getHeader: offset(646) - nameEntry(0) - nameType(0) -> symbol(offset) = value("4671") getHeader: offset(653) - nameEntry(0) - nameType(0) -> symbol(allocated) = value("3184") getHeader: offset(653) - nameEntry(0) - nameType(0) -> symbol(0) = value({ "allocated": "3184", "length": "3184", "offset": "4671" }) getHeader: offset(653) - nameEntry(0) - nameType(0) -> symbol(extents) = value({ "0": { "allocated": "3184", "length": "3184", "offset": "4671" } }) getHeader: offset(653) - nameEntry(1) - nameType(4) -> symbol(gosede) = value({ "extents": { "0": { "allocated": "3184", "length": "3184", "offset": "4671" } }, "filelength": "1630208" }) getHeader: offset(653) - nameEntry(1) - nameType(4) -> symbol(children) = value({ "etc": { "children": { "passwd": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4264" } }, "filelength": "33" }, "resolv.conf": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "4263" } }, "filelength": "20" }, "ssl": { "children": { "certs": { "children": { "ca-certificates.crt": { "extents": { "0": { "allocated": "406", "length": "406", "offset": "4265" } }, "filelength": "207436" } } } } } } }, "gosede": { "extents": { "0": { "allocated": "3184", "length": "3184", "offset": "4671" } }, "filelength": "1630208" }, "gosede.main": { "extents": { "0": { "allocated": "3184", "length": "3184", "offset": "1026" } }, "filelength": "1630208" }, "lib": { "children": { "x86_64-linux-gnu": { "children": { "libnss_dns.so.2": { "extents": { "0": { "allocated": "53", "length": "53", "offset": "4210" } }, "filelength": "26936" } } } } }, "proc": { "children": { "sys": { "children": { "kernel": { "children": { "hostname": { "extents": { "0": { "allocated": "1", "length": "1", "offset": "1025" } }, "filelength": "7" } } } } } } } }) getHeader: offset(689) - nameEntry(1) - nameType(4) -> symbol(NANOS_VERSION) = value("0.1.46") getHeader: offset(708) - nameEntry(1) - nameType(4) -> symbol(IMAGE_NAME) = value("gosede") getHeader: offset(718) - nameEntry(1) - nameType(4) -> symbol(USER) = value("root") getHeader: offset(724) - nameEntry(1) - nameType(4) -> symbol(PWD) = value("/") getHeader: offset(760) - nameEntry(1) - nameType(4) -> symbol(OPS_VERSION) = value("new-tag-types-452a413") getHeader: offset(760) - nameEntry(1) - nameType(4) -> symbol(environment) = value({ "IMAGE_NAME": "gosede", "NANOS_VERSION": "0.1.46", "OPS_VERSION": "new-tag-types-452a413", "PWD": "/", "USER": "root" }) getHeader: offset(776) - nameEntry(1) - nameType(4) -> symbol(ntp_poll_min) = value("5") getHeader: offset(793) - nameEntry(1) - nameType(4) -> symbol(ntp_poll_max) = value("10") ```

After a test patch I get the correct content:

ops/fs/tfs.go patch ```patch diff --git a/fs/tfs.go b/fs/tfs.go index 90fe59a..b60ee35 100644 --- a/fs/tfs.go +++ b/fs/tfs.go @@ -393,7 +393,7 @@ func (t *tfs) decodeTuple(buffer []byte, offset *uint64, entry byte, length uint if err != nil { return tuple, err } - if nameType != typeBuffer { + if !(nameEntry == 0 && nameType == typeBuffer) && !(nameEntry == 1 && nameType == typeString) { return tuple, errors.New("unexpected name type for symbol") } symbol, err := t.decodeSymbol(buffer, offset, nameEntry, n) ```
output after test patch ```sh / | proc | | sys | | | kernel | | | | hostname | gosede.main | lib | | x86_64-linux-gnu | | | libnss_dns.so.2 | etc | | resolv.conf | | passwd | | ssl | | | certs | | | | ca-certificates.crt | gosede ```

This is related to https://github.com/nanovms/nanos/pull/1903#issuecomment-1649882096

wjhun commented 1 year ago

Cannot load image gosed: cannot read filesystem log extension: unexpected name type for symbol

Indeed, the ops tfs writing was using an incorrect type for the symbol string. I force-pushed a fix.

Thanks!