V2 B-Trees (ASCII Signature BTHD) for Record type 8 (CURR: REFACTORING)

ksharonin commented 7 months ago

See HDF5 Docs: https://docs.hdfgroup.org/hdf5/v1_10/_f_m_t3.html#Btrees:~:text=III.A.2.%20Disk%20Format%3A%20Level%201A2%20%2D%20Version%202%20B%2Dtrees

Lua args:

File: OR_ABI-L2-FDCC-M3_G17_s20182390052191_e20182390054564_c20182390055159.nc
Path: "/Power/standard_name"
core.TEXT

Dump log:

...
2024-01-22T02:24:37.648Z 172.17.0.2:CRITICAL:H5Coro.cpp:2908 received attr_name: _FillValue
2024-01-22T02:24:37.648Z 172.17.0.2:CRITICAL:H5Coro.cpp:2249 datamessage databits: 7968

----------------
B-Tree Node: 0x3ce4
----------------
Node Level:                                                      0
Entries Used:                                                    60
2024-01-22T02:24:37.741Z 172.17.0.2:DEBUG:H5Coro.cpp:3733 Read 1500 elements (6000 bytes) from local//Power/standard_name
2024-01-22T02:24:37.741Z 172.17.0.2:DEBUG:LuaObject.cpp:304 Created object of type DeviceIO/DeviceReader
2024-01-22T02:24:37.742Z 172.17.0.2:DEBUG:DeviceReader.cpp:147 shutting down device and exiting reader
vals recieved with type: nil
scripts/selftests/h5coro.lua:91: attempt to get length of a nil value (global 'vals')
stack traceback:
        scripts/selftests/h5coro.lua:91: in main chunk
        [C]: in ?
...

ksharonin commented 7 months ago

Assume entry address is the same with: metaData.offsetsize = readField(1, &pos);

/* Read B-Tree */
readBTreeV1(metaData.address, buffer, buffer_size, buffer_offset);

ksharonin commented 7 months ago

Alternatively per suggestion: I would still see if you can read the variables using h5coro with only the V1 btree. It could be that the hdf5 library favors the improved method but doesn't absolutely need to.

ksharonin commented 7 months ago

 thread #1, queue = 'com.apple.main-thread', stop reason = step over
  * frame #0: 0x00000001002d3d68 libhdf5.310.dylib`H5O__attr_decode(f=<unavailable>, open_oh=<unavailable>, mesg_flags=<unavailable>, ioflags=<unavailable>, p_size=<unavailable>, p="fire_radiative_power\U00000003") at H5Oattr.c:286:11 [opt]
    frame #1: 0x00000001002d2208 libhdf5.310.dylib`H5O__attr_shared_decode(f=<unavailable>, open_oh=<unavailable>, mesg_flags=<unavailable>, ioflags=0x000000016fdfe9bc, p_size=<unavailable>, p=<unavailable>) at H5Oshared.h:73:34 [opt]
    frame #2: 0x0000000100302900 libhdf5.310.dylib`H5O_msg_decode(f=<unavailable>, open_oh=0x0000000000000000, type_id=12, buf_size=<unavailable>, buf=<unavailable>) at H5Omessage.c:1635:30 [opt]
    frame #3: 0x000000010012be34 libhdf5.310.dylib`H5A__dense_fh_name_cmp(obj=0x0000000101812740, obj_len=55, _udata=0x000000016fdfeb10) at H5Abtree2.c:153:26 [opt]
    frame #4: 0x000000010028f308 libhdf5.310.dylib`H5HF__man_op_real(hdr=0x0000000100b42f00, id=<unavailable>, op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb10, op_flags=<unavailable>) at H5HFman.c:404:9 [opt]
    frame #5: 0x000000010028f4e4 libhdf5.310.dylib`H5HF__man_op(hdr=0x0000000100b42f00, id="", op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb10) at H5HFman.c:508:9 [opt]
    frame #6: 0x0000000100276e38 libhdf5.310.dylib`H5HF_op(fh=0x0000000100b3ef00, _id=0x0000000100b43918, op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb10) at H5HF.c:656:13 [opt]
    frame #7: 0x000000010012b7bc libhdf5.310.dylib`H5A__dense_btree2_name_compare(_bt2_udata=0x000000016fdfec90, _bt2_rec=0x0000000100b43918, result=0x000000016fdfebfc) at H5Abtree2.c:263:13 [opt]
    frame #8: 0x000000010014b530 libhdf5.310.dylib`H5B2__locate_record(type=0x00000001005ad810, nrec=<unavailable>, rec_off=0x0000000100b42d78, native="", udata=0x000000016fdfec90, idx=0x000000016fdfebf8, cmp=0x000000016fdfebfc) at H5B2int.c:103:13 [opt]
    frame #9: 0x00000001001449d4 libhdf5.310.dylib`H5B2_find(bt2=<unavailable>, udata=0x000000016fdfec90, found=0x000000016fdfec8f, op=0x0000000000000000, op_data=0x0000000000000000) at H5B2.c:605:13 [opt]
    frame #10: 0x000000010012c6f8 libhdf5.310.dylib`H5A__dense_open(f=0x0000000100b3c550, ainfo=<unavailable>, name="standard_name") at H5Adense.c:380:9 [opt]
    frame #11: 0x00000001002d53b0 libhdf5.310.dylib`H5O__attr_open_by_name(loc=0x000000016fdfeda0, name="standard_name") at H5Oattribute.c:472:40 [opt]
    frame #12: 0x0000000100133974 libhdf5.310.dylib`H5A__open_by_name(loc=0x000000016fdfee20, obj_name="/Power", attr_name="standard_name") at H5Aint.c:637:25 [opt]
    frame #13: 0x000000010045e038 libhdf5.310.dylib`H5VL__native_attr_open(obj=0x0000000100b3c550, loc_params=0x000000016fdfef68, attr_name="standard_name", aapl_id=792633534417207311, dxpl_id=<unavailable>, req=<unavailable>) at H5VLnative_attr.c:168:29 [opt]
    frame #14: 0x0000000100441954 libhdf5.310.dylib`H5VL__attr_open(obj=0x0000000100b3c550, loc_params=0x000000016fdfef68, cls=0x0000000100b10310, name="standard_name", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1104:30 [opt]
    frame #15: 0x00000001004417a4 libhdf5.310.dylib`H5VL_attr_open(vol_obj=0x0000000100b3d6e0, loc_params=0x000000016fdfef68, name="standard_name", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1136:30 [opt]
    frame #16: 0x000000010012b26c libhdf5.310.dylib`H5A__open_common(vol_obj=0x0000000100b3d6e0, loc_params=0x000000016fdfef68, attr_name="standard_name", aapl_id=792633534417207311, token_ptr=0x0000000000000000) at H5A.c:463:17 [opt]
    frame #17: 0x00000001001235e4 libhdf5.310.dylib`H5A__open_by_name_api_common(loc_id=72057594037927936, obj_name="/Power", attr_name="standard_name", aapl_id=<unavailable>, lapl_id=0, token_ptr=0x0000000000000000, _vol_obj_ptr=<unavailable>) at H5A.c:640:22 [opt]
    frame #18: 0x0000000100123344 libhdf5.310.dylib`H5Aopen_by_name(loc_id=72057594037927936, obj_name="/Power", attr_name="standard_name", aapl_id=0, lapl_id=0) at H5A.c:678:14 [opt]
    frame #19: 0x0000000100003a10 run_hdf`main at run_hdf5.c:23:20
    frame #20: 0x000000018ac64420 libdyld.dylib`start + 4

ksharonin commented 7 months ago

"The entry into a version 2 B-tree is a header which contains global information about the structure of the B-tree. The root node address field in the header points to the B-tree root node, which is either an internal or leaf node, depending on the value in the header’s depth field. An internal node consists of records plus pointers to further leaf or internal nodes in the tree. A leaf node consists of solely of records. The format of the records depends on the B-tree type (stored in the header)."

Requires both for potential navigation; persistence however is not necessary when navigating downwards

ksharonin commented 7 months ago

Bt for setting name hash break set --file H5checksum.c --line 367 requires several continuations

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x000000010011b978 libhdf5.310.dylib`H5_checksum_lookup3(key=0x0000000100f07540, length=50, initval=0) at H5checksum.c:370:5 [opt]
    frame #1: 0x000000010011bb8c libhdf5.310.dylib`H5_checksum_metadata(data=<unavailable>, len=<unavailable>, initval=0) at H5checksum.c:482:5 [opt]
    frame #2: 0x0000000100208048 libhdf5.310.dylib`H5F_get_checksums(buf="FHIB", buf_size=54, s_chksum=0x000000016fdfe7ac, c_chksum=0x000000016fdfe7a8) at H5Fio.c:521:21 [opt]
    frame #3: 0x000000010027b590 libhdf5.310.dylib`H5HF__cache_iblock_verify_chksum(_image=0x0000000100f07540, len=54, _udata=<unavailable>) at H5HFcache.c:877:5 [opt]
    frame #4: 0x0000000100167b34 libhdf5.310.dylib`H5C__load_entry(f=0x0000000100b3c490, type=0x00000001005afbf0, addr=7212, udata=0x000000016fdfe918) at H5Centry.c:1177:28 [opt]
    frame #5: 0x0000000100164d78 libhdf5.310.dylib`H5C_protect(f=0x0000000100b3c490, type=0x00000001005afbf0, addr=7212, udata=0x000000016fdfe918, flags=512) at H5Centry.c:3120:30 [opt]
    frame #6: 0x000000010013b19c libhdf5.310.dylib`H5AC_protect(f=0x0000000100b3c490, type=0x00000001005afbf0, addr=7212, udata=0x000000016fdfe918, flags=512) at H5AC.c:1276:26 [opt]
    frame #7: 0x000000010028ae08 libhdf5.310.dylib`H5HF__man_iblock_protect(hdr=0x0000000102004f10, iblock_addr=7212, iblock_nrows=<unavailable>, par_iblock=0x0000000000000000, par_entry=0, must_protect=false, flags=512, did_protect=0x000000016fdfe9d7) at H5HFiblock.c:1154:50 [opt]
    frame #8: 0x00000001002832c4 libhdf5.310.dylib`H5HF__man_dblock_locate(hdr=0x0000000102004f10, obj_off=441, ret_iblock=0x000000016fdfea48, ret_entry=0x000000016fdfea40, ret_did_protect=0x000000016fdfea47, flags=512) at H5HFdblock.c:525:27 [opt]
    frame #9: 0x000000010028ef0c libhdf5.310.dylib`H5HF__man_op_real(hdr=0x0000000102004f10, id=<unavailable>, op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb40, op_flags=<unavailable>) at H5HFman.c:352:13 [opt]
    frame #10: 0x000000010028f4e4 libhdf5.310.dylib`H5HF__man_op(hdr=0x0000000102004f10, id="", op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb40) at H5HFman.c:508:9 [opt]
    frame #11: 0x0000000100276e38 libhdf5.310.dylib`H5HF_op(fh=0x0000000102004630, _id=0x0000000100f072e0, op=(libhdf5.310.dylib`H5A__dense_fh_name_cmp at H5Abtree2.c:143), op_data=0x000000016fdfeb40) at H5HF.c:656:13 [opt]
    frame #12: 0x000000010012b7bc libhdf5.310.dylib`H5A__dense_btree2_name_compare(_bt2_udata=0x000000016fdfecc0, _bt2_rec=0x0000000100f072e0, result=0x000000016fdfec2c) at H5Abtree2.c:263:13 [opt]
    frame #13: 0x000000010014b530 libhdf5.310.dylib`H5B2__locate_record(type=0x00000001005ad810, nrec=<unavailable>, rec_off=0x0000000100f06fb8, native="", udata=0x000000016fdfecc0, idx=0x000000016fdfec28, cmp=0x000000016fdfec2c) at H5B2int.c:103:13 [opt]
    frame #14: 0x00000001001449d4 libhdf5.310.dylib`H5B2_find(bt2=<unavailable>, udata=0x000000016fdfecc0, found=0x000000016fdfecbf, op=0x0000000000000000, op_data=0x0000000000000000) at H5B2.c:605:13 [opt]
    frame #15: 0x000000010012c6f8 libhdf5.310.dylib`H5A__dense_open(f=0x0000000100b3c490, ainfo=<unavailable>, name="scale_factor") at H5Adense.c:380:9 [opt]
    frame #16: 0x00000001002d53b0 libhdf5.310.dylib`H5O__attr_open_by_name(loc=0x000000016fdfedd0, name="scale_factor") at H5Oattribute.c:472:40 [opt]
    frame #17: 0x0000000100133974 libhdf5.310.dylib`H5A__open_by_name(loc=0x000000016fdfee50, obj_name="/Temp", attr_name="scale_factor") at H5Aint.c:637:25 [opt]
    frame #18: 0x000000010045e038 libhdf5.310.dylib`H5VL__native_attr_open(obj=0x0000000100b3c490, loc_params=0x000000016fdfef98, attr_name="scale_factor", aapl_id=792633534417207311, dxpl_id=<unavailable>, req=<unavailable>) at H5VLnative_attr.c:168:29 [opt]
    frame #19: 0x0000000100441954 libhdf5.310.dylib`H5VL__attr_open(obj=0x0000000100b3c490, loc_params=0x000000016fdfef98, cls=0x0000000100b10250, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1104:30 [opt]
    frame #20: 0x00000001004417a4 libhdf5.310.dylib`H5VL_attr_open(vol_obj=0x0000000100e04f50, loc_params=0x000000016fdfef98, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1136:30 [opt]
    frame #21: 0x000000010012b26c libhdf5.310.dylib`H5A__open_common(vol_obj=0x0000000100e04f50, loc_params=0x000000016fdfef98, attr_name="scale_factor", aapl_id=792633534417207311, token_ptr=0x0000000000000000) at H5A.c:463:17 [opt]
    frame #22: 0x00000001001235e4 libhdf5.310.dylib`H5A__open_by_name_api_common(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=<unavailable>, lapl_id=0, token_ptr=0x0000000000000000, _vol_obj_ptr=<unavailable>) at H5A.c:640:22 [opt]
    frame #23: 0x0000000100123344 libhdf5.310.dylib`H5Aopen_by_name(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=0, lapl_id=0) at H5A.c:678:14 [opt]
    frame #24: 0x0000000100003a14 run_hdf`main at run_hdf5.c:23:20
    frame #25: 0x0000000192ba8420 libdyld.dylib`start + 4

ksharonin commented 7 months ago

For navigation on heap reading, note that taken path: /* Check type of object in heap */ if ((id_flags & H5HF_ID_TYPE_MASK) == H5HF_ID_TYPE_MAN).

Per forum: https://forum.hdfgroup.org/t/hdf5-specs-unclear-b-tree-v2-and-fractal-heap/8723 The definition of the fractal heap ID can be found in the file format document after the definition of the fractal heap indirect block. As an overview, the ID can refer to a “tiny”, “huge”, or “managed” object. If it’s tiny, the ID contains the actual data and the heap itself does not need to be read from. If it’s huge, the ID contains the address on disk of the data or a b-tree key that can be used to find this address. If it’s managed, then it contains the offset and length within the virtual fractal heap address space (i.e. inside a direct block, possibly indexed by one or more indirect blocks). You can compute which direct and indirect blocks contains the data, and the offset within the direct block mathematically using the various parameters and algorithms described at the start of the fractal heap section - it describes an array of blocks of increasing size within a linear address space.

ksharonin commented 7 months ago

Header init; this is required for parsing out relevant info for record usage

* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000100149e28 libhdf5.310.dylib`H5B2__hdr_init(hdr=0x0000000100e04fd0, cparam=0x000000016fdfeac8, ctx_udata=0x0000000000000000, depth=0) at H5B2hdr.c:106:5 [opt]
    frame #1: 0x00000001001470bc libhdf5.310.dylib`H5B2__cache_hdr_deserialize(_image=0x0000000100d07690, len=38, _udata=0x000000016fdfec68, dirty=<unavailable>) at H5B2cache.c:286:9 [opt]
    frame #2: 0x0000000100167d80 libhdf5.310.dylib`H5C__load_entry(f=0x0000000100c3c490, type=0x00000001005adaf0, addr=744139, udata=0x000000016fdfec68) at H5Centry.c:1203:26 [opt]
    frame #3: 0x0000000100164d78 libhdf5.310.dylib`H5C_protect(f=0x0000000100c3c490, type=0x00000001005adaf0, addr=744139, udata=0x000000016fdfec68, flags=512) at H5Centry.c:3120:30 [opt]
    frame #4: 0x000000010013b19c libhdf5.310.dylib`H5AC_protect(f=0x0000000100c3c490, type=0x00000001005adaf0, addr=744139, udata=0x000000016fdfec68, flags=512) at H5AC.c:1276:26 [opt]
    frame #5: 0x000000010014af10 libhdf5.310.dylib`H5B2__hdr_protect(f=0x0000000100c3c490, hdr_addr=744139, ctx_udata=0x0000000000000000, flags=512) at H5B2hdr.c:499:38 [opt]
    frame #6: 0x0000000100143bb8 libhdf5.310.dylib`H5B2_open(f=0x0000000100c3c490, addr=744139, ctx_udata=0x0000000000000000) at H5B2.c:199:24 [opt]
    frame #7: 0x0000000100136234 libhdf5.310.dylib`H5A__get_ainfo(f=0x0000000100c3c490, oh=0x0000000100d05760, ainfo=0x000000016fdfed78) at H5Aint.c:1979:41 [opt]
    frame #8: 0x00000001002d5284 libhdf5.310.dylib`H5O__attr_open_by_name(loc=0x000000016fdfedd0, name="scale_factor") at H5Oattribute.c:455:13 [opt]
    frame #9: 0x0000000100133974 libhdf5.310.dylib`H5A__open_by_name(loc=0x000000016fdfee50, obj_name="/Temp", attr_name="scale_factor") at H5Aint.c:637:25 [opt]
    frame #10: 0x000000010045e038 libhdf5.310.dylib`H5VL__native_attr_open(obj=0x0000000100c3c490, loc_params=0x000000016fdfef98, attr_name="scale_factor", aapl_id=792633534417207311, dxpl_id=<unavailable>, req=<unavailable>) at H5VLnative_attr.c:168:29 [opt]
    frame #11: 0x0000000100441954 libhdf5.310.dylib`H5VL__attr_open(obj=0x0000000100c3c490, loc_params=0x000000016fdfef98, cls=0x0000000100c10250, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1104:30 [opt]
    frame #12: 0x00000001004417a4 libhdf5.310.dylib`H5VL_attr_open(vol_obj=0x0000000100c3d620, loc_params=0x000000016fdfef98, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1136:30 [opt]
    frame #13: 0x000000010012b26c libhdf5.310.dylib`H5A__open_common(vol_obj=0x0000000100c3d620, loc_params=0x000000016fdfef98, attr_name="scale_factor", aapl_id=792633534417207311, token_ptr=0x0000000000000000) at H5A.c:463:17 [opt]
    frame #14: 0x00000001001235e4 libhdf5.310.dylib`H5A__open_by_name_api_common(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=<unavailable>, lapl_id=0, token_ptr=0x0000000000000000, _vol_obj_ptr=<unavailable>) at H5A.c:640:22 [opt]
    frame #15: 0x0000000100123344 libhdf5.310.dylib`H5Aopen_by_name(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=0, lapl_id=0) at H5A.c:678:14 [opt]
    frame #16: 0x0000000100003a14 run_hdf`main at run_hdf5.c:23:20
    frame #17: 0x0000000192ba8420 libdyld.dylib`start + 4

ksharonin commented 7 months ago

H5G based record parsing

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x0000000100255f08 libhdf5.310.dylib`H5G__dense_btree2_name_decode(raw="\x9c\U0000000eE\x88", _nrecord=0x0000000100c043d0, ctx=<unavailable>) at H5Gbtree2.c:293:5 [opt]
    frame #1: 0x0000000100148848 libhdf5.310.dylib`H5B2__cache_leaf_deserialize(_image=0x0000000100c04080, len=512, _udata=0x000000016fdfe4d0, dirty=<unavailable>) at H5B2cache.c:1042:13 [opt]
    frame #2: 0x0000000100167d80 libhdf5.310.dylib`H5C__load_entry(f=0x0000000100a3e740, type=0x00000001005adbc0, addr=779578, udata=0x000000016fdfe4d0) at H5Centry.c:1203:26 [opt]
    frame #3: 0x0000000100164d78 libhdf5.310.dylib`H5C_protect(f=0x0000000100a3e740, type=0x00000001005adbc0, addr=779578, udata=0x000000016fdfe4d0, flags=512) at H5Centry.c:3120:30 [opt]
    frame #4: 0x000000010013b19c libhdf5.310.dylib`H5AC_protect(f=0x0000000100a3e740, type=0x00000001005adbc0, addr=779578, udata=0x000000016fdfe4d0, flags=512) at H5AC.c:1276:26 [opt]
    frame #5: 0x00000001001545d4 libhdf5.310.dylib`H5B2__protect_leaf(hdr=0x0000000100a40340, parent=0x0000000000000000, node_ptr=0x000000016fdfe560, shadow=false, flags=512) at H5B2leaf.c:187:40 [opt]
    frame #6: 0x0000000100144994 libhdf5.310.dylib`H5B2_find(bt2=<unavailable>, udata=0x000000016fdfe5e8, found=0x000000016fdfe87b, op=0x0000000000000000, op_data=0x0000000000000000) at H5B2.c:594:29 [opt]
    frame #7: 0x0000000100258a88 libhdf5.310.dylib`H5G__dense_lookup(f=0x0000000100a3e740, linfo=<unavailable>, name="Temp", found=0x000000016fdfe87b, lnk=0x000000016fdfe880) at H5Gdense.c:510:9 [opt]
    frame #8: 0x000000010026d950 libhdf5.310.dylib`H5G__obj_lookup(grp_oloc=0x000000016fdfe920, name="Temp", found=0x000000016fdfe87b, lnk=0x000000016fdfe880) at H5Gobj.c:1059:17 [opt]
    frame #9: 0x0000000100274a08 libhdf5.310.dylib`H5G__traverse_real(_loc=<unavailable>, name="Temp", target=0, op=(libhdf5.310.dylib`H5G__loc_find_cb at H5Gloc.c:373), op_data=0x000000016fdfee18) at H5Gtraverse.c:550:13 [opt]
    frame #10: 0x00000001002745f8 libhdf5.310.dylib`H5G_traverse(loc=0x000000016fdfeeb0, name="/Temp", target=0, op=(libhdf5.310.dylib`H5G__loc_find_cb at H5Gloc.c:373), op_data=0x000000016fdfee18) at H5Gtraverse.c:815:9 [opt]
    frame #11: 0x00000001002652b8 libhdf5.310.dylib`H5G_loc_find(loc=0x000000016fdfeeb0, name="/Temp", obj_loc=0x000000016fdfee70) at H5Gloc.c:420:9 [opt]
    frame #12: 0x00000001001b3a90 libhdf5.310.dylib`H5D__open_name(loc=0x000000016fdfeeb0, name="/Temp", dapl_id=792633534417207303) at H5Dint.c:1417:9 [opt]
    frame #13: 0x000000010045fb90 libhdf5.310.dylib`H5VL__native_dataset_open(obj=0x0000000100a3e740, loc_params=0x000000016fdfefa8, name="/Temp", dapl_id=792633534417207303, dxpl_id=<unavailable>, req=<unavailable>) at H5VLnative_dataset.c:330:25 [opt]
    frame #14: 0x00000001004445f0 libhdf5.310.dylib`H5VL__dataset_open(obj=0x0000000100a3e740, loc_params=0x000000016fdfefa8, cls=0x0000000100a12500, name="/Temp", dapl_id=792633534417207303, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1946:30 [opt]
    frame #15: 0x0000000100444440 libhdf5.310.dylib`H5VL_dataset_open(vol_obj=0x0000000100a3f8d0, loc_params=0x000000016fdfefa8, name="/Temp", dapl_id=792633534417207303, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1978:30 [opt]
    frame #16: 0x0000000100182a74 libhdf5.310.dylib`H5D__open_api_common(loc_id=72057594037927936, name="/Temp", dapl_id=<unavailable>, token_ptr=0x0000000000000000, _vol_obj_ptr=<unavailable>) at H5D.c:362:25 [opt]
    frame #17: 0x0000000100182848 libhdf5.310.dylib`H5Dopen2(loc_id=72057594037927936, name="/Temp", dapl_id=0) at H5D.c:402:22 [opt]
    frame #18: 0x00000001000039f8 run_hdf`main at run_hdf5.c:22:18
    frame #19: 0x0000000192ba8420 libdyld.dylib`start + 4

VS. H5A based decode

(lldb) bt
* thread #1, queue = 'com.apple.main-thread', stop reason = breakpoint 1.1
  * frame #0: 0x000000010012b96c libhdf5.310.dylib`H5A__dense_btree2_name_decode(raw="", _nrecord=0x0000000100a45b88, ctx=<unavailable>) at H5Abtree2.c:321:22 [opt]
    frame #1: 0x0000000100148848 libhdf5.310.dylib`H5B2__cache_leaf_deserialize(_image=0x0000000100a45830, len=512, _udata=0x000000016fdfeb70, dirty=<unavailable>) at H5B2cache.c:1042:13 [opt]
    frame #2: 0x0000000100167d80 libhdf5.310.dylib`H5C__load_entry(f=0x0000000100a3e800, type=0x00000001005adbc0, addr=745745, udata=0x000000016fdfeb70) at H5Centry.c:1203:26 [opt]
    frame #3: 0x0000000100164d78 libhdf5.310.dylib`H5C_protect(f=0x0000000100a3e800, type=0x00000001005adbc0, addr=745745, udata=0x000000016fdfeb70, flags=512) at H5Centry.c:3120:30 [opt]
    frame #4: 0x000000010013b19c libhdf5.310.dylib`H5AC_protect(f=0x0000000100a3e800, type=0x00000001005adbc0, addr=745745, udata=0x000000016fdfeb70, flags=512) at H5AC.c:1276:26 [opt]
    frame #5: 0x00000001001545d4 libhdf5.310.dylib`H5B2__protect_leaf(hdr=0x0000000100a44b60, parent=0x0000000000000000, node_ptr=0x000000016fdfec00, shadow=false, flags=512) at H5B2leaf.c:187:40 [opt]
    frame #6: 0x0000000100144994 libhdf5.310.dylib`H5B2_find(bt2=<unavailable>, udata=0x000000016fdfec90, found=0x000000016fdfec8f, op=0x0000000000000000, op_data=0x0000000000000000) at H5B2.c:594:29 [opt]
    frame #7: 0x000000010012c6f8 libhdf5.310.dylib`H5A__dense_open(f=0x0000000100a3e800, ainfo=<unavailable>, name="scale_factor") at H5Adense.c:380:9 [opt]
    frame #8: 0x00000001002d53b0 libhdf5.310.dylib`H5O__attr_open_by_name(loc=0x000000016fdfeda0, name="scale_factor") at H5Oattribute.c:472:40 [opt]
    frame #9: 0x0000000100133974 libhdf5.310.dylib`H5A__open_by_name(loc=0x000000016fdfee20, obj_name="/Temp", attr_name="scale_factor") at H5Aint.c:637:25 [opt]
    frame #10: 0x000000010045e038 libhdf5.310.dylib`H5VL__native_attr_open(obj=0x0000000100a3e800, loc_params=0x000000016fdfef68, attr_name="scale_factor", aapl_id=792633534417207311, dxpl_id=<unavailable>, req=<unavailable>) at H5VLnative_attr.c:168:29 [opt]
    frame #11: 0x0000000100441954 libhdf5.310.dylib`H5VL__attr_open(obj=0x0000000100a3e800, loc_params=0x000000016fdfef68, cls=0x0000000100a125c0, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1104:30 [opt]
    frame #12: 0x00000001004417a4 libhdf5.310.dylib`H5VL_attr_open(vol_obj=0x0000000100a3f990, loc_params=0x000000016fdfef68, name="scale_factor", aapl_id=792633534417207311, dxpl_id=792633534417207304, req=0x0000000000000000) at H5VLcallback.c:1136:30 [opt]
    frame #13: 0x000000010012b26c libhdf5.310.dylib`H5A__open_common(vol_obj=0x0000000100a3f990, loc_params=0x000000016fdfef68, attr_name="scale_factor", aapl_id=792633534417207311, token_ptr=0x0000000000000000) at H5A.c:463:17 [opt]
    frame #14: 0x00000001001235e4 libhdf5.310.dylib`H5A__open_by_name_api_common(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=<unavailable>, lapl_id=0, token_ptr=0x0000000000000000, _vol_obj_ptr=<unavailable>) at H5A.c:640:22 [opt]
    frame #15: 0x0000000100123344 libhdf5.310.dylib`H5Aopen_by_name(loc_id=72057594037927936, obj_name="/Temp", attr_name="scale_factor", aapl_id=0, lapl_id=0) at H5A.c:678:14 [opt]
    frame #16: 0x0000000100003a14 run_hdf`main at run_hdf5.c:23:20
    frame #17: 0x0000000192ba8420 libdyld.dylib`start + 4

ksharonin commented 6 months ago

 H5FileBuffer::readAttributeMsg (this=0xffff439fa030, pos=775649, hdr_flags=45 '-', dlvl=1, 
    size=103) at /host/packages/h5/H5Coro.cpp:3038
#1  0x0000aaaabcce1834 in H5FileBuffer::readMessage (this=0xffff439fa030, 
    msg_type=H5FileBuffer::ATTRIBUTE_MSG, size=103, pos=775628, hdr_flags=45 '-', dlvl=1)
    at /host/packages/h5/H5Coro.cpp:2143
#2  0x0000aaaabcce07cc in H5FileBuffer::readMessages (this=0xffff439fa030, pos=775628, end=776106, 
    hdr_flags=45 '-', dlvl=1) at /host/packages/h5/H5Coro.cpp:1959
--Type <RET> for more, q to quit, c to continue without paging--
#3  0x0000aaaabcce0430 in H5FileBuffer::readObjHdr (this=0xffff439fa030, pos=775227, dlvl=1)
    at /host/packages/h5/H5Coro.cpp:1928
#4  0x0000aaaabcce4028 in H5FileBuffer::readLinkMsg (this=0xffff439fa030, pos=772220, 
    hdr_flags=44 ',', dlvl=0) at /host/packages/h5/H5Coro.cpp:2688
#5  0x0000aaaabcce1758 in H5FileBuffer::readMessage (this=0xffff439fa030, 
    msg_type=H5FileBuffer::LINK_MSG, size=311, pos=772184, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:2139
#6  0x0000aaaabccdc73c in H5FileBuffer::readDirectBlock (this=0xffff439fa030, 
    heap_info=0xffff439f9a98, block_size=512, pos=772184, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:1332
#7  0x0000aaaabccdd378 in H5FileBuffer::readIndirectBlock (this=0xffff439fa030, 
    heap_info=0xffff439f9a98, block_size=0, pos=785066, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:1464
#8  0x0000aaaabccdbd48 in H5FileBuffer::readFractalHeap (this=0xffff439fa030, 
    msg_type=H5FileBuffer::LINK_MSG, pos=336217, hdr_flags=44 ',', dlvl=0, heap_info_ptr=0x0)
    at /host/packages/h5/H5Coro.cpp:1243

With functionality, need to pass the following arguments for completion: uint8_t hdr_flags, int dlvl, uint64_t size

ksharonin commented 6 months ago

Fractal heap is summoned by the attribute info message, which has the following info

hdr_flags
dlvl
MISSING: message size (extracted from object header) <-- need to find where in dblock implementation this is grabbed out (SOL: use read out obj_len size_t to call down)

readMessage called with size that is calc'd in msg_size of readMessages

ksharonin commented 6 months ago

Improper decoding likely messing with the result

(gdb) p attr_name
$3 = "add_offset\000@\377\377\000\000\020\220\273M\377\377\000\000&\027\f", '\000' <repeats 13 times>, 
"\210\220\337N\377\377\000\000`\264\001@\377\377\000\000\020?\000@\377\377\000\000\000\264\361&w\265\314\336\060\220\337N\377\377\000\000\
360Il\304\252\252\000\000\210\220\337N\377\377\000\000\005\000\000\000\000\000\000\000\320\220\337N\377\377\000\000\060\240\337N\
377\377\000\000,\034\000\000\000\000\000"

VS. Functioning example

(gdb) p attr_name
$2 = "cell_methods\000\377\377\377\000\264\361&w\265\314\001\322\236\v\000\000\000\000\000%%%%%%%%", ' ' <repeats 16 times>, 
"\322\236\v\000\000\000\000\000%02X", '\000' <repeats 12 times>, "\020\220\273M\377\377\000\000&\027\f", '\000' <repeats 32 times>, 
"@\001\004\020@"

ksharonin commented 6 months ago

If mlog artifically surpassed, then this is hit in readDatatypeMsg

2336        mlog(CRITICAL, "datamessage databits: %lld", (long long) databits);
(gdb) p databits
$2 = 7968
(gdb) n
malloc(): corrupted top size

ksharonin commented 6 months ago

Now track back info stuffing

#0  H5FileBuffer::readAttributeMsg (this=0xffff51ffa030, pos=775649, hdr_flags=45 '-', dlvl=1, 
    size=103) at /host/packages/h5/H5Coro.cpp:3040
#1  0x0000aaaae267d848 in H5FileBuffer::readMessage (this=0xffff51ffa030, 
    msg_type=H5FileBuffer::ATTRIBUTE_MSG, size=103, pos=775628, hdr_flags=45 '-', dlvl=1)
    at /host/packages/h5/H5Coro.cpp:2145
#2  0x0000aaaae267c7e0 in H5FileBuffer::readMessages (this=0xffff51ffa030, pos=775628, end=776106, 
    hdr_flags=45 '-', dlvl=1) at /host/packages/h5/H5Coro.cpp:1961
#3  0x0000aaaae267c444 in H5FileBuffer::readObjHdr (this=0xffff51ffa030, pos=775227, dlvl=1)
#4  0x0000aaaae2680000 in H5FileBuffer::readLinkMsg (this=0xffff51ffa030, pos=772220, 
    hdr_flags=44 ',', dlvl=0) at /host/packages/h5/H5Coro.cpp:2690
#5  0x0000aaaae267d76c in H5FileBuffer::readMessage (this=0xffff51ffa030, 
    msg_type=H5FileBuffer::LINK_MSG, size=311, pos=772184, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:2141
#6  0x0000aaaae2678750 in H5FileBuffer::readDirectBlock (this=0xffff51ffa030, 
    heap_info=0xffff51ff9a98, block_size=512, pos=772184, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:1334
#7  0x0000aaaae267938c in H5FileBuffer::readIndirectBlock (this=0xffff51ffa030, 
    heap_info=0xffff51ff9a98, block_size=0, pos=785066, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:1466
#8  0x0000aaaae2677d5c in H5FileBuffer::readFractalHeap (this=0xffff51ffa030, 
    msg_type=H5FileBuffer::LINK_MSG, pos=336217, hdr_flags=44 ',', dlvl=0, heap_info_ptr=0x0)
    at /host/packages/h5/H5Coro.cpp:1245
#9  0x0000aaaae267e474 in H5FileBuffer::readLinkInfoMsg (this=0xffff51ffa030, pos=111, 
    hdr_flags=44 ',', dlvl=0) at /host/packages/h5/H5Coro.cpp:2313
#10 0x0000aaaae267d694 in H5FileBuffer::readMessage (this=0xffff51ffa030, 
    msg_type=H5FileBuffer::LINK_INFO_MSG, size=34, pos=77, hdr_flags=44 ',', dlvl=0)
    at /host/packages/h5/H5Coro.cpp:2138
#11 0x0000aaaae267c7e0 in H5FileBuffer::readMessages (this=0xffff51ffa030, pos=77, end=251, 
    hdr_flags=44 ',', dlvl=0) at /host/packages/h5/H5Coro.cpp:1961
#12 0x0000aaaae267c444 in H5FileBuffer::readObjHdr (this=0xffff51ffa030, pos=71, dlvl=0)
    at /host/packages/h5/H5Coro.cpp:1930
#13 0x0000aaaae2672198 in H5FileBuffer::H5FileBuffer (this=0xffff51ffa030, info=0xffff51ffa2a0, 
    context=0x0, asset=0xffff44008590, 
    resource=0xffff440094e0 "OR_ABI-L2-FDCC-M3_G17_s20182390052191_e20182390054564_c20182390055159.nc", dataset=0xffff44016ba0 "/minimum_fire_area/coordinates", startrow=0, numrows=-1, _meta_only=false)
    at /host/packages/h5/H5Coro.cpp:359
#14 0x0000aaaae268b66c in H5Coro::read (asset=0xffff44008590, 
    resource=0xffff440094e0 "OR_ABI-L2-FDCC-M3_G17_s20182390052191_e20182390054564_c20182390055159.nc", datasetname=0xffff44016ba0 "/minimum_fire_area/coordinates", valtype=RecordObject::INTEGER, col=0, 
#15 0x0000aaaae269495c in H5DatasetDevice::H5DatasetDevice (this=0xffff4400b0d0, L=0xaaab0422ec68, 
    _role=DeviceObject::READER, _asset=0xffff44008590, 
    _resource=0xffff440033f8 "OR_ABI-L2-FDCC-M3_G17_s20182390052191_e20182390054564_c20182390055159.nc", dataset_name=0xffff44003528 "/minimum_fire_area/coordinates", id=0, raw_mode=true, 
    datatype=RecordObject::INTEGER, col=0, startrow=0, numrows=-1)
    at /host/packages/h5/H5DatasetDevice.cpp:137
#16 0x0000aaaae2694118 in H5DatasetDevice::luaCreate (L=0xaaab0422ec68)
    at /host/packages/h5/H5DatasetDevice.cpp:83
#17 0x0000ffff9bc01940 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#18 0x0000ffff9bc0ee28 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#19 0x0000ffff9bc01d54 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#20 0x0000ffff9bc01d98 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#21 0x0000ffff9bc0112c in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#22 0x0000ffff9bc02028 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#23 0x0000ffff9bbfd43c in lua_pcallk () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#24 0x0000aaaae24c32c8 in LuaEngine::docall (this=0xaaab03f25c60, narg=0, nres=-1)
    at /host/packages/core/LuaEngine.cpp:817
#25 0x0000aaaae24c47c8 in LuaEngine::handlescript (this=0xaaab03f25c60, 
    fname=0xaaab04236620 "scripts/selftests/h5coro.lua") at /host/packages/core/LuaEngine.cpp:1014
#26 0x0000aaaae24c5238 in LuaEngine::pmain (L=0xaaab0422ec68) at /host/packages/core/LuaEngine.cpp:1130
#27 0x0000ffff9bc01940 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#28 0x0000ffff9bc01d18 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#29 0x0000ffff9bc01d98 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#30 0x0000ffff9bc0112c in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#31 0x0000ffff9bc02028 in ?? () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#32 0x0000ffff9bbfd43c in lua_pcallk () from /lib/aarch64-linux-gnu/liblua5.3.so.0
#33 0x0000aaaae24c1ccc in LuaEngine::protectedThread (parm=0xaaab0422ac20)
    at /host/packages/core/LuaEngine.cpp:555
#34 0x0000ffff9bc5c624 in start_thread (arg=0xaaaae24c1b94 <LuaEngine::protectedThread(void*)>)
    at pthread_create.c:477
#35 0x0000ffff98b4062c in thread_start () at ../sysdeps/unix/sysv/linux/aarch64/clone.S:78

ksharonin / kerchunkC

V2 B-Trees (ASCII Signature BTHD) for Record type 8 (CURR: REFACTORING) #4