delete sample from database: segmentation fault on CLI

[nextgenusfs]

It seems there is a way to delete samples from a uri/database via the CLI, ie tiledbvcf delete. I'm getting an error when I try this and need help understanding why this is happening.

$ tiledbvcf delete -u s3://path/to/db -s awesome_sample
[2024-03-27 19:11:12.150] [tiledb-vcf] [Process: 24392] [Thread: 24392] [critical] Exception: SubarrayPartitioner: Trying to partition a unary range because of memory budget, this will cause the query to run very slow. Increase `sm.memory_budget` and `sm.memory_budget_var` through the configuration settings to avoid this issue. To override and run the query with the same budget, set `sm.skip_unary_partitioning_budget_check` to `true`.
Segmentation fault

## version
$ tiledbvcf --version
TileDB-VCF version 0234b10-modified
TileDB version 2.18.3
htslib version 1.19

And then it seems like there is no way to delete samples from the python API?

>>> import tiledbvcf
>>> tiledbvcf.version
'0.27.1'
>>> dir(tiledbvcf.Dataset)
['__class__', '__delattr__', '__dict__', '__dir__', '__doc__', '__eq__', '__format__', '__ge__', '__getattribute__', '__gt__', '__hash__', '__init__', '__init_subclass__', '__le__', '__lt__', '__module__', '__ne__', '__new__', '__reduce__', '__reduce_ex__', '__repr__', '__setattr__', '__sizeof__', '__str__', '__subclasshook__', '__weakref__', '_set_read_cfg', '_set_samples', '_set_write_cfg', 'attributes', 'continue_read', 'continue_read_arrow', 'count', 'create_dataset', 'export', 'ingest_samples', 'read', 'read_arrow', 'read_completed', 'read_iter', 'read_variant_stats', 'sample_count', 'samples', 'schema_version', 'tiledb_stats', 'version']

[nextgenusfs]

Follow up, I upgraded to newer version (on conda) and same error with a stack trace:

$ tiledbvcf --version
TileDB-VCF version 2ec4de4-modified
TileDB version 2.20.1
htslib version 1.19

$ tiledbvcf delete -u s3://path/to/db -s awesome_sample
[2024-03-27 19:46:41.319] [tiledb-vcf] [Process: 28631] [Thread: 28631] [critical] Exception: SubarrayPartitioner: Trying to partition a unary range because of memory budget, this will cause the query to run very slow. Increase `sm.memory_budget` and `sm.memory_budget_var` through the configuration settings to avoid this issue. To override and run the query with the same budget, set `sm.skip_unary_partitioning_budget_check` to `true`.
*** Error in `tiledbvcf': double free or corruption (!prev): 0x000056215a68e910 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x81329)[0x7f2198478329]
/bioinfo/apps/all_apps/miniconda3/envs/vtools/bin/../lib/libtiledb.so.2.20(+0x1b3d7e)[0x7f2198d77d7e]
/bioinfo/apps/all_apps/miniconda3/envs/vtools/bin/../lib/libtiledb.so.2.20(+0x18649d)[0x7f2198d4a49d]
/bioinfo/apps/all_apps/miniconda3/envs/vtools/bin/../lib/libtiledb.so.2.20(+0x189380)[0x7f2198d4d380]
/bioinfo/apps/all_apps/miniconda3/envs/vtools/bin/../lib/libtiledb.so.2.20(tiledb_array_close+0x54)[0x7f2198dd9934]
tiledbvcf(+0xa9162)[0x5621577e4162]
/lib64/libc.so.6(+0x39d39)[0x7f2198430d39]
/lib64/libc.so.6(+0x39d87)[0x7f2198430d87]
tiledbvcf(+0x65c4b)[0x5621577a0c4b]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7f2198419555]
tiledbvcf(+0x93ea5)[0x5621577ceea5]
======= Memory map: ========
56215773b000-56215779e000 r--p 00000000 00:28 2761396322001156603        /all_apps/miniconda3/envs/vtools/bin/tiledbvcf
56215779e000-562157959000 r-xp 00063000 00:28 2761396322001156603        /all_apps/miniconda3/envs/vtools/bin/tiledbvcf
562157959000-5621579bd000 r--p 0021e000 00:28 2761396322001156603        /all_apps/miniconda3/envs/vtools/bin/tiledbvcf
5621579bd000-5621579c4000 r--p 00282000 00:28 2761396322001156603        /all_apps/miniconda3/envs/vtools/bin/tiledbvcf
5621579c4000-5621579c5000 rw-p 00289000 00:28 2761396322001156603        /all_apps/miniconda3/envs/vtools/bin/tiledbvcf
5621579c5000-5621579c6000 rw-p 00000000 00:00 0 
562158834000-56215af75000 rw-p 00000000 00:00 0                          [heap]
7f2114000000-7f2114048000 rw-p 00000000 00:00 0 
7f2114048000-7f2118000000 ---p 00000000 00:00 0 
7f2118000000-7f2118022000 rw-p 00000000 00:00 0 
7f2118022000-7f211c000000 ---p 00000000 00:00 0 
7f211c000000-7f211c021000 rw-p 00000000 00:00 0 
7f211c021000-7f2120000000 ---p 00000000 00:00 0 
7f213dabd000-7f2144000000 r--p 00000000 103:01 28958                     /usr/lib/locale/locale-archive
7f2144000000-7f2144dca000 rw-p 00000000 00:00 0 
7f2144dca000-7f2148000000 ---p 00000000 00:00 0 
7f2148000000-7f2148b88000 rw-p 00000000 00:00 0 
7f2148b88000-7f214c000000 ---p 00000000 00:00 0 
7f214c000000-7f214cc58000 rw-p 00000000 00:00 0 
7f214cc58000-7f2150000000 ---p 00000000 00:00 0 
7f2150000000-7f2150c61000 rw-p 00000000 00:00 0 
7f2150c61000-7f2154000000 ---p 00000000 00:00 0 
7f2154000000-7f215403c000 rw-p 00000000 00:00 0 
7f215403c000-7f2158000000 ---p 00000000 00:00 0 
7f2158000000-7f2158dd3000 rw-p 00000000 00:00 0 
7f2158dd3000-7f215c000000 ---p 00000000 00:00 0 
7f2160000000-7f2160382000 rw-p 00000000 00:00 0 
7f2160382000-7f2164000000 ---p 00000000 00:00 0 
7f2164000000-7f2164021000 rw-p 00000000 00:00 0 
7f2164021000-7f2168000000 ---p 00000000 00:00 0 
7f21687f9000-7f21687fa000 ---p 00000000 00:00 0 
7f21687fa000-7f2168ffa000 rw-p 00000000 00:00 0 
7f2168ffa000-7f2168ffb000 ---p 00000000 00:00 0 
7f2168ffb000-7f21697fb000 rw-p 00000000 00:00 0 
7f21697fb000-7f21697fc000 ---p 00000000 00:00 0 
7f21697fc000-7f2169ffc000 rw-p 00000000 00:00 0 
7f2169ffc000-7f2169ffd000 ---p 00000000 00:00 0 
7f2169ffd000-7f216a7fd000 rw-p 00000000 00:00 0 
7f216a7fd000-7f216a7fe000 ---p 00000000 00:00 0 
7f216a7fe000-7f216affe000 rw-p 00000000 00:00 0 
7f216affe000-7f216afff000 ---p 00000000 00:00 0 
7f216afff000-7f216b7ff000 rw-p 00000000 00:00 0 
7f216b7ff000-7f216b800000 ---p 00000000 00:00 0 
7f216b800000-7f216c000000 rw-p 00000000 00:00 0 
7f216c000000-7f216c027000 rw-p 00000000 00:00 0 
7f216c027000-7f2170000000 ---p 00000000 00:00 0 
7f2170000000-7f2171147000 rw-p 00000000 00:00 0 
7f2171147000-7f2174000000 ---p 00000000 00:00 0 
7f2174000000-7f21750dd000 rw-p 00000000 00:00 0 
7f21750dd000-7f2178000000 ---p 00000000 00:00 0 
7f2178000000-7f21790ec000 rw-p 00000000 00:00 0 
7f21790ec000-7f217c000000 ---p 00000000 00:00 0 
7f217c000000-7f217c23a000 rw-p 00000000 00:00 0 
7f217c23a000-7f2180000000 ---p 00000000 00:00 0 
7f2180000000-7f2180021000 rw-p 00000000 00:00 0 
7f2180021000-7f2184000000 ---p 00000000 00:00 0 
7f21845f2000-7f21845f8000 r-xp 00000000 103:01 28990                     /usr/lib64/libnss_dns-2.17.so
7f21845f8000-7f21847f7000 ---p 00006000 103:01 28990                     /usr/lib64/libnss_dns-2.17.so
7f21847f7000-7f21847f8000 r--p 00005000 103:01 28990                     /usr/lib64/libnss_dns-2.17.so
7f21847f8000-7f21847f9000 rw-p 00006000 103:01 28990                     /usr/lib64/libnss_dns-2.17.so
7f21847f9000-7f21847fa000 ---p 00000000 00:00 0 
7f21847fa000-7f2184ffa000 rw-p 00000000 00:00 0 
7f2184ffa000-7f2184ffb000 ---p 00000000 00:00 0 
7f2184ffb000-7f21857fb000 rw-p 00000000 00:00 0 
7f21857fb000-7f21857fc000 ---p 00000000 00:00 0 
7f21857fc000-7f2185ffc000 rw-p 00000000 00:00 0 
7f2188000000-7f2188021000 rw-p 00000000 00:00 0 
7f2188021000-7f218c000000 ---p 00000000 00:00 0 
7f218c086000-7f218c092000 r-xp 00000000 103:01 108514                    /usr/lib64/libnss_files-2.17.so
7f218c092000-7f218c291000 ---p 0000c000 103:01 108514                    /usr/lib64/libnss_files-2.17.so
7f218c291000-7f218c292000 r--p 0000b000 103:01 108514                    /usr/lib64/libnss_files-2.17.so
7f218c292000-7f218c293000 rw-p 0000c000 103:01 108514                    /usr/lib64/libnss_files-2.17.so
7f218c293000-7f218c299000 rw-p 00000000 00:00 0 
7f218c2a6000-7f218c2a8000 r--p 00000000 00:28 17653856053507271133       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3_write.so
7f218c2a8000-7f218c2ab000 r-xp 00002000 00:28 17653856053507271133       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3_write.so
7f218c2ab000-7f218c2ac000 r--p 00005000 00:28 17653856053507271133       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3_write.so
7f218c2ac000-7f218c2ad000 r--p 00005000 00:28 17653856053507271133       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3_write.so
7f218c2ad000-7f218c2ae000 rw-p 00006000 00:28 17653856053507271133       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3_write.so
7f218c2ae000-7f218c2b0000 r--p 00000000 00:28 10479277734955115640       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3.so
7f218c2b0000-7f218c2b6000 r-xp 00002000 00:28 10479277734955115640       /all_apps/miniconda3/envs/vtools/libexec/htslib/hfile_s3.soAborted

[gspowley]

Thank you @nextgenusfs for reporting the issue. You are correct, sample deletion is currently only supported in the CLI.

To debug further, please add the options below to skip the budget check and enable debug messages.

tiledbvcf delete -u s3://path/to/db -s awesome_sample --tiledb-config sm.skip_unary_partitioning_budget_check=true --log-level debug

Can you share any details about the dataset, like number of samples, average records per sample, and how the samples were ingested?

[nextgenusfs]

It's not an overly large genome (only ~ 35 MB) or anything, although there are 293 samples.

[Mar 27 10:00 PM] Database stats:
  293 samples
  37225 unique variants

So when I run the above it just hangs after about 30 seconds, like it was getting stuck on one of the chunks. I ended up adding the following to config which allowed it to complete shortly, so seems like just a memory issue, was trying to specify 4 GB and 1 GB but seems like these settings do not take "shorthand" ie 4g.

--tiledb-config sm.memory_budget=4000000000 sm.memory_budget_var=1000000000

[gspowley]

Yes, this does look like a memory issue. Thanks for the feedback, we will look into increasing the memory budget by default.

[ihnorton]

@nextgenusfs if possible, could you please share a reproduction for the failure in this comment -- we would like to follow up on the double free if we can get a reproduction.