The functions for writing the text representation of DNS records in host2str.c generally use ldns_buffer_printf() to write to the ldns_buffer object. This function resizes the buffer if the existing one does not have enough capacity for the additional data. But in some cases, ldns_buffer_write_u8() is used instead to write a single character to the buffer. This function generates an assert instead of resizing the buffer if there is not enough capacity for the one additional character.
The buffer that is used is initially sized to LDNS_MAX_PACKETLEN (65535 bytes) in most cases, such as with calls to ldns_rr2str_fmt(), which should be sufficient for nearly all records and avoids any potential issues with reaching the capacity limit. But in some pathological cases, the text format can exceed this size and ldns_buffer_write_u8() may be called when at the capacity limit, resulting in an assert. This was only caught with fuzz testing of unexpected wire format DNS data.
A proper fix for this would be to ensure that ldns_buffer_write_u8() checks for the capacity and resizes the buffer as necessary. Alternatively, the buffer could be initialized to an even larger value (maybe 8x of LDNS_MAX_PACKETLEN) that is sufficient for even the largest records. That would work for ldns_rr2str_fmt() being used for a single record, but direct calls to ldns_rr2buffer_str_fmt() for writing multiple records to a single buffer would need a solution that checks for the capacity before writing in all cases.
The functions for writing the text representation of DNS records in host2str.c generally use ldns_buffer_printf() to write to the ldns_buffer object. This function resizes the buffer if the existing one does not have enough capacity for the additional data. But in some cases, ldns_buffer_write_u8() is used instead to write a single character to the buffer. This function generates an assert instead of resizing the buffer if there is not enough capacity for the one additional character.
The buffer that is used is initially sized to LDNS_MAX_PACKETLEN (65535 bytes) in most cases, such as with calls to ldns_rr2str_fmt(), which should be sufficient for nearly all records and avoids any potential issues with reaching the capacity limit. But in some pathological cases, the text format can exceed this size and ldns_buffer_write_u8() may be called when at the capacity limit, resulting in an assert. This was only caught with fuzz testing of unexpected wire format DNS data.
A proper fix for this would be to ensure that ldns_buffer_write_u8() checks for the capacity and resizes the buffer as necessary. Alternatively, the buffer could be initialized to an even larger value (maybe 8x of LDNS_MAX_PACKETLEN) that is sufficient for even the largest records. That would work for ldns_rr2str_fmt() being used for a single record, but direct calls to ldns_rr2buffer_str_fmt() for writing multiple records to a single buffer would need a solution that checks for the capacity before writing in all cases.