Check for UTF encodings and transcode them into UTF-32 to count at least codepoints (and then visible symbols) as line lengths instead of just bytes. The following algorithm is to be used:
Check BOM, if has UTF-16LE/BE, UTF-8 BOM then try to transcode correspondingly.
If there are interleaving NULs then try transcode as UTF-16LE/BE accordingly.
If none of the above try to transcode as UTF-8.
If transcoding failed then if it has interleaving NULs report this file as possibly binary one and do not add it to the statistics.
If transcoding failed and there is no NULs interpret this file as native 1-byte ASCII-compatible encoding (e.g. Latin-1).
Problem: codecvt is deprecated and is to removed in C++26 and we have no standard facility to transcode UTF.
Check for UTF encodings and transcode them into UTF-32 to count at least codepoints (and then visible symbols) as line lengths instead of just bytes. The following algorithm is to be used:
Problem: codecvt is deprecated and is to removed in C++26 and we have no standard facility to transcode UTF.