我也来分享一下我的基于语义角色标注（SRL）的文本切分方式

AnonymousmousCoder commented 3 months ago

看到https://github.com/RVC-Boss/GPT-SoVITS/issues/1256 分享的方法后，我也跟大家一起分享一下我的文本切分方法。和其他基于字数和子串的切句方法不同，我所采用的方法更着重于一句话的主谓宾的切分，更具体地说就是根据施事者、受事者等进行句子成分的重组和切分。适用于长难句和大段逗号的情况，效果是提升长句内部的断句（停顿）准确性。


import hanlp
hanlp.pretrained.mtl.ALL
HanLP=hanlp.load(hanlp.pretrained.mtl.CLOSE_TOK_POS_NER_SRL_DEP_SDP_CON_ERNIE_GRAM_ZH)
def amazing_cut(sentence):
    doc = HanLP(sentence,tasks=['tok/coarse','srl'],skip_tasks='tok/fine')
    srls = doc['srl']
    tokens = doc['tok/coarse']
    sent_pos = []
    last_start = 0
    last_end = 0
    temp_srls=[]
    for srl in srls:
        start = srl[0][2]
        end = srl[-1][3]
        if(start>= last_start and end<=last_end):
            continue
        temp_srls.append(srl)
        # print(srl)
        last_start =start
        last_end = end
    useful_srls = []
    for i in range(len(temp_srls)-1,-1,-1):
        srl = temp_srls[i]
        start = srl[0][2]
        end = srl[-1][3]
        if(start>= last_start and end<=last_end):
            continue
        useful_srls.append(srl)
        # print(srl)
        last_start =start
        last_end = end

    for srl in useful_srls:
        sent_pos.append(srl[-1][3])
        sent_pos = list(set(sent_pos))
    sent_pos.sort()
    # print(sent_pos)
    ans_tokens = tokens.copy()
    # 在指定位置给sentence加入换行符
    i = 0
    punctuation = "".join(splits)
    for pos in sent_pos:
        if(len(ans_tokens)> pos+1+i):
            # print(ans_tokens[pos+1+i])
            next_punk_pos = 0
            for x in tokens[pos:]: #最多往后面看4个
                if(x in punctuation) or next_punk_pos>=5:
                    next_punk_pos+=1
                    break
                else:
                    next_punk_pos+=1
            if(next_punk_pos <5 and len(ans_tokens) >pos+i+next_punk_pos):
                ans_tokens.insert(pos+i+next_punk_pos,"\n")  
            else:
                ans_tokens.insert(pos+i,"\n") 
        i+=1
    return "".join(ans_tokens) #因为已经自带换行符了

@register_method("cut1")
def cut1(inp):
    inp = inp.strip("\n")  # 去除字符串开头和结尾的换行符
    inps = split(inp)  # 使用自定义的 split 函数将输入切割成子串列表
    sen_length = len(inps)
    if(sen_length >= 6): #哪些适合基于srl的切分，sen_length参数应视情况调整
        return amazing_cut(inp)
    else:
        opts = []
        gap = 3
        if(sen_length == 4):
            gap = 2
        for i in range(0, sen_length, gap): 
            group = inps[i:i+gap] 
            opts.append("".join(group))  
        return "\n".join(opts)

`

TinaChen95 commented 3 months ago

请问可以给一些切分case吗？

RVC-Boss commented 3 months ago

很有趣同楼上，想看看case

RVC-Boss / GPT-SoVITS

我也来分享一下我的基于语义角色标注（SRL）的文本切分方式 #1272