search

vncorenlp / VnCoreNLP

A Vietnamese natural language processing toolkit (NAACL 2018)
Other
587 stars 145 forks source link

Connection error when using the Python wrapper to process a big text file #19

Closed vncorenlp closed 4 years ago

vncorenlp commented 4 years ago

When processing a big text file (>= 1GB text), there might be a case of connection error that unfortunately stops the program. The following example might be used to fix this error:

for _ in range(5):
    try:      
        # Process the data:
        annotated_text = annotator.annotate(text)   
        word_segmented_text = annotator.tokenize(text)

        break 
    except:
        # Reconnect if there is a connection error:
        print("Retry in 5 seconds")
        time.sleep(5)