vesoft-inc / nebula-importer

Nebula Graph Importer with Go
Apache License 2.0
90 stars 60 forks source link

panic when import a large number of files #95

Closed IanSmith123 closed 1 year ago

IanSmith123 commented 3 years ago

question mentioned at https://discuss.nebula-graph.com.cn/t/topic/1579/21?u=les1ie

If I try to import a larger number of files, eg: 1000 csv files, with a yaml file more than 10000 lines, nebula_importer will panic error

How to reproduce:

  1. ganerate sample csv files
    
    from pathlib import Path
    import os

dump_dir = Path('./dump') if not os.path.exists(dump_dir): os.mkdir(dump_dir)

def generate_csv(): num = 10000 for i in range(num): with open(f'{dumpdir}/vertex{i}.csv', 'w') as f: f.write("123\n")

generate_csv()

2. download error config.yaml
[out.zip](https://github.com/vesoft-inc/nebula-importer/files/5458161/out.zip)

3. import csv

python3 reproduce.py cp path_to_nebula_impoerter_exec dump/ cd dump ./nebula_importer -c out.yaml


4. screenshot
![image](https://user-images.githubusercontent.com/19611084/97572662-96bbce00-1a23-11eb-9c80-faf4823c37a3.png)