search

LiyuanLucasLiu / ArabicNER

Arabic NER system with a strong performance
https://www.aclweb.org/anthology/W19-4607.pdf
Apache License 2.0
34 stars 10 forks source link

test #5

Open zakarianamikaz opened 5 years ago

zakarianamikaz commented 5 years ago

after training i'm getting memory error bash test.sh ./data ./wdata === Data Pre-processing === Traceback (most recent call last): File "pre_process_test.py", line 23, in encoder = strRealTimeEncoderWrapper(args) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 35, in init self.build_pipelines(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 281, in build_pipelines self.pipeline_dict[key] = strWERealTimePipeline(arg['strEncoder'][key]) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 66, in init self.build_pipeline(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 159, in build_pipeline for line in f.readlines(): File "/home/zakaria/anaconda3/envs/py37/lib/python3.7/codecs.py", line 709, in readlines return self.reader.readlines(sizehint) File "/home/zakaria/anaconda3/envs/py37/lib/python3.7/codecs.py", line 618, in readlines data = self.read() File "/home/zakaria/anaconda3/envs/py37/lib/python3.7/codecs.py", line 504, in read newchars, decodedbytes = self.decode(data, self.errors) MemoryError === Model Ensembling and Inferencing === [2019-09-04 13:45:26,511] CPU would be used.
[2019-09-04 13:45:26,520] Loading the data...
[2019-09-04 13:45:26,520] Building lm pipeline...
[2019-09-04 13:45:26,520] File not exist: /home/zakaria/Downloads/ArabicNER/wdata/dev.json Traceback (most recent call last): File "ensemble_ner.py", line 48, in dev_data = strFromFileEncoderWrapper(args, processed_file = conf.input[0]) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 301, in init super(strFromFileEncoderWrapper, self).init(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 35, in init self.build_pipelines(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 309, in build_pipelines self.pipeline_dict[key] = strFromFilePipeline(arg['strEncoder'][key], key) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 252, in init super(strFromFilePipeline, self).init(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 66, in init self.build_pipeline(arg) File "/home/zakaria/Downloads/ArabicNER/abnlp/encoder/str_encoder.py", line 257, in build_pipeline with open(arg['processed_file'], 'r') as fin: FileNotFoundError: [Errno 2] No such file or directory: '/home/zakaria/Downloads/ArabicNER/wdata/dev.json'

If you suspect this is an IPython bug, please report it at: https://github.com/ipython/ipython/issues or send an email to the mailing list at ipython-dev@python.org

You can print a more detailed traceback right now with "%tb", or use "%debug" to interactively debug it.

Extra-detailed tracebacks for bug-reporting purposes can be enabled via: %config Application.verbose_crash=True

=== Dictionary Based Model Inferencing === === Results Merging === Traceback (most recent call last): File "post_process/merge_result.py", line 13, in with codecs.open(args.model_csv, 'r', 'utf-8') as fin: File "/home/zakaria/anaconda3/envs/py37/lib/python3.7/codecs.py", line 904, in open file = builtins.open(filename, mode, buffering) FileNotFoundError: [Errno 2] No such file or directory: '/home/zakaria/Downloads/ArabicNER/wdata/tmp0.csv'

LiyuanLucasLiu commented 5 years ago

It seems that the error occurs when the program trying to loading the file. However, I'm not sure about the exact reason.

zakarianamikaz commented 5 years ago

me too i couldn't find the reason

LiyuanLucasLiu commented 5 years ago

Maybe you can try packages like ipdb or pdb (especially the set_trace method). They are very helpful in debugging programs.

zakarianamikaz commented 5 years ago

it can be a memory or disk capacity problem !