DNAS is crashing

[jwbensley]

$ docker-compose logs --tail=100 -f

dnas_parser    | 2023-03-20 01:28:46,441|INFO|Checking file /opt/dnas_data/downloads/RCC24/updates.20230320.0015.gz
dnas_parser    | 2023-03-20 01:28:46,445|INFO|Processing /opt/dnas_data/downloads/RCC24/updates.20230320.0015.gz...
dnas_parser    | RPython traceback:
dnas_parser    |   File "pypy_module_pypyjit.c", line 446, in jump_absolute__AccessDirect_None
dnas_parser    |   File "rpython_jit_metainterp.c", line 39790, in bound_reached__star_5_1
dnas_parser    |   File "rpython_jit_metainterp_1.c", line 3680, in compile_and_run_once___rpython_jit_metainterp_ji_7
dnas_parser    |   File "rpython_jit_metainterp.c", line 25799, in MetaInterp__compile_and_run_once
dnas_parser    |   File "rpython_jit_metainterp.c", line 43307, in MetaInterp_interpret
dnas_parser    |   File "rpython_jit_metainterp_1.c", line 52730, in MetaInterp__interpret
dnas_parser    |   File "rpython_jit_metainterp_3.c", line 19827, in MIFrame_run_one_step
dnas_parser    |   File "rpython_jit_metainterp_5.c", line 22719, in MIFrame_opimpl_jit_merge_point
dnas_parser    |   File "rpython_jit_metainterp_6.c", line 18216, in MetaInterp_reached_loop_header
dnas_parser    |   File "rpython_jit_metainterp_7.c", line 6462, in MetaInterp_compile_trace
dnas_parser    |   File "rpython_jit_metainterp_7.c", line 20996, in compile_trace
dnas_parser    |   File "rpython_jit_metainterp_7.c", line 61163, in ResumeFromInterpDescr_compile_and_attach
dnas_parser    |   File "rpython_jit_metainterp_8.c", line 12188, in send_loop_to_backend
dnas_parser    |   File "rpython_jit_backend_x86.c", line 1135, in Assembler386_assemble_loop
dnas_parser    |   File "rpython_jit_backend_llsupport.c", line 3417, in MachineDataBlockWrapper_malloc_aligned
dnas_parser    |   File "rpython_jit_backend_llsupport.c", line 5854, in MachineDataBlockWrapper__allocate_next_block
dnas_parser    |   File "rpython_jit_backend_llsupport.c", line 11765, in AsmMemoryManager__allocate_block
dnas_parser    | ~~~ Crash in JIT! <KeyError object at 0x7f8dceb11d28>

[jwbensley]

Again here, somehow the parser has crashed which then caused multi-processing to shit the bed:

ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/multiprocessing/pool.py", line 48, in mapstar
ESC[33mdnas_parser    |ESC[0m     return list(map(*args))
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/../dnas/mrt_parser.py", line 151, in parse_upd_dump
ESC[33mdnas_parser    |ESC[0m     for idx, mrt_e in enumerate(mrt_entries):
ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/site-packages/mrtparse/__init__.py", line 86, in __next__
ESC[33mdnas_parser    |ESC[0m     self.unpack_hdr(mrt)
ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/site-packages/mrtparse/__init__.py", line 116, in unpack_hdr
ESC[33mdnas_parser    |ESC[0m     mrt.unpack()
ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/site-packages/mrtparse/__init__.py", line 228, in unpack
ESC[33mdnas_parser    |ESC[0m     self.data['subtype'] = {st: MRT_ST[t][st]}
ESC[33mdnas_parser    |ESC[0m KeyError: 0
ESC[33mdnas_parser    |ESC[0m """
ESC[33mdnas_parser    |ESC[0m 
ESC[33mdnas_parser    |ESC[0m The above exception was the direct cause of the following exception:
ESC[33mdnas_parser    |ESC[0m 
ESC[33mdnas_parser    |ESC[0m Traceback (most recent call last):
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/parse_mrts.py", line 599, in <module>
ESC[33mdnas_parser    |ESC[0m     main()
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/parse_mrts.py", line 581, in main
ESC[33mdnas_parser    |ESC[0m     continuous(args)
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/parse_mrts.py", line 99, in continuous
ESC[33mdnas_parser    |ESC[0m     parse_files(filelist=filelist, args=args)
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/parse_mrts.py", line 330, in parse_files
ESC[33mdnas_parser    |ESC[0m     mrt_s = parse_file(filename=file, multi=args["multi"])
ESC[33mdnas_parser    |ESC[0m   File "/opt/dnas/scripts/parse_mrts.py", line 268, in parse_file
ESC[33mdnas_parser    |ESC[0m     mrt_chunks = Pool.map(mrt_parser.parse_upd_dump, file_chunks)
ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/multiprocessing/pool.py", line 364, in map
ESC[33mdnas_parser    |ESC[0m     return self._map_async(func, iterable, mapstar, chunksize).get()
ESC[33mdnas_parser    |ESC[0m   File "/opt/pypy3.8-v7.3.7-linux64/lib/pypy3.8/multiprocessing/pool.py", line 771, in get
ESC[33mdnas_parser    |ESC[0m     raise self._value
ESC[33mdnas_parser    |ESC[0m KeyError: 0

[jwbensley]

This is related to issue #168

[jwbensley]

docker-compose should be set to restart "always"

[jwbensley]

DNAS crashed again shortly after 2023-07-26 16:09:00 but no logs are available.

[jwbensley]

Now that the out of memory issues are fixed, this hasn't occurred again so closing.