search

BreezeWhite / oemer

End-to-end Optical Music Recognition (OMR) system. Transcribe phone-taken music sheet image into MusicXML, which can be edited and converted to MIDI.
https://breezewhite.github.io/oemer/
MIT License
386 stars 46 forks source link

Program fails with "IndexError: index 1633 is out of bounds for axis 1 with size 1633" #7

Closed nn9dev closed 2 years ago

nn9dev commented 2 years ago

I have tried this on both Colab and my local machine, and this error presents itself on both.

C:\Users\nn9dev>oemer C:\Users\nn9dev\Downloads\where_evil_grows_upscaled.png
2022-05-06 22:51:08 Extracting staffline and symbols
c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\onnxruntime\capi\onnxruntime_inference_collection.py:55: UserWarning: Specified provider 'CUDAExecutionProvider' is not in available provider names.Available providers: 'CPUExecutionProvider'
  warnings.warn("Specified provider '{}' is not in available provider names."
1633 2251
2022-05-06 22:55:08 Extracting layers of different symbols
1633 2251
2022-05-06 23:00:18 Dewarping
2022-05-06 23:00:41 Extracting stafflines
c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\numpy\core\fromnumeric.py:3440: RuntimeWarning: Mean of empty slice.
  return _methods._mean(a, axis=axis, dtype=dtype,
c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\numpy\core\_methods.py:189: RuntimeWarning: invalid value encountered in double_scalars
  ret = ret.dtype.type(ret / rcount)
0 203 7
204 407 11
408 611 11
612 815 11
816 1019 11
1020 1223 10
1224 1427 10
2022-05-06 23:00:49 Extracting noteheads
2022-05-06 23:00:49 Analyzing notehead bboxes
2022-05-06 23:00:52 Instanitiating notes
2022-05-06 23:00:57 Grouping noteheads
2022-05-06 23:01:16 Extracting symbols
2022-05-06 23:01:24 Extracting rhythm types
Traceback (most recent call last):
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\runpy.py", line 193, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\runpy.py", line 86, in _run_code
    exec(code, run_globals)
  File "C:\Users\nn9dev\AppData\Local\Programs\Python\Python38\Scripts\oemer.exe\__main__.py", line 7, in <module>
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\oemer\ete.py", line 261, in main
    mxl_path = extract(args)
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\oemer\ete.py", line 188, in extract
    rhythm_extract()
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\oemer\rhythm_extraction.py", line 584, in extract
    parse_dot(max_area_ratio=dot_max_area_ratio, min_area_ratio=dot_min_area_ratio)
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\oemer\rhythm_extraction.py", line 83, in parse_dot
    has_dot = scan_dot(no_stem, note_id_map, bbox, unit_size, min_count, max_count)
  File "c:\users\nn9dev\appdata\local\programs\python\python38\lib\site-packages\oemer\rhythm_extraction.py", line 25, in scan_dot
    cur_scan_line = note_id_map[int(start_y):int(bbox[3]), int(right_bound)]
IndexError: index 1633 is out of bounds for axis 1 with size 1633
1428 1632
C:\Users\nn9dev>

This is the file that I've been using where_evil_grows_upscaled

I did upscale the original using ESRGAN in hopes to get a better result, and that's the file attached above. Using the original file gives a completely different error.

This is the error given using the original file:

env: DEBIAN_FRONTEND=noninteractive
env: QT_QPA_PLATFORM=offscreen
2022-05-07 03:25:25 Extracting staffline and symbols
2022-05-07 03:25:25.532034248 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19277 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.532094938 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19277
2022-05-07 03:25:25.532107152 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: model/conv2d_transpose_1/conv2d_transpose to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.532115172 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: model/conv2d_transpose_1/conv2d_transpose
2022-05-07 03:25:25.532319090 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19295 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.532329781 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19295
2022-05-07 03:25:25.532631272 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19313 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.532641132 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19313
2022-05-07 03:25:25.532822728 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19323 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.532832648 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19323
2022-05-07 03:25:25.533134634 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19341 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.533143895 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19341
2022-05-07 03:25:25.533327263 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19351 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.533337086 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19351
2022-05-07 03:25:25.533634334 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__19365 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:25:25.533643239 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__19365
1633 2251
tcmalloc: large alloc 1073741824 bytes == 0x87f80000 @  0x7fe1bc882b6b 0x7fe1bc8a2379 0x7fe17c87b345 0x7fe17c886785 0x7fe17c886d77 0x7fe17c8a9565 0x7fe17c9454ac 0x7fe17c92c10a 0x7fe17c92e211 0x7fe17c1aa02c 0x7fe17c1aa540 0x7fe17c1aac16 0x7fe17c16cf20 0x7fe17c103922 0x593784 0x594731 0x548cc1 0x51566f 0x549576 0x593fce 0x511e2c 0x549576 0x593fce 0x5118f8 0x593dd7 0x5118f8 0x593dd7 0x5118f8 0x593dd7 0x5118f8 0x549576
2022-05-07 03:26:14 Extracting layers of different symbols
2022-05-07 03:26:14.230867310 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__2063 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:26:14.230907548 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__2063
2022-05-07 03:26:14.232523547 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__2103 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:26:14.232541102 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__2103
2022-05-07 03:26:14.233284465 [W:onnxruntime:Default, cuda_execution_provider.cc:2295 ConvTransposeNeedFallbackToCPU] Dropping the ConvTranspose node: ConvTranspose__2123 to CPU because it requires asymmetric padding which the CUDA EP currently does not support
2022-05-07 03:26:14.233295166 [W:onnxruntime:Default, cuda_execution_provider.cc:2404 GetCapability] CUDA kernel not supported. Fallback to CPU execution provider for Op type: ConvTranspose node name: ConvTranspose__2123
1633 2251
2022-05-07 03:27:11 Dewarping
Traceback (most recent call last):
  File "/usr/local/bin/oemer", line 8, in <module>
    sys.exit(main())
  File "/usr/local/lib/python3.7/dist-packages/oemer/ete.py", line 261, in main
    mxl_path = extract(args)
  File "/usr/local/lib/python3.7/dist-packages/oemer/ete.py", line 136, in extract
    coords_x, coords_y = estimate_coords(staff)
  File "/usr/local/lib/python3.7/dist-packages/oemer/dewarp.py", line 271, in estimate_coords
    new_gg_map = connect_nearby_grid_group(gg_map, grid_groups, grid_map, grids)
  File "/usr/local/lib/python3.7/dist-packages/oemer/dewarp.py", line 186, in connect_nearby_grid_group
    assert grid_id in grid_groups[label].gids, f"{grid_id}, {label}"
AssertionError: -1, -1

And this is the original file: 0023164_where-evil-grows-terry-jacks-recorded-by-the-poppy-family_550

I'm guessing the Assertion Error is from it being too low resolution as you said in another issue, but the first error still confounds me.

BreezeWhite commented 2 years ago

Since oemer is built up with lots of hand-crafted, rule-based algorithms, the score should be as simple and clear as possible. The score you provided has too many disconnections of symbols, as you said this is probably due to the low resolution of the original image. They looks too fuzzy. Second, oemer is developed and tested on two-hand piano scores, which means it may not work on scores having three different stafflines. Third, there are guitar tab on the score, which is not expected by oemer and may thus make wrong predictions.

nn9dev commented 2 years ago

I see. Thank you for the information.