search

BreezeWhite / oemer

End-to-end Optical Music Recognition (OMR) system. Transcribe phone-taken music sheet image into MusicXML, which can be edited and converted to MIDI.
https://breezewhite.github.io/oemer/
MIT License
386 stars 46 forks source link

Add support for GIF #48

Closed mm3509 closed 11 months ago

mm3509 commented 11 months ago

Describe the bug

I install the package with Conda, Python 3.8 and PIP. I run it on a GIF image and get this error:

Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/oemer/bin/oemer", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 280, in main
    mxl_path = extract(args)
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 131, in extract
    staff, symbols, stems_rests, notehead, clefs_keys = generate_pred(str(img_path), use_tf=args.use_tf)
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 51, in generate_pred
    staff_symbols_map, _ = inference(
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/inference.py", line 60, in inference
    image = Image.fromarray(image).convert("RGB")
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/PIL/Image.py", line 3084, in fromarray
    arr = obj.__array_interface__
AttributeError: 'NoneType' object has no attribute '__array_interface__'

I get the same error using oemer[tf].

Input Image

I attach the PNG and the GIF version. Only the GIF version causes the error. ss015 ss015

Full Traceback

2023-11-17 10:45:48 Extracting staffline and symbols
/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/onnxruntime/capi/onnxruntime_inference_collection.py:69: UserWarning: Specified provider 'CoreMLExecutionProvider' is not in available provider names.Available providers: 'TensorrtExecutionProvider, CUDAExecutionProvider, AzureExecutionProvider, CPUExecutionProvider'
  warnings.warn(
2023-11-17 10:45:50.437510351 [E:onnxruntime:Default, provider_bridge_ort.cc:1480 TryGetProviderInfo_CUDA] /onnxruntime_src/onnxruntime/core/session/provider_bridge_ort.cc:1193 onnxruntime::Provider& onnxruntime::ProviderLibrary::Get() [ONNXRuntimeError] : 1 : FAIL : Failed to load library libonnxruntime_providers_cuda.so with error: libcublasLt.so.11: cannot open shared object file: No such file or directory

2023-11-17 10:45:50.437708300 [W:onnxruntime:Default, onnxruntime_pybind_state.cc:747 CreateExecutionProviderInstance] Failed to create CUDAExecutionProvider. Please reference https://onnxruntime.ai/docs/execution-providers/CUDA-ExecutionProvider.html#requirements to ensure all dependencies are met.
Traceback (most recent call last):
  File "/home/ubuntu/miniconda3/envs/oemer/bin/oemer", line 8, in <module>
    sys.exit(main())
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 280, in main
    mxl_path = extract(args)
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 131, in extract
    staff, symbols, stems_rests, notehead, clefs_keys = generate_pred(str(img_path), use_tf=args.use_tf)
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/ete.py", line 51, in generate_pred
    staff_symbols_map, _ = inference(
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/oemer/inference.py", line 60, in inference
    image = Image.fromarray(image).convert("RGB")
  File "/home/ubuntu/miniconda3/envs/oemer/lib/python3.8/site-packages/PIL/Image.py", line 3084, in fromarray
    arr = obj.__array_interface__
AttributeError: 'NoneType' object has no attribute '__array_interface__'

Command You Execute

$ oemer ss015.gif
BreezeWhite commented 11 months ago

For the scenario of OMR tasks, I think the support of GIF format is a little bit weird, since such tasks always expect input format like JPG or PNG, not GIF. The best way is convert the GIF image into a static image format as mentioned before. Closing this as the stated reason above.

mm3509 commented 11 months ago

That's fair and thanks for the quick reply. How about just improving the error message, such as GIF images not supported, convert first to PNG or JPG?

BreezeWhite commented 11 months ago

That's a reasonable modification. If you have time on this issue, I'll be glad to assist you on the code refactoring.