search

pygfx / pyshader

Write modern GPU shaders in Python!
BSD 2-Clause "Simplified" License
73 stars 1 forks source link

Compute example crashes in create_compute_pipeline #48

Closed pbouda closed 4 years ago

pbouda commented 4 years ago

(edit by @Korijn - transferred this issue here from wgpu-py repo)

I am trying to run the compute_noop.py example and get the following error:

$ RUST_BACKTRACE=1 python compute_noop.py
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Other', /Users/runner/runners/2.166.2/work/1/s/wgpu/wgpu-core/src/device/mod.rs:1895:17
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::panicking::default_hook::{{closure}}
   4: std::panicking::default_hook
   5: std::panicking::rust_panic_with_hook
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: wgpu_core::device::<impl wgpu_core::hub::Global<G>>::device_create_compute_pipeline
  10: ffi_call_unix64
  11: ffi_call_int
  12: cdata_call
  13: _PyObject_MakeTpCall
  14: call_function
  15: _PyEval_EvalFrameDefault
  16: _PyEval_EvalCodeWithName
  17: _PyFunction_Vectorcall
  18: method_vectorcall
  19: call_function
  20: _PyEval_EvalFrameDefault
  21: _PyEval_EvalCodeWithName
  22: _PyFunction_Vectorcall
  23: call_function
  24: _PyEval_EvalFrameDefault
  25: _PyEval_EvalCodeWithName
  26: PyEval_EvalCode
  27: PyRun_FileExFlags
  28: PyRun_SimpleFileExFlags
  29: Py_RunMain
  30: pymain_main
  31: Py_BytesMain
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6

How could I debug this further? I am on Mac btw. Any ideas?

almarklein commented 4 years ago

Hi, thanks for the report. I think it'd be good to wait for the upcoming release first. We're currently using a rather old version of wgpu-native. There was a new release of wgpu-native yesterday, so we must now update the wrapper and do a release. That will likely be next week.

With a bit of luck, that might solve the issue. Otherwise we'll see how we can dig deeper. (The new release should also have/allow more precise warnings etc.)

pbouda commented 4 years ago

Ok, thanks for the quick update, I will test with the next release then.

almarklein commented 4 years ago

v0.3.0 is available on Pypi!

pbouda commented 4 years ago

Now I get a bit more info about the error:

$ RUST_BACKTRACE=1 python compute_noop.py
Error compiling the shader CompilationFailed("Compilation failed: \n\nprogram_source:59:5: error: use of undeclared identifier \'_15\'\n    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];\n    ^\nprogram_source:59:13: error:  from vector \'uint3\' (vector of 3 \'unsigned int\' values) to scalar \'int\' of different size\n    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];\n            ^~~~~~~~~~~~~~~~~~~~~~~~~\nprogram_source:59:51: error:  from vector \'uint3\' (vector of 3 \'unsigned int\' values) to scalar \'int\' of different size\n    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];\n                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~\nprogram_source:59:43: error: use of undeclared identifier \'_11\'\n    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];\n                                          ^\n")
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Other', /Users/runner/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/6650b94/wgpu-core/src/device/mod.rs:2119:17
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::panicking::default_hook::{{closure}}
   4: std::panicking::default_hook
   5: std::panicking::rust_panic_with_hook
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: wgpu_core::device::<impl wgpu_core::hub::Global<G>>::device_create_compute_pipeline
  10: ffi_call_unix64
  11: ffi_call_int
  12: cdata_call
  13: _PyObject_MakeTpCall
  14: call_function
  15: _PyEval_EvalFrameDefault
  16: _PyEval_EvalCodeWithName
  17: _PyFunction_Vectorcall
  18: method_vectorcall
  19: call_function
  20: _PyEval_EvalFrameDefault
  21: _PyEval_EvalCodeWithName
  22: _PyFunction_Vectorcall
  23: call_function
  24: _PyEval_EvalFrameDefault
  25: _PyEval_EvalCodeWithName
  26: PyEval_EvalCode
  27: PyRun_FileExFlags
  28: PyRun_SimpleFileExFlags
  29: Py_RunMain
  30: pymain_main
  31: Py_BytesMain
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6
Korijn commented 4 years ago

seems like an issue in the generated shader (Error compiling the shader CompilationFailed). breaking the newlines in the error message gives us:

Error compiling the shader CompilationFailed("Compilation failed: 

program_source:59:5: error: use of undeclared identifier \'_15\'
    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];
    ^
program_source:59:13: error:  from vector \'uint3\' (vector of 3 \'unsigned int\' values) to scalar \'int\' of different size
    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];
            ^~~~~~~~~~~~~~~~~~~~~~~~~
program_source:59:51: error:  from vector \'uint3\' (vector of 3 \'unsigned int\' values) to scalar \'int\' of different size
    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];
                                                  ^~~~~~~~~~~~~~~~~~~~~~~~~
program_source:59:43: error: use of undeclared identifier \'_11\'
    _15._m0[int(gl_GlobalInvocationID)] = _11._m0[int(gl_GlobalInvocationID)];
                                          ^
")

I just tried running this on my computer and it's working for me on latest master (windows 10 x64 python 3.6, pyshader 0.6.2)

pbouda commented 4 years ago

Sure, I have:

Korijn commented 4 years ago

Would you mind seeing if the pyshader test suite passes on your machine? If there is a failure there, we should migrate this issue and we can narrow it down further over there.

BTW, we tested the pyshader 0.6.2 release on macos against python 3.8.3 without problems, as you can see here: https://github.com/pygfx/pyshader/runs/855426903?check_suite_focus=true#step:3:5

pbouda commented 4 years ago

I did run the pyshader tests, now. I took pyshader from master. Then I get this error, not sure if it's related:

$ pytest -v --cov=pyshader
=========================================================================================================== test session starts ============================================================================================================
platform darwin -- Python 3.8.2, pytest-5.4.3, py-1.9.0, pluggy-0.13.1 -- /Users/pbouda/.pyenv/versions/3.8.2/bin/python3.8
cachedir: .pytest_cache
rootdir: /Users/pbouda/Projects/tmp/pyshader
plugins: cov-2.10.0
collected 105 items

tests/test_api.py::test_api PASSED                                                                                                                                                                                                   [  0%]
tests/test_api.py::test_shader_module_class PASSED                                                                                                                                                                                   [  1%]
tests/test_api.py::test_spirv_constants PASSED                                                                                                                                                                                       [  2%]
tests/test_api.py::test_that_bytecode_generator_matches_opcode_definitions PASSED                                                                                                                                                    [  3%]
tests/test_api.py::test_some_internal_apis_too PASSED                                                                                                                                                                                [  4%]
tests/test_dev.py::test_run SKIPPED                                                                                                                                                                                                  [  5%]
tests/test_dev.py::test_fails SKIPPED                                                                                                                                                                                                [  6%]
tests/test_py.py::test_null_shader PASSED                                                                                                                                                                                            [  7%]
tests/test_py.py::test_triangle_shader PASSED                                                                                                                                                                                        [  8%]
tests/test_py.py::test_no_duplicate_constants SKIPPED                                                                                                                                                                                [  9%]
tests/test_py.py::test_compute_shader PASSED                                                                                                                                                                                         [ 10%]
tests/test_py.py::test_cannot_assign_same_slot PASSED                                                                                                                                                                                [ 11%]
tests/test_py.py::test_texture_2d_f32 PASSED                                                                                                                                                                                         [ 12%]
tests/test_py.py::test_texture_1d_i32 PASSED                                                                                                                                                                                         [ 13%]
tests/test_py.py::test_texture_3d_r16i PASSED                                                                                                                                                                                        [ 14%]
tests/test_py.py::test_texcomp_2d_rg32i PASSED                                                                                                                                                                                       [ 15%]
tests/test_py.py::test_tuple_unpacking Fatal Python error: Aborted

Current thread 0x000000011aeb0dc0 (most recent call first):
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/wgpu/backends/rs.py", line 869 in create_compute_pipeline
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/wgpu/utils/_compute.py", line 161 in compute_with_buffers
  File "/Users/pbouda/Projects/tmp/pyshader/tests/test_py.py", line 168 in test_tuple_unpacking
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/python.py", line 182 in pytest_pyfunc_call
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/python.py", line 1477 in runtest
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 135 in pytest_runtest_call
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 217 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 244 in from_call
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 216 in call_runtest_hook
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 186 in call_and_report
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 100 in runtestprotocol
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/runner.py", line 85 in pytest_runtest_protocol
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/main.py", line 272 in pytest_runtestloop
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/main.py", line 247 in _main
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/main.py", line 191 in wrap_session
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/main.py", line 240 in pytest_cmdline_main
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/callers.py", line 187 in _multicall
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 84 in <lambda>
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/manager.py", line 93 in _hookexec
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/pluggy/hooks.py", line 286 in __call__
  File "/Users/pbouda/.pyenv/versions/3.8.2/lib/python3.8/site-packages/_pytest/config/__init__.py", line 124 in main
  File "/Users/pbouda/.pyenv/versions/3.8.2/bin/pytest", line 8 in <module>
Abort trap: 6
Korijn commented 4 years ago

Yeah that looks like the same issue to me. Thanks for taking the time to do that, it helps a lot.

Korijn commented 4 years ago

Alright, final question: can you share with us which wgpu-native binary you are using? Is there a possibility you are using an outdated binary from when you switched to the latest version of wgpu-py (e.g. forgot to rerun the download script if you were running in development mode)?

almarklein commented 4 years ago

Mmm. So GlobalInvocationId is indeed a ivec3. It looks like on some systems it can be implicitly cast to int, but probably not on all drivers. Will have a look to make sure this is done the correct way.

To make sure that this is the problem, could you try changing the shader in the compute example to:

@python2shader
def compute_shader(
    index: ("input", "GlobalInvocationId", "ivec3"),
    data1: ("buffer", 0, Array(i32)),
    data2: ("buffer", 1, Array(i32)),
):
    data2[index.x] = data1[index.x]
Korijn commented 4 years ago

Do you have any idea how we could reproduce this @almarklein ? Or is it specific to the machine @pbouda is working with, you think?

almarklein commented 4 years ago

I think this issue is driver-specific.

Korijn commented 4 years ago

But if GlobalInvocationId is really ivec3 (or uvec3), shouldn't we update our example code and stop relying on a driver feature to do the automatic conversion?

pbouda commented 4 years ago

Here are my wgpu-native files, I found them in site-packages/wgpu/resources/, I hope that's what you need:

native.zip

I think it's the version that came with the Python package, so guess it's recent...?

pbouda commented 4 years ago

I also tried to change the example, as you suggested, but then I still get:

RUST_BACKTRACE=1 python compute_noop.py
Error compiling the shader CompilationFailed("Compilation failed: \n\nprogram_source:59:5: error: use of undeclared identifier \'_16\'\n    _16._m0[int(gl_GlobalInvocationID.x)] = _12._m0[int(gl_GlobalInvocationID.x)];\n    ^\nprogram_source:59:45: error: use of undeclared identifier \'_12\'\n    _16._m0[int(gl_GlobalInvocationID.x)] = _12._m0[int(gl_GlobalInvocationID.x)];\n                                            ^\n")
thread '<unnamed>' panicked at 'called `Result::unwrap()` on an `Err` value: Other', /Users/runner/.cargo/git/checkouts/wgpu-53e70f8674b08dd4/6650b94/wgpu-core/src/device/mod.rs:2119:17
stack backtrace:
   0: <std::sys_common::backtrace::_print::DisplayBacktrace as core::fmt::Display>::fmt
   1: core::fmt::write
   2: std::io::Write::write_fmt
   3: std::panicking::default_hook::{{closure}}
   4: std::panicking::default_hook
   5: std::panicking::rust_panic_with_hook
   6: rust_begin_unwind
   7: core::panicking::panic_fmt
   8: core::result::unwrap_failed
   9: wgpu_core::device::<impl wgpu_core::hub::Global<G>>::device_create_compute_pipeline
  10: ffi_call_unix64
  11: ffi_call_int
  12: cdata_call
  13: _PyObject_MakeTpCall
  14: call_function
  15: _PyEval_EvalFrameDefault
  16: _PyEval_EvalCodeWithName
  17: _PyFunction_Vectorcall
  18: method_vectorcall
  19: call_function
  20: _PyEval_EvalFrameDefault
  21: _PyEval_EvalCodeWithName
  22: _PyFunction_Vectorcall
  23: call_function
  24: _PyEval_EvalFrameDefault
  25: _PyEval_EvalCodeWithName
  26: PyEval_EvalCode
  27: PyRun_FileExFlags
  28: PyRun_SimpleFileExFlags
  29: Py_RunMain
  30: pymain_main
  31: Py_BytesMain
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
fatal runtime error: failed to initiate panic, error 5
Abort trap: 6
almarklein commented 4 years ago

Well, we did go from 4 errors to 2:

Error compiling the shader CompilationFailed("Compilation failed: 

program_source:59:5: error: use of undeclared identifier \'_16\'
    _16._m0[int(gl_GlobalInvocationID.x)] = _12._m0[int(gl_GlobalInvocationID.x)];
    ^
program_source:59:45: error: use of undeclared identifier \'_12\'
    _16._m0[int(gl_GlobalInvocationID.x)] = _12._m0[int(gl_GlobalInvocationID.x)];
                                            ^
almarklein commented 4 years ago

It looks like your driver is converting the SpirV to glsl, and in that process it lost the declaration of the storage arrays. I'll see if implementing this snippet in glsl, and compiling that to SpirV produces some SpirV instructions that we could add too ...

almarklein commented 4 years ago

All occurrences of GlobalInvocationID have been fixed in pyshader, wgpu-py and pygfx.