search

olokelo / jxlpy

Cython bindings and Pillow plugin for JPEG XL
MIT License
49 stars 12 forks source link

Access Violation on Windows Build when encoding #7

Open EraYaN opened 2 years ago

EraYaN commented 2 years ago

When building this wheel on Windows using:

vcpkg install libjxl --triplet x64-windows

pip wheel --use-feature=in-tree-build ./jxlpy

delvewheel repair -v --add-path C:\tools\vcpkg\installed\x64-windows\bin --no-dll msvcp140.dll --no-mangle-all jxlpy-0.9.1-cp39-cp39-win_amd64.whl

pip uninstall jxlpy -y
pip install wheelhouse/jxlpy-0.9.1-cp39-cp39-win_amd64.whl

python decode_simple.py

python encode_decode_pillow.py

echo %errorlevel%

This gives an access violation on calling im.save, with the call stack

Exception thrown at 0x00007FFFDF941372 (jxl.dll) in python.exe: 0xC0000005: Access violation reading location 0x0000000000003C80.
[Inline Frame] jxl.dll!hwy::N_AVX2::LoadDup128(hwy::N_AVX2::Simd<float,8>) Line 1994
    at C:\tools\vcpkg\installed\x64-windows\include\hwy\ops\x86_256-inl.h(1994)
jxl.dll!jxl::N_AVX2::Symmetric5Interior<jxl::WrapUnchanged>(const jxl::Plane<float> & in, const jxl::Rect & rect, const __int64 ix, const __int64 iy, const jxl::WeightsSymmetric5 & weights, float * row_out) Line 95
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\convolve.cc(95)
jxl.dll!jxl::N_AVX2::Symmetric5Row<jxl::WrapUnchanged>(const jxl::Plane<float> & in, const jxl::Rect & rect, const __int64 iy, const jxl::WeightsSymmetric5 & weights, float * row_out) Line 131
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\convolve.cc(131)
[Inline Frame] jxl.dll!jxl::N_AVX2::Symmetric5::__l2::<lambda_89f43947037a03667304206744c598ea>::operator()(const int) Line 960
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\convolve.cc(960)
jxl.dll!jxl::ThreadPool::RunCallState<jxl::Status __cdecl(unsigned __int64),<lambda_89f43947037a03667304206744c598ea>>::CallDataFunc(void * jpegxl_opaque, unsigned int value, unsigned __int64 thread_id) Line 89
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\base\data_parallel.h(89)
jxl_threads.dll!jpegxl::ThreadParallelRunner::Runner(void * runner_opaque, void * jpegxl_opaque, int(*)(void *, unsigned __int64) init, void(*)(void *, unsigned int, unsigned __int64) func, unsigned int start_range, unsigned int end_range) Line 72
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\threads\thread_parallel_runner_internal.cc(72)
[Inline Frame] jxl.dll!jxl::ThreadPool::Run(unsigned int) Line 51
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\base\data_parallel.h(51)
[Inline Frame] jxl.dll!jxl::ThreadPool::Run(unsigned int) Line 60
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\base\data_parallel.h(60)
[Inline Frame] jxl.dll!jxl::RunOnPool(jxl::ThreadPool *) Line 117
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\base\data_parallel.h(117)
jxl.dll!jxl::N_AVX2::Symmetric5(const jxl::Plane<float> & in, const jxl::Rect & rect, const jxl::WeightsSymmetric5 & weights, jxl::ThreadPool * pool, jxl::Plane<float> * out) Line 964
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\convolve.cc(964)
jxl.dll!jxl::GaborishInverse(jxl::Image3<float> * in_out, float mul, jxl::ThreadPool * pool) Line 62
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\gaborish.cc(62)
jxl.dll!jxl::DefaultEncoderHeuristics::LossyFrameHeuristics(jxl::PassesEncoderState * enc_state, jxl::ModularFrameEncoder * modular_frame_encoder, const jxl::ImageBundle * original_pixels, jxl::Image3<float> * opsin, jxl::ThreadPool * pool, jxl::AuxOut * aux_out) Line 865
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\enc_heuristics.cc(865)
jxl.dll!jxl::LossyFrameEncoder::ComputeEncodingData(const jxl::ImageBundle * linear, jxl::Image3<float> * opsin, jxl::ThreadPool * pool, jxl::ModularFrameEncoder * modular_frame_encoder, jxl::BitWriter * writer, jxl::FrameHeader * frame_header) Line 497
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\enc_frame.cc(497)
jxl.dll!jxl::EncodeFrame(const jxl::CompressParams & cparams_orig, const jxl::FrameInfo & frame_info, const jxl::CodecMetadata * metadata, const jxl::ImageBundle & ib, jxl::PassesEncoderState * passes_enc_state, jxl::ThreadPool * pool, jxl::BitWriter * writer, jxl::AuxOut * aux_out) Line 1187
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\enc_frame.cc(1187)
jxl.dll!JxlEncoderStruct::RefillOutputByteQueue() Line 94
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\encode.cc(94)
jxl.dll!JxlEncoderProcessOutput(JxlEncoderStruct * enc, unsigned char * * next_out, unsigned __int64 * avail_out) Line 458
    at C:\tools\vcpkg\buildtrees\libjxl\src\v0.6.1-d6e48761e1.clean\lib\jxl\encode.cc(458)
_jxlpy.cp39-win_amd64.pyd!__pyx_pf_6_jxlpy_12JXLPyEncoder_4get_output(__pyx_obj_6_jxlpy_JXLPyEncoder * __pyx_v_self) Line 5423
    at C:\Users\ErwindeHaan\jxlpy\jxlpy\_jxlpy\_jxl.cpp(5423)
python39.dll!method_vectorcall_NOARGS(_object * func, _object * const * args, unsigned __int64 nargsf, _object * kwnames) Line 435
    at \objects\descrobject.c(435)
[Inline Frame] python39.dll!_PyObject_VectorcallTstate(_ts *) Line 118
    at \include\cpython\abstract.h(118)
[Inline Frame] python39.dll!PyObject_Vectorcall(_object * nargsf, _object * const *) Line 127
    at \include\cpython\abstract.h(127)
python39.dll!call_function(_ts * tstate, _object * * * pp_stack, __int64 oparg, _object * kwnames) Line 5072
    at \python\ceval.c(5072)
python39.dll!_PyEval_EvalFrameDefault(_ts * tstate, _frame * f, int throwflag) Line 3505
    at \python\ceval.c(3505)
[Inline Frame] python39.dll!_PyEval_EvalFrame(_ts *) Line 40
    at \include\internal\pycore_ceval.h(40)
python39.dll!_PyEval_EvalCode(_ts * tstate, _object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4327
    at \python\ceval.c(4327)
[Inline Frame] python39.dll!_PyFunction_Vectorcall(_object * stack, _object * const *) Line 396
    at \objects\call.c(396)
[Inline Frame] python39.dll!_PyObject_VectorcallTstate(_ts *) Line 118
    at \include\cpython\abstract.h(118)
[Inline Frame] python39.dll!PyObject_Vectorcall(_object * nargsf, _object * const *) Line 127
    at \include\cpython\abstract.h(127)
python39.dll!call_function(_ts * tstate, _object * * * pp_stack, __int64 oparg, _object * kwnames) Line 5072
    at \python\ceval.c(5072)
python39.dll!_PyEval_EvalFrameDefault(_ts * tstate, _frame * f, int throwflag) Line 3519
    at \python\ceval.c(3519)
[Inline Frame] python39.dll!_PyEval_EvalFrame(_ts *) Line 40
    at \include\internal\pycore_ceval.h(40)
python39.dll!_PyEval_EvalCode(_ts * tstate, _object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4327
    at \python\ceval.c(4327)
[Inline Frame] python39.dll!_PyFunction_Vectorcall(_object * stack, _object * const *) Line 396
    at \objects\call.c(396)
[Inline Frame] python39.dll!_PyObject_VectorcallTstate(_ts *) Line 118
    at \include\cpython\abstract.h(118)
python39.dll!method_vectorcall(_object * method, _object * const * args, unsigned __int64 nargsf, _object * kwnames) Line 53
    at \objects\classobject.c(53)
[Inline Frame] python39.dll!_PyObject_VectorcallTstate(_ts *) Line 118
    at \include\cpython\abstract.h(118)
[Inline Frame] python39.dll!PyObject_Vectorcall(_object * nargsf, _object * const *) Line 127
    at \include\cpython\abstract.h(127)
python39.dll!call_function(_ts * tstate, _object * * * pp_stack, __int64 oparg, _object * kwnames) Line 5072
    at \python\ceval.c(5072)
python39.dll!_PyEval_EvalFrameDefault(_ts * tstate, _frame * f, int throwflag) Line 3536
    at \python\ceval.c(3536)
[Inline Frame] python39.dll!_PyEval_EvalFrame(_ts *) Line 40
    at \include\internal\pycore_ceval.h(40)
python39.dll!_PyEval_EvalCode(_ts * tstate, _object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4327
    at \python\ceval.c(4327)
python39.dll!_PyEval_EvalCodeWithName(_object * _co, _object * globals, _object * locals, _object * const * args, __int64 argcount, _object * const * kwnames, _object * const * kwargs, __int64 kwcount, int kwstep, _object * const * defs, __int64 defcount, _object * kwdefs, _object * closure, _object * name, _object * qualname) Line 4366
    at \python\ceval.c(4366)
python39.dll!PyEval_EvalCodeEx(_object * _co, _object * globals, _object * locals, _object * const * args, int argcount, _object * const * kws, int kwcount, _object * const * defs, int defcount, _object * kwdefs, _object * closure) Line 4382
    at \python\ceval.c(4382)
python39.dll!PyEval_EvalCode(_object * co, _object * globals, _object * locals) Line 832
    at \python\ceval.c(832)
python39.dll!run_eval_code_obj(_ts * tstate, PyCodeObject * co, _object * globals, _object * locals) Line 1220
    at \python\pythonrun.c(1220)
python39.dll!run_mod(_mod * mod, _object * filename, _object * globals, _object * locals, PyCompilerFlags * flags, _arena * arena) Line 1241
    at \python\pythonrun.c(1241)
python39.dll!pyrun_file(_iobuf * fp, _object * filename, int start, _object * globals, _object * locals, int closeit, PyCompilerFlags * flags) Line 1138
    at \python\pythonrun.c(1138)
python39.dll!pyrun_simple_file(_iobuf * fp, _object * filename, int closeit, PyCompilerFlags * flags) Line 452
    at \python\pythonrun.c(452)
python39.dll!PyRun_SimpleFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 483
    at \python\pythonrun.c(483)
python39.dll!PyRun_AnyFileExFlags(_iobuf * fp, const char * filename, int closeit, PyCompilerFlags * flags) Line 92
    at \python\pythonrun.c(92)
python39.dll!pymain_run_file(const PyConfig * config, PyCompilerFlags * cf) Line 379
    at \modules\main.c(379)
python39.dll!pymain_run_python(int * exitcode) Line 605
    at \modules\main.c(605)
python39.dll!Py_RunMain() Line 685
    at \modules\main.c(685)
python39.dll!pymain_main(_PyArgv * args) Line 1106
    at \modules\main.c(1106)
python39.dll!Py_Main(int argc, wchar_t * * argv) Line 1118
    at \modules\main.c(1118)

So it seems to happen on the call here.

/* "_jxlpy/_jxl.pyx":804
 *         while self.status == JXL_ENC_NEED_MORE_OUTPUT:
 * 
 *             self.status = JxlEncoderProcessOutput(             # <<<<<<<<<<<<<<
 *                 self.encoder, &next_out, &avail_out
 *             )
 */

Do note to make this work you need to add the follwing to setup.py

jxlpy_ext = Extension(
    name="_jxlpy",
    sources=["_jxlpy/_jxl.pyx"],
    libraries=['jxl','jxl_threads'],
    extra_compile_args=['-Ox','-Zi'], # For debugging only, use O2 for release build
    extra_link_args=['-debug:full'], # For debugging only
    language='c++'
)
from sys import platform
if platform == "win32":
    jxlpy_ext.include_dirs=['C:\\tools\\vcpkg\\installed\\x64-windows\\include']
    jxlpy_ext.library_dirs=['C:\\tools\\vcpkg\\installed\\x64-windows\\lib']

Any ideas?

olokelo commented 2 years ago

Hi, thank you very much for detailed instructions. I was able to follow along even though I'm not into Windows development. It might help creating windows prebuilt wheels of jxlpy.

As for your issue, unfortunately I wasn't able to reproduce it. For me, all examples work fine. I really hate being but it works on my pc guy but that's what it is :/

In the meantime, I did a quick search for this issue and found out, it might be related to Cython. (source 1, 2)

I will attach my prebuilt and repaired wheel below if that helps, maybe try using it.

jxlpy-0.9.1-cp39-cp39-win_amd64.whl.zip

EraYaN commented 2 years ago

Right that gives the same result here, so it's probably some problem with my Cython version or environment Would you be able to tell me how you installed cython and what version you are running? here it installed Cython-0.29.26-py2.py3-none-any.whl from pypi

EDIT; Seems like lossless mode does work, but for example quality=75, effort=6 does not. Also tried is on an other Windows 10 machine and it also does not work there, not sure what is wrong. Did you build with VS2022?

olokelo commented 2 years ago

My Cython version is the same as yours, installed automatically as a dependency of jxlpy. I'm running Windows 10 build 19042 on VirtualBox. Python 3.9.10, cython 0.29.26. My CPU (amd64) supports avx2 (in your error log there was something about libhwy executing some avx2 instructions but I doubt it's relevant). I installed libjxl with VS2022 using vcpkg.

I did some encoding/decoding tests with lossless and lossy mode and everything seems fine..

EraYaN commented 2 years ago

Alright I'm running python 3.9.5 maybe if I upgrade that and I just tried with 3.9.10 and same problem my windows version is 21H1 (build 19043). But still the same behavior, the actual site of the access violation is within preprocessor guards for the MSVC compiler. (using some intrinsics) Can cython build with clang on windows?

I'm stumped as to why this is happening.