pymc-devs / pytensor

PyTensor allows you to define, optimize, and efficiently evaluate mathematical expressions involving multi-dimensional arrays.
https://pytensor.readthedocs.io
Other
366 stars 109 forks source link

BUG: pytensor.config.cxx sets wrong compiler from old environment #991

Open twiecki opened 2 months ago

twiecki commented 2 months ago

Describe the issue:

On current PyMC (5.16.2) and PyTensor (2.25.4) on OSX 15.0 Beta (24A5327a) all compilation fails. This was also reported in https://discourse.pymc.io/t/environment-not-working-anymore-on-macos/14210.

Reproducable code example:

import pymc as pm
with pm.Model():
    pm.Normal("x")
    pm.sample()

Error message:

<details>
ERROR (pytensor.graph.rewriting.basic): Rewrite failure due to: constant_folding
ERROR (pytensor.graph.rewriting.basic): node: Cast{float64}(0)
ERROR (pytensor.graph.rewriting.basic): TRACEBACK:
ERROR (pytensor.graph.rewriting.basic): Traceback (most recent call last):
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/graph/rewriting/basic.py", line 1909, in process_node
    replacements = node_rewriter.transform(fgraph, node)
                   ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/graph/rewriting/basic.py", line 1081, in transform
    return self.fn(fgraph, node)
           ^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/tensor/rewriting/basic.py", line 1122, in constant_folding
    thunk = node.op.make_thunk(node, storage_map, compute_map, no_recycling=[])
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/op.py", line 119, in make_thunk
    return self.make_c_thunk(node, storage_map, compute_map, no_recycling)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/op.py", line 84, in make_c_thunk
    outputs = cl.make_thunk(
              ^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/basic.py", line 1182, in make_thunk
    cthunk, module, in_storage, out_storage, error_storage = self.__compile__(
                                                             ^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/basic.py", line 1103, in __compile__
    thunk, module = self.cthunk_factory(
                    ^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/basic.py", line 1627, in cthunk_factory
    module = cache.module_from_key(key=key, lnk=self)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/cmodule.py", line 1255, in module_from_key
    module = lnk.compile_cmodule(location)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/basic.py", line 1528, in compile_cmodule
    module = c_compiler.compile_str(
             ^^^^^^^^^^^^^^^^^^^^^^^
  File "/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/cmodule.py", line 2654, in compile_str
    raise CompileError(
pytensor.link.c.exceptions.CompileError: Compilation failed (return status=1):
/Users/twiecki/micromamba/envs/pymc5/bin/clang++ -dynamiclib -g -O3 -fno-math-errno -Wno-unused-label -Wno-unused-variable -Wno-write-strings -Wno-c++11-narrowing -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables -DNPY_NO_DEPRECATED_API=NPY_1_7_API_VERSION -fPIC -undefined dynamic_lookup -I/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/numpy/core/include -I/Users/twiecki/micromamba/envs/pymc5_try2/include/python3.11 -I/Users/twiecki/micromamba/envs/pymc5_try2/lib/python3.11/site-packages/pytensor/link/c/c_code -L/Users/twiecki/micromamba/envs/pymc5_try2/lib -fvisibility=hidden -o /Users/twiecki/.pytensor/compiledir_macOS-15.0-arm64-arm-64bit-arm-3.11.10-64/tmpqe6xp4vb/m0de214ec04e404e0b98e77bebd2d95875e2e4b7b75c150c7f6478c42e61da0fa.so /Users/twiecki/.pytensor/compiledir_macOS-15.0-arm64-arm-64bit-arm-3.11.10-64/tmpqe6xp4vb/mod.cpp
In file included from /Users/twiecki/.pytensor/compiledir_macOS-15.0-arm64-arm-64bit-arm-3.11.10-64/tmpqe6xp4vb/mod.cpp:1:
/Users/twiecki/micromamba/envs/pymc5_try2/include/python3.11/Python.h:23:12: fatal error: 'stdlib.h' file not found
#  include <stdlib.h>
           ^~~~~~~~~~
1 error generated.
</details>

PyTensor version information:

floatX ({'float16', 'float32', 'float64'}) Doc: Default floating-point precision for python casts. Note: float16 support is experimental, use at your own risk. Value: float64 warn_float64 ({'ignore', 'pdb', 'raise', 'warn'}) Doc: Do an action when a tensor variable with float64 dtype is created. Value: ignore pickle_test_value (>) Doc: Dump test values while pickling model. If True, test values will be dumped with model. Value: True cast_policy ({'numpy+floatX', 'custom'}) Doc: Rules for implicit type casting Value: custom device (cpu) Doc: Default device for computations. only cpu is supported for now Value: cpu conv__assert_shape (>) Doc: If True, AbstractConv* ops will verify that user-provided shapes match the runtime shapes (debugging option, may slow down compilation) Value: False print_global_stats (>) Doc: Print some global statistics (time spent) at the end Value: False unpickle_function (>) Doc: Replace unpickled PyTensor functions with None. This is useful to unpickle old graphs that pickled them when it shouldn't Value: True Doc: Default compilation mode Value: Mode cxx () Doc: The C++ compiler to use. Currently only g++ is supported, but supporting additional compilers should not be too difficult. If it is empty, no C++ code is compiled. Value: /Users/twiecki/micromamba/envs/pymc5/bin/clang++ linker ({'cvm_nogc', 'c', 'c|py', 'py', 'vm_nogc', 'c|py_nogc', 'cvm', 'vm'}) Doc: Default linker used if the pytensor flags mode is Mode Value: cvm allow_gc (>) Doc: Do we default to delete intermediate results during PyTensor function calls? Doing so lowers the memory requirement, but asks that we reallocate memory at the next function call. This is implemented for the default linker, but may not work for all linkers. Value: True optimizer ({'fast_compile', 'o4', 'o2', 'None', 'fast_run', 'merge', 'o1', 'unsafe', 'o3'}) Doc: Default optimizer. If not None, will use this optimizer with the Mode Value: o4 optimizer_verbose (>) Doc: If True, we print all optimization being applied Value: False on_opt_error ({'warn', 'pdb', 'ignore', 'raise'}) Doc: What to do when an optimization crashes: warn and skip it, raise the exception, or fall into the pdb debugger. Value: warn nocleanup (>) Doc: Suppress the deletion of code files that did not compile cleanly Value: False on_unused_input ({'warn', 'ignore', 'raise'}) Doc: What to do if a variable in the 'inputs' list of pytensor.function() is not used in the graph. Value: raise gcc__cxxflags () Doc: Extra compiler flags for gcc Value: -Wno-c++11-narrowing -fno-exceptions -fno-unwind-tables -fno-asynchronous-unwind-tables cmodule__warn_no_version (>) Doc: If True, will print a warning when compiling one or more Op with C code that can't be cached because there is no c_code_cache_version() function associated to at least one of those Ops. Value: False cmodule__remove_gxx_opt (>) Doc: If True, will remove the -O* parameter passed to g++.This is useful to debug in gdb modules compiled by PyTensor.The parameter -g is passed by default to g++ Value: False cmodule__compilation_warning (>) Doc: If True, will print compilation warnings. Value: False cmodule__preload_cache (>) Doc: If set to True, will preload the C module cache at import time Value: False cmodule__age_thresh_use () Doc: In seconds. The time after which PyTensor won't reuse a compile c module. Value: 2073600 cmodule__debug (>) Doc: If True, define a DEBUG macro (if not exists) for any compiled C code. Value: False compile__wait () Doc: Time to wait before retrying to acquire the compile lock. Value: 5 compile__timeout () Doc: In seconds, time that a process will wait before deciding to override an existing lock. An override only happens when the existing lock is held by the same owner *and* has not been 'refreshed' by this owner for more than this period. Refreshes are done every half timeout period for running processes. Value: 120 tensor__cmp_sloppy () Doc: Relax pytensor.tensor.math._allclose (0) not at all, (1) a bit, (2) more Value: 0 lib__amdlibm (>) Doc: Use amd's amdlibm numerical library Value: False tensor__insert_inplace_optimizer_validate_nb () Doc: -1: auto, if graph have less then 500 nodes 1, else 10 Value: -1 traceback__limit () Doc: The number of stack to trace. -1 mean all. Value: 8 traceback__compile_limit () Doc: The number of stack to trace to keep during compilation. -1 mean all. If greater then 0, will also make us save PyTensor internal stack trace. Value: 0 warn__ignore_bug_before ({'0.6', '0.8.1', '1.0.2', '0.7', '0.5', '0.10', '1.0.4', '0.8.2', '1.0.5', 'all', '0.8', '0.4', '1.0.3', '0.3', '0.9', '0.4.1', '1.0', '1.0.1', 'None'}) Doc: If 'None', we warn about all PyTensor bugs found by default. If 'all', we don't warn about PyTensor bugs found by default. If a version, we print only the warnings relative to PyTensor bugs found after that version. Warning for specific bugs can be configured with specific [warn] flags. Value: 0.9 exception_verbosity ({'high', 'low'}) Doc: If 'low', the text of exceptions will generally refer to apply nodes with short names such as Elemwise{add_no_inplace}. If 'high', some exceptions will also refer to apply nodes with long descriptions like: A. Elemwise{add_no_inplace} B. log_likelihood_v_given_h C. log_likelihood_h Value: low print_test_value (>) Doc: If 'True', the __eval__ of an PyTensor variable will return its test_value when this is available. This has the practical consequence that, e.g., in debugging `my_var` will print the same as `my_var.tag.test_value` when a test value is defined. Value: False compute_test_value ({'warn', 'pdb', 'raise', 'ignore', 'off'}) Doc: If 'True', PyTensor will run each op at graph build time, using Constants, SharedVariables and the tag 'test_value' as inputs to the function. This helps the user track down problems in the graph before it gets optimized. Value: off compute_test_value_opt ({'warn', 'pdb', 'raise', 'ignore', 'off'}) Doc: For debugging PyTensor optimization only. Same as compute_test_value, but is used during PyTensor optimization Value: off check_input (>) Doc: Specify if types should check their input in their C code. It can be used to speed up compilation, reduce overhead (particularly for scalars) and reduce the number of generated C files. Value: True NanGuardMode__nan_is_error (>) Doc: Default value for nan_is_error Value: True NanGuardMode__inf_is_error (>) Doc: Default value for inf_is_error Value: True NanGuardMode__big_is_error (>) Doc: Default value for big_is_error Value: True NanGuardMode__action ({'warn', 'pdb', 'raise'}) Doc: What NanGuardMode does when it finds a problem Value: raise DebugMode__patience () Doc: Optimize graph this many times to detect inconsistency Value: 10 DebugMode__check_c (>) Doc: Run C implementations where possible Value: True DebugMode__check_py (>) Doc: Run Python implementations where possible Value: True DebugMode__check_finite (>) Doc: True -> complain about NaN/Inf results Value: True DebugMode__check_strides () Doc: Check that Python- and C-produced ndarrays have same strides. On difference: (0) - ignore, (1) warn, or (2) raise error Value: 0 DebugMode__warn_input_not_reused (>) Doc: Generate a warning when destroy_map or view_map says that an op works inplace, but the op did not reuse the input for its output. Value: True DebugMode__check_preallocated_output () Doc: Test thunks with pre-allocated memory as output storage. This is a list of strings separated by ":". Valid values are: "initial" (initial storage in storage map, happens with Scan),"previous" (previously-returned memory), "c_contiguous", "f_contiguous", "strided" (positive and negative strides), "wrong_size" (larger and smaller dimensions), and "ALL" (all of the above). Value: DebugMode__check_preallocated_output_ndim () Doc: When testing with "strided" preallocated output memory, test all combinations of strides over that number of (inner-most) dimensions. You may want to reduce that number to reduce memory or time usage, but it is advised to keep a minimum of 2. Value: 4 profiling__time_thunks (>) Doc: Time individual thunks when profiling Value: True profiling__n_apply () Doc: Number of Apply instances to print by default Value: 20 profiling__n_ops () Doc: Number of Ops to print by default Value: 20 profiling__output_line_width () Doc: Max line width for the profiling output Value: 512 profiling__min_memory_size () Doc: For the memory profile, do not print Apply nodes if the size of their outputs (in bytes) is lower than this threshold Value: 1024 profiling__min_peak_memory (>) Doc: The min peak memory usage of the order Value: False profiling__destination () Doc: File destination of the profiling output Value: stderr profiling__debugprint (>) Doc: Do a debugprint of the profiled functions Value: False profiling__ignore_first_call (>) Doc: Do we ignore the first call of an PyTensor function. Value: False on_shape_error ({'warn', 'raise'}) Doc: warn: print a warning and use the default value. raise: raise an error Value: warn openmp (>) Doc: Allow (or not) parallel computation on the CPU with OpenMP. This is the default value used when creating an Op that supports OpenMP parallelization. It is preferable to define it via the PyTensor configuration file ~/.pytensorrc or with the environment variable PYTENSOR_FLAGS. Parallelization is only done for some operations that implement it, and even for operations that implement parallelism, each operation is free to respect this flag or not. You can control the number of threads used with the environment variable OMP_NUM_THREADS. If it is set to 1, we disable openmp in PyTensor by default. Value: False openmp_elemwise_minsize () Doc: If OpenMP is enabled, this is the minimum size of vectors for which the openmp parallelization is enabled in element wise ops. Value: 200000 optimizer_excluding () Doc: When using the default mode, we will remove optimizer with these tags. Separate tags with ':'. Value: optimizer_including () Doc: When using the default mode, we will add optimizer with these tags. Separate tags with ':'. Value: optimizer_requiring () Doc: When using the default mode, we will require optimizer with these tags. Separate tags with ':'. Value: optdb__position_cutoff () Doc: Where to stop earlier during optimization. It represent the position of the optimizer where to stop. Value: inf optdb__max_use_ratio () Doc: A ratio that prevent infinite loop in EquilibriumGraphRewriter. Value: 8.0 cycle_detection ({'fast', 'regular'}) Doc: If cycle_detection is set to regular, most inplaces are allowed,but it is slower. If cycle_detection is set to faster, less inplacesare allowed, but it makes the compilation faster.The interaction of which one give the lower peak memory usage iscomplicated and not predictable, so if you are close to the peakmemory usage, triyng both could give you a small gain. Value: regular check_stack_trace ({'log', 'raise', 'off', 'warn'}) Doc: A flag for checking the stack trace during the optimization process. default (off): does not check the stack trace of any optimization log: inserts a dummy stack trace that identifies the optimizationthat inserted the variable that had an empty stack trace.warn: prints a warning if a stack trace is missing and also a dummystack trace is inserted that indicates which optimization insertedthe variable that had an empty stack trace.raise: raises an exception if a stack trace is missing Value: off metaopt__verbose () Doc: 0 for silent, 1 for only warnings, 2 for full output withtimings and selected implementation Value: 0 unittests__rseed () Doc: Seed to use for randomized unit tests. Special value 'random' means using a seed of None. Value: 666 warn__round (>) Doc: Warn when using `tensor.round` with the default mode. Round changed its default from `half_away_from_zero` to `half_to_even` to have the same default as NumPy. Value: False profile (>) Doc: If VM should collect profile information Value: False profile_optimizer (>) Doc: If VM should collect optimizer profile information Value: False profile_memory (>) Doc: If VM should collect memory profile information and print it Value: False Doc: Useful only for the VM Linkers. When lazy is None, auto detect if lazy evaluation is needed and use the appropriate version. If the C loop isn't being used and lazy is True, use the Stack VM; otherwise, use the Loop VM. Value: None numba__vectorize_target ({'cpu', 'cuda', 'parallel'}) Doc: Default target for numba.vectorize. Value: cpu numba__fastmath (>) Doc: If True, use Numba's fastmath mode. Value: True numba__cache (>) Doc: If True, use Numba's file based caching. Value: True compiledir_format () Doc: Format string for platform-dependent compiled module subdirectory (relative to base_compiledir). Available keys: device, gxx_version, hostname, numpy_version, platform, processor, pytensor_version, python_bitwidth, python_int_bitwidth, python_version, short_platform. Defaults to compiledir_%(short_platform)s-%(processor)s- %(python_version)s-%(python_bitwidth)s. Value: compiledir_%(short_platform)s-%(processor)s-%(python_version)s-%(python_bitwidth)s Doc: platform-independent root directory for compiled modules Value: /Users/twiecki/.pytensor Doc: platform-dependent cache directory for compiled modules Value: /Users/twiecki/.pytensor/compiledir_macOS-15.0-arm64-arm-64bit-arm-3.11.10-64 blas__ldflags () Doc: lib[s] to include for [Fortran] level-3 blas implementation Value: blas__check_openmp (>) Doc: Check for openmp library conflict. WARNING: Setting this to False leaves you open to wrong results in blas-related operations. Value: True scan__allow_gc (>) Doc: Allow/disallow gc inside of Scan (default: False) Value: False scan__allow_output_prealloc (>) Doc: Allow/disallow memory preallocation for outputs inside of scan (default: True) Value: True

Context for the issue:

No response

twiecki commented 2 months ago

For some reason it was using clang++ from a different conda environment. Setting it manually fixed the issue. But not sure why it's not selecting clang++ from the currently active env.

tigretigre commented 2 months ago

For me it was selecting clang++ from the environment, but what got me going again was moving that out of the way so it got the version from /usr/bin, i.e. the one from Xcode dev tools, as suggested here: https://discourse.pymc.io/t/pytensor-compilation-error/11047/13

ricardoV94 commented 1 month ago

CC @lucianopaz

lucianopaz commented 1 month ago

This makes very little sense. To set cxx to "clang++", pytensor simply tries to run clang++ -v on a subprocess. If it succeeds, then cxx is set to "clang++", if it fails, it tries to call shutil.which("clang++") which should set the full path to the clang++ that has highest precedence in the environments path search order. The only reason you might get a clang++ that is not part of your active env, is that something went wrong with the environment activation and it didn't set the proper values to some environment variables.

On the other hand, what @tigretigre is pointing at is different. He says that he had to use the system's clang instead of the one that came with the conda, mamba or anaconda environment. That usually happens when a mamba env clang that was installed under one OS version is migrated into a newer OS version. Sometimes, some of the header files have #include <stdlib.h> or #include <unistd.h>. These precompiler directives attempt to include headers that should be in the compiler's default include path. Sometimes, these get messed up or vary slightly with OS updates. Sometimes, these problems go away if you uninstall and then reinstall conda/mamba/etc. I've seen reports of people saying that reinstalling the environment alone didn't help, but reinstalling conda did. I don't know why, to be honest.

The last part is that these problems usually seem to pop out in VSCode. I have absolutely no idea why that happens, and what VSCode does with the env activation that can lead to so much trouble down the line. My last suggestion is that if you're seeing this problem in VSCode, first try to check if the problem is also present in a regular shell or jupyter kernel.

maresb commented 1 month ago

I have absolutely no idea why that happens, and what VSCode does with the env activation that can lead to so much trouble down the line.

I have a suspicion. In order for a Conda environment to be properly activated, there are scripts in $CONDA_PREFIX/etc/conda/activate.d/ that must be executed. Often laziness gets in the way, a few directories are added to $PATH, and then whatever tool calls it good.

This is not so easy to diagnose, since if you run

!source "$CONDA_PREFIX"/etc/conda/activate.d/*.sh

in Jupyter this launches in a subshell where all the changes to your environment will be forgotten after the subshell is terminated.

But it would be interesting to check this directory for anything related to clang.