search

llvm / llvm-project

The LLVM Project is a collection of modular and reusable compiler and toolchain technologies.
http://llvm.org
Other
29.38k stars 12.14k forks source link

[mlir python] Python development auto-detection is unreliable #54906

Open stellaraccident opened 2 years ago

stellaraccident commented 2 years ago

I just spent another 20 minutes trying to figure out why Python auto-detection was failing for the standalone example and pin-pointed it to some bad state in a cached build directory. This is a continuation of the trend where the CMake FindPython macros are just broken for complicated projects and I personally don't think that they can be fixed at this point barring a rewrite.

This is unfortunate because what we are using it for is really simple and can be accomplished with ~a dozen lines of a really straight-forward Python script that introspects the running interpreter via sysconfig. I know this because this is what I had in the pre-cursor to the MLIR Python API out of tree, but when upstreaming, I thought I was doing the right thing by switching to something more standard. I am quite confident that I could write something robust here that would also be a lot more debuggable and adaptable if not trying to spelunk through the decade of Python2/3/whatever CMake autoconfig tragedy.

In order to not cause more thrash than necessary, I propose adding this under yet another temporary flag for auto-detecting Python. We put a bit of mileage on it, and if it works, we just drop the old escalation of hacks that have accumulated in MLIRDetectPythonEnv.cmake. Seems low risk to try. Any objections?

llvmbot commented 2 years ago

@llvm/issue-subscribers-mlir-python

joker-eph commented 2 years ago

LGTM

mikeurbach commented 2 years ago

Thanks @stellaraccident, this makes sense and the plan sounds good to me. I'm happy to take it for a test drive in CIRCT.

stellaraccident commented 2 years ago

I implemented a POC of this here: https://reviews.llvm.org/D125182

stellaraccident commented 2 years ago

It ended up being more than "a dozen lines" once I made it nice, added error reporting, and the like :)