search

JuliaPy / PythonCall.jl

Python and Julia in harmony.
https://juliapy.github.io/PythonCall.jl/stable/
MIT License
721 stars 61 forks source link

`@pyexec` fails to define functions or set global variables #370

Open LilithHafner opened 9 months ago

LilithHafner commented 9 months ago

Affects: PythonCall

When I run pyexec("x = 1", Main), it successfully sets the global variable. When, on the other hand, I run @pyexec "x = 1", it does not. This extends to defining functions as well. It feels like a pretty serious bug to me. Maybe I'm misunderstanding how global environment management works?

julia> pyexec("x = 1", Main)

julia> pyeval("x", Main)
Python: 1

julia> @pyexec "y = 2"

julia> @pyeval "y"
ERROR: Python: NameError: name 'y' is not defined
Python stacktrace:
 [1] <module>
   @ REPL[5]:1:1
Stacktrace:
 [1] pythrow()
   @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/err.jl:94
 [2] errcheck
   @ ~/.julia/packages/PythonCall/qTEA1/src/err.jl:10 [inlined]
 [3] pycallargs(f::Py, args::Py)
   @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:210
 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}})
   @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:228
 [5] pycall
   @ ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:218 [inlined]
 [6] #_#11
   @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined]
 [7] Py
   @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined]
 [8] pyeval(::Type{Py}, code::Py, globals::Module, locals::Tuple{})
   @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:43
 [9] top-level scope
   @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:146
Full log ```julia (@v1.9) pkg> activate --temp Activating new project at `/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY` (jl_NIT4tY) pkg> add PythonCall [ Info: Use `./PythonCall` to add or develop the local directory at `~/.julia/dev/PythonCall`. Resolving package versions... Installed CondaPkg ─ v0.2.20 Updating `/private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/Project.toml` [6099a3de] + PythonCall v0.9.14 Updating `/private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/Manifest.toml` [992eb4ea] + CondaPkg v0.2.20 [9a962f9c] + DataAPI v1.15.0 [e2d170a0] + DataValueInterfaces v1.0.0 [82899510] + IteratorInterfaceExtensions v1.0.0 [692b3bcd] + JLLWrappers v1.5.0 [0f8b85d8] + JSON3 v1.13.2 [1914dd2f] + MacroTools v0.5.11 [0b3b1443] + MicroMamba v0.1.14 [bac558e1] + OrderedCollections v1.6.2 [69de0a69] + Parsers v2.7.2 [fa939f87] + Pidfile v1.3.0 [aea7be01] + PrecompileTools v1.2.0 [21216c6a] + Preferences v1.4.1 [6099a3de] + PythonCall v0.9.14 [ae029012] + Requires v1.3.0 [6c6a2e73] + Scratch v1.2.0 [856f2bd8] + StructTypes v1.10.0 [3783bdb8] + TableTraits v1.0.1 [bd369af6] + Tables v1.11.0 [e17b2a0c] + UnsafePointers v1.0.0 [f8abcde7] + micromamba_jll v1.4.9+0 [0dad84c5] + ArgTools v1.1.1 [56f22d72] + Artifacts [2a0f44e3] + Base64 [ade2ca70] + Dates [f43a241f] + Downloads v1.6.0 [7b1f6079] + FileWatching [b77e0a4c] + InteractiveUtils [4af54fe1] + LazyArtifacts [b27032c2] + LibCURL v0.6.3 [76f85450] + LibGit2 [8f399da3] + Libdl [37e2e46d] + LinearAlgebra [56ddb016] + Logging [d6f4376e] + Markdown [a63ad114] + Mmap [ca575930] + NetworkOptions v1.2.0 [44cfe95a] + Pkg v1.9.2 [de0858da] + Printf [3fa0cd96] + REPL [9a3f8284] + Random [ea8e919c] + SHA v0.7.0 [9e88b42a] + Serialization [6462fe0b] + Sockets [fa267f1f] + TOML v1.0.3 [a4e569a6] + Tar v1.10.0 [8dfed614] + Test [cf7118a7] + UUIDs [4ec0a83e] + Unicode [e66e0078] + CompilerSupportLibraries_jll v1.0.5+0 [deac9b47] + LibCURL_jll v7.84.0+0 [29816b5a] + LibSSH2_jll v1.10.2+0 [c8ffd9c3] + MbedTLS_jll v2.28.2+0 [14a3606d] + MozillaCACerts_jll v2022.10.11 [4536629a] + OpenBLAS_jll v0.3.21+4 [83775a58] + Zlib_jll v1.2.13+0 [8e850b90] + libblastrampoline_jll v5.8.0+0 [8e850ede] + nghttp2_jll v1.48.0+0 [3f19e933] + p7zip_jll v17.4.0+0 Precompiling project... 5 dependencies successfully precompiled in 12 seconds. 20 already precompiled. julia> using PythonCall [ Info: Precompiling PythonCall [6099a3de-0909-46bc-b1f4-468b9a2dfc0d] CondaPkg Found dependencies: /Users/x/.julia/packages/PythonCall/qTEA1/CondaPkg.toml CondaPkg Found dependencies: /Users/x/.julia/packages/SymPyPythonCall/yNd8x/CondaPkg.toml CondaPkg Resolving changes + python + sympy CondaPkg Creating environment │ /Users/x/.julia/artifacts/5767a45b62528e0be4c40dbae502ad3e132f1d62/bin/micromamba │ -r /Users/x/.julia/scratchspaces/0b3b1443-0f03-428d-bdfb-f27f9c1191ea/root │ create │ -y │ -p /var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/.CondaPkg/env │ --override-channels │ --no-channel-priority │ python[version='>=3.7,<4',channel='conda-forge',build='*cpython*'] │ python[version='*'] │ sympy[version='*'] └ -c conda-forge conda-forge/noarch 12.5MB @ 3.0MB/s 4.2s conda-forge/osx-arm64 6.8MB @ 1.1MB/s 6.4s Transaction Prefix: /private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/.CondaPkg/env Updating specs: - conda-forge::python[version='>=3.7,<4',build=*cpython*] - python=* - sympy=* Package Version Build Channel Size ──────────────────────────────────────────────────────────────────────────── Install: ──────────────────────────────────────────────────────────────────────────── + xz 5.2.6 h57fd34a_0 conda-forge Cached + libexpat 2.5.0 hb7217d7_1 conda-forge Cached + ncurses 6.4 h7ea286d_0 conda-forge Cached + libffi 3.4.2 h3422bc3_5 conda-forge Cached + libzlib 1.2.13 h53f4e23_5 conda-forge Cached + ca-certificates 2023.7.22 hf0a4a13_0 conda-forge Cached + python_abi 3.11 4_cp311 conda-forge Cached + libcxx 16.0.6 h4653b0c_0 conda-forge Cached + bzip2 1.0.8 h3422bc3_4 conda-forge Cached + readline 8.2 h92ec313_1 conda-forge Cached + libsqlite 3.43.0 hb31c410_0 conda-forge Cached + tk 8.6.12 he1e0b03_0 conda-forge Cached + openssl 3.1.3 h53f4e23_0 conda-forge Cached + gmp 6.2.1 h9f76cd9_0 conda-forge Cached + mpfr 4.2.0 he09a6ba_0 conda-forge Cached + mpc 1.3.1 h91ba8db_0 conda-forge Cached + tzdata 2023c h71feb2d_0 conda-forge Cached + python 3.11.5 h47c9636_0_cpython conda-forge Cached + wheel 0.41.2 pyhd8ed1ab_0 conda-forge Cached + setuptools 68.2.2 pyhd8ed1ab_0 conda-forge Cached + pip 23.2.1 pyhd8ed1ab_0 conda-forge Cached + mpmath 1.3.0 pyhd8ed1ab_0 conda-forge Cached + gmpy2 2.1.2 py311h2ba9262_1 conda-forge Cached + sympy 1.12 pypyh9d50eac_103 conda-forge Cached Summary: Install: 24 packages Total download: 0 B ──────────────────────────────────────────────────────────────────────────── Transaction starting Linking xz-5.2.6-h57fd34a_0 Linking libexpat-2.5.0-hb7217d7_1 Linking ncurses-6.4-h7ea286d_0 Linking libffi-3.4.2-h3422bc3_5 Linking libzlib-1.2.13-h53f4e23_5 Linking ca-certificates-2023.7.22-hf0a4a13_0 Linking python_abi-3.11-4_cp311 Linking libcxx-16.0.6-h4653b0c_0 Linking bzip2-1.0.8-h3422bc3_4 Linking readline-8.2-h92ec313_1 Linking libsqlite-3.43.0-hb31c410_0 Linking tk-8.6.12-he1e0b03_0 Linking openssl-3.1.3-h53f4e23_0 Linking gmp-6.2.1-h9f76cd9_0 Linking mpfr-4.2.0-he09a6ba_0 Linking mpc-1.3.1-h91ba8db_0 Linking tzdata-2023c-h71feb2d_0 Linking python-3.11.5-h47c9636_0_cpython Linking wheel-0.41.2-pyhd8ed1ab_0 Linking setuptools-68.2.2-pyhd8ed1ab_0 Linking pip-23.2.1-pyhd8ed1ab_0 Linking mpmath-1.3.0-pyhd8ed1ab_0 Linking gmpy2-2.1.2-py311h2ba9262_1 Linking sympy-1.12-pypyh9d50eac_103 Transaction finished To activate this environment, use: micromamba activate /private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/.CondaPkg/env Or to execute a single command in this environment, use: micromamba run -p /private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/.CondaPkg/env mycommand julia> pyexec("x = 1", Main) julia> pyeval("x", Main) Python: 1 julia> @pyexec "y = 2" julia> @pyeval "y" ERROR: Python: NameError: name 'y' is not defined Python stacktrace: [1] @ REPL[7]:1:1 Stacktrace: [1] pythrow() @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/err.jl:94 [2] errcheck @ ~/.julia/packages/PythonCall/qTEA1/src/err.jl:10 [inlined] [3] pycallargs(f::Py, args::Py) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:210 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:228 [5] pycall @ ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:218 [inlined] [6] #_#11 @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [7] Py @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [8] pyeval(::Type{Py}, code::Py, globals::Module, locals::Tuple{}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:43 [9] top-level scope @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:146 julia> @pyexec "y" ERROR: Python: NameError: name 'y' is not defined Python stacktrace: [1] @ REPL[8]:1:1 Stacktrace: [1] pythrow() @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/err.jl:94 [2] errcheck @ ~/.julia/packages/PythonCall/qTEA1/src/err.jl:10 [inlined] [3] pycallargs(f::Py, args::Py) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:210 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:228 [5] pycall @ ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:218 [inlined] [6] #_#11 @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [7] Py @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [8] pyexec(::Type{Nothing}, code::Py, globals::Module, locals::Tuple{}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:98 [9] top-level scope @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:146 julia> @pyexec """ def f(x): return x + 1 """ julia> @pyeval "f(1)" ERROR: Python: NameError: name 'f' is not defined Python stacktrace: [1] @ REPL[10]:1:1 Stacktrace: [1] pythrow() @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/err.jl:94 [2] errcheck @ ~/.julia/packages/PythonCall/qTEA1/src/err.jl:10 [inlined] [3] pycallargs(f::Py, args::Py) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:210 [4] pycall(::Py, ::Py, ::Vararg{Py}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:228 [5] pycall @ ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:218 [inlined] [6] #_#11 @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [7] Py @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [8] pyeval(::Type{Py}, code::Py, globals::Module, locals::Tuple{}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:43 [9] top-level scope @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:146 julia> pyeval("f(1)", Main) ERROR: Python: NameError: name 'f' is not defined Python stacktrace: [1] @ :1 Stacktrace: [1] pythrow() @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/err.jl:94 [2] errcheck @ ~/.julia/packages/PythonCall/qTEA1/src/err.jl:10 [inlined] [3] pycallargs(f::Py, args::Py) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:210 [4] pycall(::Py, ::String, ::Vararg{Any}; kwargs::Base.Pairs{Symbol, Union{}, Tuple{}, NamedTuple{(), Tuple{}}}) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:228 [5] pycall @ ~/.julia/packages/PythonCall/qTEA1/src/abstract/object.jl:218 [inlined] [6] #_#11 @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [7] Py @ ~/.julia/packages/PythonCall/qTEA1/src/Py.jl:341 [inlined] [8] pyeval @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:43 [inlined] [9] pyeval @ ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:47 [inlined] [10] pyeval(code::String, globals::Module) @ PythonCall ~/.julia/packages/PythonCall/qTEA1/src/concrete/code.jl:47 [11] top-level scope @ REPL[11]:1 julia> @pyeval "x" Python: 1 (jl_NIT4tY) pkg> st -m Status `/private/var/folders/hc/fn82kz1j5vl8w7lwd4l079y80000gn/T/jl_NIT4tY/Manifest.toml` [992eb4ea] CondaPkg v0.2.20 [9a962f9c] DataAPI v1.15.0 [e2d170a0] DataValueInterfaces v1.0.0 [82899510] IteratorInterfaceExtensions v1.0.0 [692b3bcd] JLLWrappers v1.5.0 [0f8b85d8] JSON3 v1.13.2 [1914dd2f] MacroTools v0.5.11 [0b3b1443] MicroMamba v0.1.14 [bac558e1] OrderedCollections v1.6.2 [69de0a69] Parsers v2.7.2 [fa939f87] Pidfile v1.3.0 [aea7be01] PrecompileTools v1.2.0 [21216c6a] Preferences v1.4.1 [6099a3de] PythonCall v0.9.14 [ae029012] Requires v1.3.0 [6c6a2e73] Scratch v1.2.0 [856f2bd8] StructTypes v1.10.0 [3783bdb8] TableTraits v1.0.1 [bd369af6] Tables v1.11.0 [e17b2a0c] UnsafePointers v1.0.0 [f8abcde7] micromamba_jll v1.4.9+0 [0dad84c5] ArgTools v1.1.1 [56f22d72] Artifacts [2a0f44e3] Base64 [ade2ca70] Dates [f43a241f] Downloads v1.6.0 [7b1f6079] FileWatching [b77e0a4c] InteractiveUtils [4af54fe1] LazyArtifacts [b27032c2] LibCURL v0.6.3 [76f85450] LibGit2 [8f399da3] Libdl [37e2e46d] LinearAlgebra [56ddb016] Logging [d6f4376e] Markdown [a63ad114] Mmap [ca575930] NetworkOptions v1.2.0 [44cfe95a] Pkg v1.9.2 [de0858da] Printf [3fa0cd96] REPL [9a3f8284] Random [ea8e919c] SHA v0.7.0 [9e88b42a] Serialization [6462fe0b] Sockets [fa267f1f] TOML v1.0.3 [a4e569a6] Tar v1.10.0 [8dfed614] Test [cf7118a7] UUIDs [4ec0a83e] Unicode [e66e0078] CompilerSupportLibraries_jll v1.0.5+0 [deac9b47] LibCURL_jll v7.84.0+0 [29816b5a] LibSSH2_jll v1.10.2+0 [c8ffd9c3] MbedTLS_jll v2.28.2+0 [14a3606d] MozillaCACerts_jll v2022.10.11 [4536629a] OpenBLAS_jll v0.3.21+4 [83775a58] Zlib_jll v1.2.13+0 [8e850b90] libblastrampoline_jll v5.8.0+0 [8e850ede] nghttp2_jll v1.48.0+0 [3f19e933] p7zip_jll v17.4.0+0 julia> versioninfo() Julia Version 1.9.3 Commit bed2cd540a1 (2023-08-24 14:43 UTC) Build Info: Official https://julialang.org/ release Platform Info: OS: macOS (arm64-apple-darwin22.4.0) CPU: 8 × Apple M2 WORD_SIZE: 64 LIBM: libopenlibm LLVM: libLLVM-14.0.6 (ORCJIT, apple-m1) Threads: 1 on 4 virtual cores Environment: JULIA_EDITOR = code ```
LilithHafner commented 9 months ago

I see now that this is intended behavior; however, perhaps it makes sense to change this intended behavior so that @pyeval and pyeval have the same default with respect to what the local variables are. The default for pyeval being locals=globals and for @pyeval being locals=() is surprising.

cjdoris commented 9 months ago

The rationale for the existing behaviour is:

All the py* functions in PythonCall behave identically to the corresponding Python function (with functionality extended in sensible ways when inputs are not Python objects.)

On the other hand, @pyexec is mostly useful when called many times because it parses and compiles the Python code only once. So it's unlikely that @pyexec is being used in global Julia scope, so it doesn't make much sense for the Python code to be executed in global scope either.

So I don't think it makes much sense to change the behaviour of either. But perhaps the documentation could be improved to clarify the distinction between pyeval, pyexec, @pyeval and @pyexec.