OSError('could not get source code') under libpython

[nickmitchko]

Describe the bug Taichi cannot find sources when using a libpython environment.

To Reproduce I've provided a sample c program using libpython that shows the error.

Main.c

#include <dlfcn.h>
#include <stdio.h>
#include <stdlib.h>
#include "Python.h" 
int main(void)
{    
    Py_Initialize(); 
    if ( !Py_IsInitialized() ) 
    { 
        return -1; 
    }     printf("doing import math in main program:\n");
    PyRun_SimpleString("import math");
    PyRun_SimpleString("import sys");
    PyRun_SimpleString("print(sys.path)");
    PyRun_SimpleString("exec(open(\"/home/nmitchko/projects/taichi/taichi_nmitchko.py\").read())");
    Py_Finalize();
    return 0;
}

/home/nmitchko/projects/taichi/taichi_nmitchko.py

import taichi as ti
ti.init()
@ti.func
def is_prime(n: int):
    result = True
    for k in range(2, int(n ** 0.5) + 1):
        if n % k == 0:
            result = False
            break
    return result
@ti.kernel
def count_primes(n: int) -> int:
    count = 0
    for k in range(2, n):
        if is_prime(k):
            count += 1
    return count
print("primes before 1000: ",count_primes(1000))

Ubuntu 20.04 with gcc

$> gcc -o taichi_libpython main.c -I/usr/include/python3.8 -I/usr/include/python3.8 -L/usr/lib/python3.8/config-3.8-x86_64-linux-gnu -L/usr/lib -lpython3.8 -lcrypt -lpthread -ldl -lutil -lm -lm

$> ./taichi_libpython
doing import math in main program:
['/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/nmitchko/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
[Taichi] version 1.2.2, llvm 10.0.0, commit 608e4b57, linux, python 3.8.10
[Taichi] Starting on arch=x64
Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 150, in our_findsource
    return inspect._si_old_findsource(object)
  File "/usr/lib/python3.8/inspect.py", line 798, in findsource
    raise OSError('could not get source code')
OSError: could not get source codeDuring handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 96, in blender_findsourcetext
    import bpy
ModuleNotFoundError: No module named 'bpy'During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 153, in our_findsource
    return blender_findsource(object)
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 110, in blender_findsource
    lines, text_name = blender_findsourcetext(object)
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 98, in blender_findsourcetext
    raise IOError('Not in Blender environment!')
OSError: Not in Blender environment!

Log/Screenshots

doing import math in main program:
['/usr/lib/python38.zip', '/usr/lib/python3.8', '/usr/lib/python3.8/lib-dynload', '/home/nmitchko/.local/lib/python3.8/site-packages', '/usr/local/lib/python3.8/dist-packages', '/usr/lib/python3/dist-packages']
[Taichi] version 1.2.2, llvm 10.0.0, commit 608e4b57, linux, python 3.8.10
[Taichi] Starting on arch=x64
Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 150, in our_findsource
    return inspect._si_old_findsource(object)
  File "/usr/lib/python3.8/inspect.py", line 798, in findsource
    raise OSError('could not get source code')
OSError: could not get source codeDuring handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 96, in blender_findsourcetext
    import bpy
ModuleNotFoundError: No module named 'bpy'During handling of the above exception, another exception occurred:Traceback (most recent call last):
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 153, in our_findsource
    return blender_findsource(object)
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 110, in blender_findsource
    lines, text_name = blender_findsourcetext(object)
  File "/home/nmitchko/.local/lib/python3.8/site-packages/sourceinspect/__init__.py", line 98, in blender_findsourcetext
    raise IOError('Not in Blender environment!')
OSError: Not in Blender environment!```

**Additional comments**
Operating under libpython is important for use cases where another application (such as databases, CAD programs, etc) has embedded the python runtime using libpython. The ability to use taichi would enable speedups in libpython environments.

[strongoier]

cc: @neozhaoliang Is this problem similar to #4936?

[neozhaoliang]

Yes, it's similar to #4936. Looks like libpython opens an interpreter, and since this interpreter holds the Python string inside some virtual file in memory, Taichi's ast parser cannot find that source file.

Here is a short example to reproduce this:

import taichi as ti
ti.init()

x = ti.field(int, shape=10)

@ti.kernel
def test():
    for i in x:
        x[i] = i

test()

Save this code to test.py and in a python interactive shell, run

>>> with open("test.py", "r") as f:
...     exec(f.read())
... 

Taichi's compiler will report an error for this.

@nickmitchko I would suggest you do not run Taichi programs like this for now, and we shall discuss how to handle such user scenarios later.

[nickmitchko]

Yes, it's similar to #4936. Looks like libpython opens an interpreter, and since this interpreter holds the Python string inside some virtual file in memory, Taichi's ast parser cannot find that source file.

Here is a short example to reproduce this:

import taichi as ti
ti.init()

x = ti.field(int, shape=10)

@ti.kernel
def test():
    for i in x:
        x[i] = i

test()

Save this code to test.py and in a python interactive shell, run

>>> with open("test.py", "r") as f:
...     exec(f.read())
... 

Taichi's compiler will report an error for this.

@nickmitchko I would suggest you do not run Taichi programs like this for now, and we shall discuss how to handle such user scenarios later.

Gotcha, I'd be nice to have this sort of functionality but I understand supporting libpython's source loader is engineering effort.