Bug: Cannot Use `"` When Calling Python Function using API

This is caused by I'm wrapping commands using ", then call exec to run the job. (Don't know why I'm doing this, probably because this is the least changed made compare to the old version. Actually, the previous method works just fine, my bad decision :upside_down_face:. Totally forgot to consider this while fixing another bug. )

Won't affect API.

Method 1 -- Use Triple Quotes

To Fix

[x] Detect whether to use ''' or """ before wrapping.
[x] yt_run_FunctionArguments (yt_run.cpp)
[x] yt_run_Function (yt_run.cpp)
[x] func_status_list::run_func (func_status_list.cpp)
[x] Update doc
- [x] Mention that if user were to use triple quotes in their function, stick to either ''' or """ when using yt_run_FunctionArguments/interactive mode.
- Ex:
  - :heavy_check_mark: using the same triple quotes in arguments.
```
yt_run_FunctionArguments("funcArgs", "'''a'''", "'''b'''");
```
  - :heavy_multiplication_x: using different triple quotes in arguments.
```
yt_run_FunctionArguments("funcArgs", "'''a'''", "\"\"\"b\"\"\"");
```

libyt Dev Notes

Use PyRun_SimpleString for definite, correct results.

This API is not for handling the errors. It will return failure (which is returning non-zero) even if you wrapped the code inside try/except.

Example:

std::string Command1("try:\n"
                   "    import inline, sys\n"
                   "    exec(\"func(\"3\")\", sys.modules['inline'].__dict__)\n"
                   "except Exception as e:\n"
                   "    var1 = traceback.format_exc()\n");
if (PyRun_SimpleString(Command1.c_str()) != 0)
{
  // we will still fall to this block even though var1 is set.
  std::cout << "Return != 0" << std::endl;
}

Though except block can successfully set error messages, we want to use PyRun_SimpleString != 0 to indicated real error when running definite correct python statement in libyt.

Method 2 -- Change to `PyEval_EvalCode`

After rethinking and testing for some time, I don't think we need it. Because it is unlikely that people will pass in triple quotes in function arguments, and there is always an alternatives.

Besides, while I was implementing it, I get some unwanted results. (See comment) So I decided to move this for later updates and enhancements.

Move implementing method 2 to

To Fix

[ ] Change libyt.interactive_mode["script_globals"] to libyt.general_info["script_namespace"]. This object will exist both inside normal mode and interactive mode.
[ ] Dealing with error message.
- [ ] Checking and syncing status between MPI processes.
- [ ] Storing error message and printing them in serial.
[ ] yt_run_FunctionArguments (yt_run.cpp)
[ ] yt_run_Function (yt_run.cpp)
[ ] func_status_list::run_func (func_status_list.cpp)

Test

Test again ... :smiling_face_with_tear:

[ ] Interactive Modes: Handling errors
[ ] Normal Modes: Abort on errors

Check

[ ] Check imported Python modules, some might not needed at all.

:yarn: I think we don't necessarily need to use PyEval_EvalCode at here, since there is alternatives and nobody will pass """ as function arguments. So I'm just putting some tests and code snippets while trying to use PyEval_EvalCode to solve the problem.

This is just a code snippet trying to extract traceback stack error message using Python C API PrintPyError_C_API, or let python handles it PrintError_Python. (I didn't free some of their reference, since this is just a snippet.)

Notes

Using PyEval_EvalCode to run compiled object, and use the following to print the error msg:

PrintPyError_C_API: just won't work, even if I follow this.
PrintPyError_Python: also won't work.

Code

#define PY_SSIZE_T_CLEAN
#include <Python.h>
#include <string>
#include <iostream>

// global
PyObject *g_py_namespace;
PyObject *g_err_msg_dict;

static void PrintCommand(const std::string& command)
{
    std::cout << "===============================" << std::endl;
    std::cout << "COMMAND : " << command << std::endl;
}

static void PrintPyError_C_API()
{
    // fetch error
    PyObject *py_exc, *py_val, *py_traceback;
    PyErr_Fetch(&py_exc, &py_val, &py_traceback); // steal ref

    // importing traceback
    PyObject *py_module_name = PyUnicode_FromString("traceback");
    PyObject *py_module_traceback = PyImport_Import(py_module_name); // new ref
    Py_DECREF(py_module_name);
    std::cout << "#FLAG --> Import " << PyUnicode_AsUTF8(PyObject_GetAttrString(py_module_traceback, "__name__")) << std::endl;

    // getting traceback.format_exception()
    PyObject *py_format_exception = PyObject_GetAttrString(py_module_traceback, "format_exception"); // new ref
    if (py_format_exception && PyCallable_Check(py_format_exception))
    {
        // call traceback.format_exception()
        PyObject *py_args = PyTuple_New(3); // new ref
        PyTuple_SetItem(py_args, 0, py_exc);
        PyTuple_SetItem(py_args, 1, py_val);
        PyTuple_SetItem(py_args, 2, py_traceback);

        PyObject *py_err_value = PyObject_CallObject(py_format_exception, py_args);  // new ref

        std::cout << "+++++++++++++++++++++++++++++" << std::endl;
        if (py_err_value != NULL) 
        {
            std::cout << "py_err_value is not NULL" << std::endl;
        }
        else 
        {
            std::cout << "py_err_value is NULL" << std::endl;
        }
        std::cout << "+++++++++++++++++++++++++++++" << std::endl;
    }

}

static void PrintPyError_Python()
{
    std::cout << "++++++++++++++++++++++++++++++" << std::endl;
    PyRun_SimpleString("print('#FLAG --> globals = ', globals());");
    const char *command = "import traceback; print(traceback.format_exc());";
    PyRun_SimpleString(command);
    std::cout << "++++++++++++++++++++++++++++++" << std::endl;
}

static void RunPython(const std::string& code_str) 
{
    PrintCommand(code_str);
    PyObject *py_src = Py_CompileString(code_str.c_str(), "<libyt API>", Py_single_input);
    PyObject *py_dum;
    if (py_src != NULL) {
        py_dum = PyEval_EvalCode(py_src, g_py_namespace, g_py_namespace);
        if (PyErr_Occurred()) 
        {
            PrintPyError_Python();
        }
    }
    else 
    {
        PrintPyError_Python();
    }

    std::cout << "=============================" << std::endl;
}

int main(int argc, char *argv[])
{
    wchar_t *program = Py_DecodeLocale(argv[0], NULL);
    if (program == NULL) {
        fprintf(stderr, "Fatal error: cannot decode argv[0]\n");
        exit(1);
    }
    Py_SetProgramName(program);  /* optional but recommended */
    Py_Initialize();

    // Getting sys.modules using Python C API
    PyRun_SimpleString("import sys; sys.path.insert(0, '.'); import inline_script;");
    PyObject *py_sys = PyImport_AddModule("sys"); // borrowed ref
    PyObject *py_sys_module = PyDict_GetItemString(PyModule_GetDict(py_sys), "modules"); // both are borrowed ref
    g_py_namespace = PyModule_GetDict(PyDict_GetItemString(py_sys_module, "inline_script")); // borrowed ref

    // Compile and run code under this namespace [temporary: this is only for reproducing error msg]
    std::string code_str;

    // Handling Errors -- inside inline script itself
    // It will get the wrong line number if the imported script has been changed.
    // int temp_input;
    // std::cin >> temp_input;
    code_str = std::string("func()");
    RunPython(code_str);

    PyRun_SimpleString("print('HERE')");

    // Exit python
    Py_Finalize();

    return 0;
}

Output

===============================
COMMAND : func()
I'm inside inline_script.py
++++++++++++++++++++++++++++++
++++++++++++++++++++++++++++++
=============================
Traceback (most recent call last):
  File "<libyt API>", line 1, in <module>
  File "/home/cindytsai/Documents/GitHub/EmbeddedPython38/LoadModule/./inline_script.py", line 3, in func
    print(nothing)
NameError: name 'nothing' is not defined

Desire Output

===============================
COMMAND : func()
I'm inside inline_script.py
++++++++++++++++++++++++++++++
(Error Msg)
++++++++++++++++++++++++++++++
=============================

yt-project / libyt