second python execution fails when embedding

[61eec7a8-fcd0-4d73-837f-4924329a77a7]

BPO	17408
Nosy	@amauryfa, @ncoghlan, @pitrou, @ned-deily, @bitdancer, @berkerpeksag, @phmc
Files	Python33Test.zip

^{Note: these values reflect the state of the issue at the time it was migrated and might not reflect the current state.}

Show more details

GitHub fields: ```python assignee = None closed_at = created_at = labels = ['interpreter-core', 'type-crash'] title = 'second python execution fails when embedding' updated_at = user = 'https://bugs.python.org/theDarkBrainer' ``` bugs.python.org fields: ```python activity = actor = 'Denny Weinberg' assignee = 'none' closed = True closed_date = closer = 'pitrou' components = ['Interpreter Core'] creation = creator = 'theDarkBrainer' dependencies = [] files = ['29398'] hgrepos = [] issue_num = 17408 keywords = [] message_count = 15.0 messages = ['184081', '184082', '184083', '184085', '184261', '184299', '188393', '188395', '188427', '188428', '265851', '269374', '269377', '269378', '269380'] nosy_count = 13.0 nosy_names = ['amaury.forgeotdarc', 'ncoghlan', 'pitrou', 'ned.deily', 'Arfrever', 'r.david.murray', 'palm.kevin', 'python-dev', 'berker.peksag', 'pconnell', 'theDarkBrainer', 'isoschiz', 'Denny Weinberg'] pr_nums = [] priority = 'normal' resolution = 'fixed' stage = 'resolved' status = 'closed' superseder = None type = 'crash' url = 'https://bugs.python.org/issue17408' versions = ['Python 3.3', 'Python 3.4', 'Python 3.5'] ```

[61eec7a8-fcd0-4d73-837f-4924329a77a7]

This issue is for Python3.3 and doesn't exist in Python3.2

Detailed description with source code can be found here: http://stackoverflow.com/questions/15387035/second-python-execution-fails

[ned-deily]

Please add the detailed description of the problem and any test files to the issue here. Information stored off-site is not searchable within the issue tracker and may not be permanently available.

[61eec7a8-fcd0-4d73-837f-4924329a77a7]

I'm trying to embed the python 3.3 engine for an app that need to run custom scripts in python. Since the scripts might be completely different, and sometimes user provided, I am trying to make each execution isolated and there is not need to preserve any data between execution of the different scripts.

So, my solution is to wrap each execution between 'Py_Initialize' and 'Py_Finalize'. It looks something like that:

  void ExecuteScript(const char* script)
  {
    Py_Initialize();

    PyRun_SimpleString( script );

    Py_Finalize();
  }

However, this fails for a particular python script the second time a script is executed with:

  done!
  Traceback (most recent call last):
    File "<string>", line 8, in <module>
    File "\Python33Test\Output\Debug\Python33\Lib\copy.py", line 89, in copy
      rv = reductor(2)
  TypeError: attribute of type 'NoneType' is not callable

The python script looks like this:

  class Data:
      value1 = 'hello'
      value2 = 0

  import copy

  d = Data()
  dd = copy.copy( d )
  print ( 'done!' )

As you can see, the first time around the script was executed the 'done!' was printed out. But the second time it rises an exception inside the copy function.

It looks like the python engine was left in some weird state after the first initialize-finalize. Note, this is python 3.3.

Also, it is very interesting to note that Python 2.7 and Python 3.2 did not have this problem.

I guess there might be other examples that could reveal better what's going, but i haven't had the time to find yet.

I also put a copy of the project containing flag to switch between Python 3 and Python 2.7 (the file is 31 MB): https://docs.google.com/file/d/0B86-G0mwwxZvbWRldTd5b2NNMWM/edit?usp=sharing

Thanks, Vlad

[amauryfa]

Reproduced on Linux. The reason is in Objects/typeobject.c: import_copyreg() has a static cache of the copyreg module. When the interpreter stops, the module is filled with None... but gets reused in the next instance.

Resetting this "mod_copyreg" variable to NULL fixes the issue. pickle is also broken for the same reason.

[terryjreedy]

Can you tell if the relevant 3.2 to 3.3 change is in Py_Initialize, Py_Finalize(), or typeobject.c?

[amauryfa]

In 3.2, typeobject.c did not cache the copyreg module in import_copyreg(); PyImport_Import was always called.

[1762cc99-3127-4a62-9baf-30c3d0f51ef7]

New changeset 8c1385205a35 by Antoine Pitrou in branch '3.3': Issue bpo-17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again. http://hg.python.org/cpython/rev/8c1385205a35

New changeset 0b34fd75b937 by Antoine Pitrou in branch 'default': Issue bpo-17408: Avoid using an obsolete instance of the copyreg module when the interpreter is shutdown and then started again. http://hg.python.org/cpython/rev/0b34fd75b937

[pitrou]

Thank you for reporting! This should be fixed now.

[80036ac5-bb84-4d39-8416-02cd8e51707d]

Commit 8c1385205a35 causes segmentation fault (in non-debug build) or abort (in debug build) during interpreter shutdown after copying of e.g. bytes or object().

$ python -c 'import copy; copy.copy(b""); print("text")'
text
Segmentation fault
$ python -c 'import copy; copy.copy(object()); print("text")'
text
Segmentation fault

[pitrou]

Should be fixed in 7de9852cdc0e, sorry.

[85052b60-879f-45f8-9814-2edf855bcc5e]

Hi,

I think that the problem exists also in python 3.5.1

After calling Py_Finalize and Py_Initialize I get the message "attribute of type 'NoneType' is not callable" on the datetime.strptime method.

Example: from datetime import datetime s = '20160505 160000' refdatim = datetime.strptime(s, '%Y%m%d %H%M%S')

The first call works find but it crashes after the re initialization.

Workaround: from datetime import datetime s = '20160505 160000' try: refdatim = datetime.strptime(s, '%Y%m%d %H%M%S') except TypeError: import time refdatim = datetime.fromtimestamp(time.mktime(time.strptime(s, '%Y%m%d %H%M%S')))

Can anyone confirm this bug? Can you tell me if this will be fixed for python 3.5/3.6/...?

All the other modules are working find after the re initialization but datetime.strptime.

[85052b60-879f-45f8-9814-2edf855bcc5e]

Can we please reopen this issue?

[berkerpeksag]

We don't re-open old issues (this was closed more than 3 years ago). Please open a new issue and provide a minimal reproducer. Thanks!

[bitdancer]

The interpreter is not crashing in your case, so this is a mostly-unrelated problem. (The part that is related is that it is triggered by module finalization.) As Berker said, please open a new issue, but be warned that it may get closed as a being addressed by the issue for rewriting the python startup sequence (PEP-432 I think, which I think Nick has resurrected), since this is a systemic problem. On the other hand there might be a way to fix it in the datetime module if someone decides it is worth doing.

[85052b60-879f-45f8-9814-2edf855bcc5e]

Ok,

thank you very much for your comments.

See bpo-27400