python3 package should also have python39*.zip placed in the output with python39.dll so that embedded interpreters can find imports, etc.

[MichaelGoulding]

If you try to compile a project that embeds the python interpreter, Python needs to be able to find the python system libraries.

It looks in the following places:

In a debug build:

\python39_d.zip;.\DLLs;.\lib; Can we get python3 to copy the appropriate python39.zip or python39_d.zip to the output along with the DLL?

[JackBoosY]

Confirmed.

[JackBoosY]

I found that all the files in the compressed package are in Lib and are not compressed. Is this reasonable?

[Hoikas]

The project that embeds the interpreter needs to package the standard library in whatever format it expects. Not all embedding applications will use a zip file--some may use custom PEP 451 machinery--therefore it is inappropriate for the python3 port to enforce any decision in this regard. The standard library can be found using FindPython's Python_STDLIB result variable. If you need a zip, you can create this as part of your build process, perhaps using CMake's file(ARCHIVE_CREATE

[MichaelGoulding]

Is there some way to make it easier to understand what to do?

For me, I added pybind11 to my project, but I would get what seemed to be obscure errors when I tried to run the resulting executable.

Get Outlook for Androidhttps://aka.ms/ghei36

---

From: Adam Johnson @.> Sent: Wednesday, June 30, 2021 4:09:32 PM To: microsoft/vcpkg @.> Cc: Michael Goulding @.>; Author @.> Subject: Re: [microsoft/vcpkg] python3 package should also have python39*.zip placed in the output with python39.dll so that embedded interpreters can find imports, etc. (#18719)

The project that embeds the interpreter needs to package the standard library in whatever format it expects. Not all embedding applications will use a zip file--some may use custom PEP 451 machinery--therefore it is inappropriate for the python3 port to enforce any decision in this regard. The standard library can be found using FindPython's Python_STDLIB result variablehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcmake.org%2Fcmake%2Fhelp%2Flatest%2Fmodule%2FFindPython.html%23result-variables&data=04%7C01%7C%7C1dedc015e54e414c0f8708d93c1c1ef0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637606913734089408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=rnZK1Y%2Bm69wKpKACeIiKIcxIkFobJXxL2VXCb7vMYoA%3D&reserved=0. If you need a zip, you can create this as part of your build process, perhaps using CMake's file(ARCHIVE_CREATEhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcmake.org%2Fcmake%2Fhelp%2Flatest%2Fcommand%2Ffile.html%23archive-create&data=04%7C01%7C%7C1dedc015e54e414c0f8708d93c1c1ef0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637606913734089408%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=1SQbZB8gR9gaRWuMcacqWpuUu3oo0Y2AqfUNVEHKRt8%3D&reserved=0

— You are receiving this because you authored the thread. Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fvcpkg%2Fissues%2F18719%23issuecomment-871783946&data=04%7C01%7C%7C1dedc015e54e414c0f8708d93c1c1ef0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637606913734099401%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=pvdnvAgokUpwmFQ231SUGoL7pI8fFQNykc5z656dXCI%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEJ5SB3K3G5ZJRORIGAOGH3TVOP2ZANCNFSM47Q4NTTQ&data=04%7C01%7C%7C1dedc015e54e414c0f8708d93c1c1ef0%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637606913734099401%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=5SVm84nI1gwECD8Qx0lT7cLCw9jguy46AAcaiWNqj7Y%3D&reserved=0.

[JackBoosY]

@MichaelGoulding After comparing the python3 installed by vcpkg with the python3 installed by the installation package, I found the following items:

1. The python script (.py) in Lib is compressed into python39.zip.
2. Generated several extra binaries (venv*.exe and _freeze_importlib.exe).
3. Several pyd scripts prefixed with _ are missing in the python3 root directory.
4. Dependent libraries are installed in the DLLs instead of in the root directory.
5. Added pip3 binary to Scripts.

So what I want to confirm is:

1. Whether the script installed in Lib can be called directly without being compressed.
2. Whether the missing pyd script in item 3 is required.

[LilyWangLL]

I think this issue should be a port feature.

[Hoikas]

This issue is bogus and should be closed.

1. Whether the script installed in Lib can be called directly without being compressed.

python39.zip is only used by certain embedded interpreters. The default is for all the scripts in Lib to be uncompressed. Embedders are responsible for packaging their own Python stdlib; vcpkg should not interfere with this.

2. Whether the missing pyd script in item 3 is required.

All required pyd (DLL) files are included in the vcpkg distribution.

[LilyWangLL]

Ping @JackBoosY, could you help confirm this issue should be closed?

[MichaelGoulding]

If one links to the embedded python target, cmake should include it.

---

From: Adam Johnson @.> Sent: Friday, January 28, 2022 3:12:41 PM To: microsoft/vcpkg @.> Cc: Michael Goulding @.>; Mention @.> Subject: Re: [microsoft/vcpkg] python3 package should also have python39*.zip placed in the output with python39.dll so that embedded interpreters can find imports, etc. (#18719)

This issue is bogus and should be closed.

1. Whether the script installed in Lib can be called directly without being compressed.

python39.zip is only used by certain embedded interpreters. The default is for all the scripts in Lib to be uncompressed. Embedders are responsible for packaging their own Python stdlib; vcpkg should not interfere with this.

1. Whether the missing pyd script in item 3 is required.

All required pyd (DLL) files are included in the vcpkg distribution.

— Reply to this email directly, view it on GitHubhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fmicrosoft%2Fvcpkg%2Fissues%2F18719%23issuecomment-1024732690&data=04%7C01%7C%7C46a01c3d359646607d5008d9e2b3afa9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637790083648626752%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=ZaVyOo3XBg6kJnpitGwSMQf1A0ky1KyNmAhl3IBVE3A%3D&reserved=0, or unsubscribehttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fnotifications%2Funsubscribe-auth%2FAEJ5SB56CLNUPR7B73TV23LUYMPGTANCNFSM47Q4NTTQ&data=04%7C01%7C%7C46a01c3d359646607d5008d9e2b3afa9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637790083648626752%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=X%2BVXNKyGPJSO0b9FTsgXi04%2FgmaRyPyBemtBp9BE8Ho%3D&reserved=0. Triage notifications on the go with GitHub Mobile for iOShttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fapps.apple.com%2Fapp%2Fapple-store%2Fid1477376905%3Fct%3Dnotification-email%26mt%3D8%26pt%3D524675&data=04%7C01%7C%7C46a01c3d359646607d5008d9e2b3afa9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637790083648626752%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=B3OkAFxx0ZRsUhonW9olXN2t4luyQi9I9iGESonN7XI%3D&reserved=0 or Androidhttps://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fplay.google.com%2Fstore%2Fapps%2Fdetails%3Fid%3Dcom.github.android%26referrer%3Dutm_campaign%253Dnotification-email%2526utm_medium%253Demail%2526utm_source%253Dgithub&data=04%7C01%7C%7C46a01c3d359646607d5008d9e2b3afa9%7C84df9e7fe9f640afb435aaaaaaaaaaaa%7C1%7C0%7C637790083648626752%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000&sdata=QaogFIEaDC6q3yiN2cwY0z%2BjyHqyNxN7sAA%2BeTqOgr8%3D&reserved=0. You are receiving this because you were mentioned.Message ID: @.***>

[JackBoosY]

No we shouldn't close this issue since it's not fixed, the expected python tool should same with the pre-build one.

[Hoikas]

The vcpkg python tool is the same as the one installed by the installer on python.org

[GordonSmith]

The vcpkg python tool is the same as the one installed by the installer on python.org

But it is different from the "Windows embeddable package" on the same download page on python.org and I suspect a lot of folks here are interested in the embedded version (I am)?

[MichaelGoulding]

I think this issue should be a port feature.

I agree that a port feature might be appropriate for those using pybind to embed vs. those using pybind to create a native module.

[Osyotr]

But it is different from the "Windows embeddable package" on the same download page on python.org and I suspect a lot of folks here are interested in the embedded version (I am)?

The official embeddable package does not provide any libraries to link to. You can use it only to run python scripts. What you should really do is something like this (tested on linux, for windows you should do it at build step):

set(_py3_zip_temp_dir "${CMAKE_BINARY_DIR}/python_embeddable_zip_temp")
file(MAKE_DIRECTORY "${_py3_zip_temp_dir}")

install(
  CODE "set(_py3_zip_temp_dir \"${_py3_zip_temp_dir}\")"
  CODE "set(Python_STDLIB \"${Python_STDLIB}\")"
  CODE "set(Python_VERSION_MAJOR \"${Python_VERSION_MAJOR}\")"
  CODE "set(Python_VERSION_MINOR \"${Python_VERSION_MINOR}\")"
  CODE [[
    file(GLOB_RECURSE _py_stdlib_files RELATIVE "${Python_STDLIB}" "${Python_STDLIB}/*")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "__pycache__")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^config*/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^idlelib/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^lib2to3/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^lib-dynload/.*$") # Copied separately
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^lib-tk/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "^tkinter/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX "(^|/)tests?/.*$")
    list(FILTER _py_stdlib_files EXCLUDE REGEX ".*\\.pyo$")
    execute_process(
      COMMAND "${CMAKE_COMMAND}" -E tar cf "${_py3_zip_temp_dir}/python${Python_VERSION_MAJOR}${Python_VERSION_MINOR}.zip" --format=zip ${_py_stdlib_files}
      WORKING_DIRECTORY "${Python_STDLIB}"
    )
    file(INSTALL DESTINATION "${CMAKE_INSTALL_PREFIX}/lib" FILES "${_py3_zip_temp_dir}/python${Python_VERSION_MAJOR}${Python_VERSION_MINOR}.zip")
  ]]
)

install(
  DIRECTORY "${VCPKG_INSTALLED_DIR}/${VCPKG_TARGET_TRIPLET}/$<$<CONFIG:DEBUG>:debug/>lib/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}/lib-dynload"
  DESTINATION "${CMAKE_INSTALL_LIBDIR}/python${Python_VERSION_MAJOR}.${Python_VERSION_MINOR}"
  FILES_MATCHING
  PATTERN "*.debug" EXCLUDE
  PATTERN "*.so"
)

[GordonSmith]

@Osyotr - where did you get the EXCLUDE REGEX items from? I was building up a similar list via inspection between the python310.zip file and "Lib" contents?

Also I was planning on running ${Python3_EXECUTABLE} -m compileall -b . in the temp folder and excluding the .py files during the zip process?

...very helpful - thanks!...