Running cx_Freeze on prisma python doesn't work across systems (windows tested only)

[ntindle]

Bug description

For background, cx_freeze works by freezing all the required files for a Python program into an installable and executable package. When you run cx_freeze over a Prisma python project, it fails to copy in some files and errors on the following.

Traceback (most recent call last):
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\src\cx_Freeze\cx_Freeze\initscripts\__startup__.py", line 141, in run
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\src\cx_Freeze\cx_Freeze\initscripts\console.py", line 25, in run
  File "autogpt_server\app.py", line 47, in <module>
  File "autogpt_server\app.py", line 35, in main
  File "autogpt_server\app.py", line 26, in run_processes
  File "autogpt_server\app.py", line 22, in run_processes
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\process.py", line 50, in start
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\process.py", line 28, in execute_run_command
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\util\service.py", line 64, in run
  File "C:\hostedtoolcache\windows\Python\3.10.11\x64\lib\asyncio\base_events.py", line 649, in run_until_complete
  File "D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\autogpt_server\data\db.py", line 10, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\_base_client.py", line 430, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\_query.py", line 349, in connect
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\_query.py", line 58, in _ensure_file
  File "C:\Users\runneradmin\AppData\Local\pypoetry\Cache\virtualenvs\autogpt-server-Hzddpd5Y-py3.10\lib\site-packages\prisma\engine\utils.py", line 111, in ensure
prisma.engine.errors.BinaryNotFoundError: Expected C:\Users\nicka\prisma-query-engine-windows.exe, C:\Users\nicka\.cache\prisma-python\binaries\5.11.0\efd2449663b3d73d637ea1fd226bafbcf45b3102\prisma-query-engine-windows.exe or D:\a\AutoGPT\AutoGPT\rnd\autogpt_server\prisma\node_modules\prisma\query-engine-windows.exe to exist but none were found or could not be executed.
Try running prisma py fetch

You can set the params of prisma python using the following env when freezing to get the files included, and again when loading the binary but this still fails to run. Settings in the build pipeline

PRISMA_BINARY_CACHE_DIR: "./prisma"
PRISMA_HOME_DIR: "./prisma"

$env:PRISMA_BINARY_CACHE_DIR = "C:\Users\nicka\AppData\Local\Programs\AutoGPTServer\prisma\node_modules\prisma"
$env:PRISMA_HOME_DIR = "C:\Users\nicka\AppData\Local\Programs\AutoGPTServer\prisma"

Then rename query-engine-windows.exe to prisma-query-engine-windows.exe in prisma/node_modules/prisma

Error:

{"is_panic":false, "message": "Error querying the database: Error code 14: Unable to open the database file", "backtrace":" 0: <unknown>\n   1: <unknown>\n   2: <unknown>\n   3: <unknown>\n   4: <unknown>\n   5: <unknown>\n   6: <unknown>\n   7: <unknown>\n   8: BaseThreadInitThunk\n   9: RtlUserThreadStart\n"}

How to reproduce

Quite detailed reproduction steps and repo here: ntindle/cx_freeze_prisma_reproduction

Expected behavior

cx_Freezing a Prisma project works easily without configuration and can be run without configuration. Otherwise, with config, it can work on a machine that did not have Prisma installed before

Prisma information

datasource db {
  provider = "sqlite"
  url      = "file:./database.db"
}

generator client {
  provider             = "prisma-client-py"
  recursive_type_depth = 5
  interface            = "asyncio"
}

// This model describes the Agent Graph/Flow (Multi Agent System).
model Test {
  id   String  @id @default(uuid())
  name String?
}

Environment & setup

• OS: Windows
• Database: SQLite
• Python version: 3.10
• Prisma version:

prisma                  : 5.11.0
prisma client python    : 0.13.1
platform                : windows
expected engine version : efd2449663b3d73d637ea1fd226bafbcf45b3102
installed extras        : []
install path            : C:\Users\nicka\AppData\Local\pypoetry\Cache\virtualenvs\cx-freeze-prisma-reproduction-XPv0A5FO-py3.11\Lib\site-packages\prisma
binary cache dir        : C:\Users\nicka\.cache\prisma-python\binaries\5.11.0\efd2449663b3d73d637ea1fd226bafbcf45b3102

[ntindle]

https://github.com/marcelotduarte/cx_Freeze/discussions/2504

[ntindle]

You can see our attempts at debugging this in https://github.com/Significant-Gravitas/AutoGPT/actions/runs/9969740139/job/27547269902?pr=7406 in this pipeline with the msi artifact available here: https://github.com/Significant-Gravitas/AutoGPT/actions/runs/9969740139?pr=7406

https://github.com/Significant-Gravitas/AutoGPT/pull/7406

[ntindle]

Repro steps and repo here: https://github.com/ntindle/cx_freeze_prisma_reproduction

It is worth pointing out you MUST build in ci (or another PC), as your paths will be valid from the poetry install in the repo once you run cx_freeze.

[ntindle]

Link: https://github.com/marcelotduarte/cx_Freeze/issues/2509

[RobertCraigie]

Hey @ntindle, thanks for the detailed reproduction steps, it's much appreciated.

Unfortunately I won't have time to debug this deeply soon but I suspect you should be able to reproduce this locally by deleting / moving the downloaded binaries to a separate dir before running the app. This should replicate a "fresh install" environment, like any users of your app would have.

You can identify where the binaries are by running prisma py version & it'll be the last entry shown, binary cache dir.

Please let me know if I can be of any more help.

[ntindle]

Any idea how you would run the prisma py version with the correct python

[RobertCraigie]

using python -m as a prefix works for me on macos, e.g. python -m prisma py version

[ntindle]

That uses the version from cx freeze?

[RobertCraigie]

ah sorry I misunderstood, that shouldn't matter for just determining the location of the binary cache as it's just dependent on the current user.

[ntindle]

So with @marcelotduarte 's help I was able to get the DB "running" but hit

{"is_panic":false,"message":"Error querying the database: Error code 14: Unable to open the database file","backtrace":"   0: <unknown>\n   1: <unknown>\n   2: <unknown>\n   3: <unknown>\n   4: <unknown>\n   5: <unknown>\n   6: <unknown>\n   7: <unknown>\n   8: BaseThreadInitThunk\n   9: RtlUserThreadStart\n"}

[RobertCraigie]

@ntindle you will likely have to apply migrations (i.e. creating the database file) before attempting to connect, are you doing that?

[ntindle]

I'm attempting that and it seems to work with Postgres but not SQLite

[RobertCraigie]

👍 I suspect there's some mismatch with where the sqlite db is created.

Unfortunately it looks like there currently aren't any debug logs showing the file path. Will fix that.

You could try manually specifying the absolute file path programmatically, https://prisma-client-py.readthedocs.io/en/stable/reference/client/#datasource-overriding.