Partial type generation does not include newly generated modules

[RobertCraigie]

Bug description

When using a partial type generator and a custom output directory, the partially generated package is not imported when the partial type generator is ran.

How to reproduce

schema.prisma

datasource db {
  provider = "postgres"
  url      = env("DB_URL")
}

generator client {
  provider               = "prisma-client-py"
  partial_type_generator = "partials.py"
  output                 = "my_prisma"
}

model User {
  id Int @id
  name String
}

partials.py

from prisma.models import User

On a fresh prisma installation (not generated), trying to generate the client will error.

prisma generate

Prisma schema loaded from schema.prisma
An exception ocurred while running the partial type generator
Error:
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "partials.py", line 1, in <module>
    from prisma.models import User
ModuleNotFoundError: No module named 'prisma.models'

Expected behavior

The prisma package should point to the newly generated package.

Should be noted that in the example given above, even trying to import from my_prisma leads to an error but even if it didn't the point still stands as sometimes it may be useful to generate the client to a directory that is multiple directories away and fixing this would require knowledge of where the client was being generated to and some horrible python path patching.

Solution

I do not know how this can be solved as it is due to Python's import caching mechanism, the importlib.reload may be useful here.

In terms of how to integrate a solution into this library you will have to modify the Module.run method in src/prisma/generator/models.py.

[RobertCraigie]

This is actually independent from the use of custom output, this can happen in some situations regardless.

[landeholt]

One solution could be to block partial_type_generator until all models are generated, and thus ensuring that they exist when partials is executing

[RobertCraigie]

@landeholt We already do this: https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/generator/generator.py#L213-L217

So the issue appears to be caused by something else :(

[Gnosnay]

i also find that after we generate prima, then we use one relative path to import the User from my_prisma, there is no partial types generated.

no idea about reason.

reproduce steps:

1. generate without partial type

schema.prisma

datasource db {
  provider = "postgres"
  url      = env("DB_URL")
}

generator client {
  provider               = "prisma-client-py"
  # partial_type_generator = "partials.py"
  output                 = "my_prisma"
}

model User {
  id Int @id
  name String
}

prisma generate

2. add the partial type back, and try to generate

datasource db {
  provider = "postgres"
  url      = env("DB_URL")
}

generator client {
  provider               = "prisma-client-py"
  partial_type_generator = "partials.py"
  output                 = "my_prisma"
}

model User {
  id Int @id
  name String
}

partials.py

from my_prisma.models import User

User.create_partial("UserOnlyName", include={"name"})

prisma generate

no UserOnlyName generated.

[Gnosnay]

May i know the timing of triggering the prisma generate partial types?

[Gnosnay]

I think i know the reason.

because the prisma i use in command line tool share different context with model i import.

model i import path is <custom path>/models

but the prisma path is <project_root>/.venv/lib/python3.11/site-packages/prisma

so the partial_models_ctx is not the same context https://github.com/RobertCraigie/prisma-client-py/blob/main/src/prisma/generator/generator.py#L255

[alexjball]

Is there a workaround for this issue? Currently on 0.12.0 when I specify an output and a type generator, I get an import error about a partially initialized module:

# schema.prisma
generator py_client {
  provider               = "prisma-client-py"
  previewFeatures        = ["multiSchema", "postgresqlExtensions"]
  output                 = "../components/foo/prisma_client"
  partial_type_generator = "prisma/prisma_partials.py"
  recursive_type_depth   = "5"
}

Prisma schema loaded from prisma/schema.prisma
Error: 
An exception ocurred while running the partial type generator
Traceback (most recent call last):
  File "/home/vscode/.cache/pypoetry/virtualenvs/R7CnQsZq-py3.10/lib/python3.10/site-packages/prisma/generator/models.py", line 312, in run
    loader.exec_module(mod)
  File "<frozen importlib._bootstrap_external>", line 883, in exec_module
  File "<frozen importlib._bootstrap>", line 241, in _call_with_frames_removed
  File "/workspaces/foo-asset/prisma/prisma_partials.py", line 2, in <module>
    from foo.prisma_client import models
  File "/workspaces/foo-asset/components/foo/prisma_client/__init__.py", line 26, in <module>
    from . import (
ImportError: cannot import name 'partials' from partially initialized module 'foo.prisma_client' (most likely due to a circular import) (/workspaces/foo-asset/components/foo/prisma_client/__init__.py)

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/home/vscode/.cache/pypoetry/virtualenvs/R7CnQsZq-py3.10/lib/python3.10/site-packages/prisma/generator/generator.py", line 108, in run
    self._on_request(request)
  File "/home/vscode/.cache/pypoetry/virtualenvs/R7CnQsZq-py3.10/lib/python3.10/site-packages/prisma/generator/generator.py", line 161, in _on_request
    self.generate(data)
  File "/home/vscode/.cache/pypoetry/virtualenvs/R7CnQsZq-py3.10/lib/python3.10/site-packages/prisma/generator/generator.py", line 253, in generate
    config.partial_type_generator.run()
  File "/home/vscode/.cache/pypoetry/virtualenvs/R7CnQsZq-py3.10/lib/python3.10/site-packages/prisma/generator/models.py", line 314, in run
    raise PartialTypeGeneratorError() from exc
prisma.generator.errors.PartialTypeGeneratorError

This error is happening (I think) because the partials file doesn't exist when the partial generator runs. The partial generator tries to import the models from the generated client folder, which causes the client's init.py to run, which contains this line:

    from . import (
        models as models,
        partials as partials,
        types as types,
        bases as bases,
    )

I really don't understand how this import works, but performing a similar import in a test package with a missing file produces the same partial initialization error.

I don't understand why the import works when generating the client to site-packages. I thought maybe due to module caching, but how would the import of the cli import worked if partials.py did not exist?

[alexjball]

I was able to work around the issues, demonstrated in this repo. The two changes I had to make to get this working:

1. The partials.py file needs to exist in order for the partial generator to import the partially-generated client package. This is because when the partial generator executes, the client and fields modules have been generated, but not the partials module (ref). from . import x produces an ImportError, not a ModuleNotFoundError as produced by a missing from .client import *, so the catch block doesn't handle this case.
2. The client package has its own copy of partial_models_ctx and generates partials to that, so the site-packages prisma generator needs to pull from the same one. From within the partial generator, we can patch the context on the models module so they are generated to the correct location.

Is there a particular reason why the package is copied into the external dir before generating the templates? It seems like it could work just to copy it after generating the templates, so that the partial generator can from prisma import models. This would have it use the correct models context and avoid loading the init file again, since it would have been loaded before running the generator.