deepkit / deepkit-framework

A new full-featured and high-performance TypeScript framework
https://deepkit.io/
MIT License
3.22k stars 123 forks source link

Simple use of serialize<T>(data) is not working - "No type information received. Circular import or no runtime type available." #562

Open juanda147 opened 7 months ago

juanda147 commented 7 months ago

Hi, I'm working on POCs to migrate from @marcj/marshal to @deepkit/type . Following the steps here I should get a serialized object. I'm not doing anything complex, and I removed the @f annotation from the class just in case. Using the serialize function like below, I'm always getting:

"No type information received. Circular import or no runtime type available."

image

marcj commented 7 months ago

did you enable reflection in tsconfig and installed @deepkit/type-compiler?

juanda147 commented 7 months ago

Yes I have installed @deepkit/type-compiler. Do I need an additional library to enable reflection in tsconfig?

marcj commented 7 months ago

you have to enable reflection in tsconfig, see https://deepkit.io/documentation/runtime-types/getting-started

juanda147 commented 7 months ago

I missed the "reflection": true. But after adding it I still get the error

marcus-sa commented 7 months ago

@juanda147 you also have to run deepkit-type-install. I suggest adding it to postinstall script in package.json.

juanda147 commented 7 months ago

I have both "@deepkit/type": "^1.0.1-alpha.143", and "@deepkit/type-compiler": "^1.0.1-alpha.143", installed.

The error I got is happening here:

image

marcj commented 7 months ago

and how do you build and execute it?

juanda147 commented 7 months ago

it is failing at running the tests

marcj commented 7 months ago

what test framework

marcj commented 7 months ago

is the test framework using tsc? Use DEBUG=deepkit env variable to get more information

juanda147 commented 7 months ago

jest

marcj commented 7 months ago

can't help like that must be an user error since we use jest as well and it works very well. I would make sure that ts-jest is used correctly, the correct tsconfig is loaded (see my DEBUG note), deepkit was correctly installed as seen in the docs. then it will work

juanda147 commented 7 months ago

I created an isolated project with some basic configuration and basically the same as the main project, and I got the same error, please find attached zip file. test_deepkit.zip

marcus-sa commented 7 months ago

@juanda147 your repro is broken

marcus-sa@pop-os:~/Downloads/test_deepkit/playground$ npx jest
 FAIL  DeepKit/Playground.test.ts
  ● given a serialized PhoneNumber › when the PhoneNumber class is deserialized › Then the class get methods functions as expected

    TypeError: Cannot read properties of undefined (reading 'getNationalNumber')

      49 |
      50 |   public get formatPhoneNumberAsE164(): string {
    > 51 |     return this.phoneUtil.format(this.parsedNumber, libphonenumber.PhoneNumberFormat.E164);
         |                           ^
      52 |   }
      53 |
      54 |   private static parsePhoneNumber = function (

      at i18n.phonenumbers.PhoneNumberUtil.format (node_modules/google-libphonenumber/dist/libphonenumber.js:935:74)
      at PhoneNumber.get formatPhoneNumberAsE164 [as formatPhoneNumberAsE164] (DeepKit/entities/PhoneNumber.ts:51:27)
      at Object.<anonymous> (DeepKit/Playground.test.ts:18:48)

Test Suites: 1 failed, 1 total
Tests:       1 failed, 1 total
Snapshots:   0 total
Time:        0.888 s

Did you do what I mentioned here https://github.com/deepkit/deepkit-framework/issues/562#issuecomment-2025776335 ?

juanda147 commented 7 months ago

This is really odd. I removed all external dependencies and left a very simple class, and still seeing the error. Just run it using: npx ts-node .\DeepKit\Run.ts

Thanks a lot for your help so far

playground_poco_class.zip

image

marcj commented 7 months ago

Can you please provide a proper Git repository. We are here on Github, there is no need to share source-code as Zip files.

juanda147 commented 7 months ago

https://github.com/juanda147/deepkit-test

marcj commented 7 months ago

Works fine

docker run -ti --rm node:20-alpine /bin/sh
/ # apk add git openssh
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/main/aarch64/APKINDEX.tar.gz
fetch https://dl-cdn.alpinelinux.org/alpine/v3.19/community/aarch64/APKINDEX.tar.gz
(1/20) Installing ca-certificates (20240226-r0)
(2/20) Installing brotli-libs (1.1.0-r1)
(3/20) Installing c-ares (1.27.0-r0)
(4/20) Installing libunistring (1.1-r2)
(5/20) Installing libidn2 (2.3.4-r4)
(6/20) Installing nghttp2-libs (1.58.0-r0)
(7/20) Installing libcurl (8.5.0-r0)
(8/20) Installing libexpat (2.6.2-r0)
(9/20) Installing pcre2 (10.42-r2)
(10/20) Installing git (2.43.0-r0)
(11/20) Installing openssh-keygen (9.6_p1-r0)
(12/20) Installing ncurses-terminfo-base (6.4_p20231125-r0)
(13/20) Installing libncursesw (6.4_p20231125-r0)
(14/20) Installing libedit (20230828.3.1-r3)
(15/20) Installing openssh-client-common (9.6_p1-r0)
(16/20) Installing openssh-client-default (9.6_p1-r0)
(17/20) Installing openssh-sftp-server (9.6_p1-r0)
(18/20) Installing openssh-server-common (9.6_p1-r0)
(19/20) Installing openssh-server (9.6_p1-r0)
(20/20) Installing openssh (9.6_p1-r0)
Executing busybox-1.36.1-r15.trigger
Executing ca-certificates-20240226-r0.trigger
OK: 30 MiB in 37 packages

/ # git clone https://github.com/juanda147/deepkit-test.git
Cloning into 'deepkit-test'...
remote: Enumerating objects: 14, done.
remote: Counting objects: 100% (14/14), done.
remote: Compressing objects: 100% (12/12), done.
remote: Total 14 (delta 1), reused 10 (delta 0), pack-reused 0
Receiving objects: 100% (14/14), 9.36 KiB | 2.34 MiB/s, done.
Resolving deltas: 100% (1/1), done.

/ # cd deepkit-test/playground

/deepkit-test/playground # npm i

added 52 packages, and audited 53 packages in 2s

9 packages are looking for funding
  run `npm fund` for details

found 0 vulnerabilities

/deepkit-test/playground # ./node_modules/.bin/deepkit-type-install
Deepkit Type: Injected TypeScript transformer at /deepkit-test/playground/node_modules/typescript/lib

/deepkit-test/playground # npx ts-node ./DeepKit/Run.ts
{ empCode: 1, empName: 'John Doe' }
1
juanda147 commented 7 months ago

I don't understand at this point what could be different on my side

image

marcus-sa commented 7 months ago

@juanda147 because you still haven't done what I've asked you to do twice already. https://github.com/deepkit/deepkit-framework/issues/562#issuecomment-2025776335

vany0114 commented 7 months ago

Hey @marcj @marcus-sa first things first, thanks for the help here! really appreciate it since we're trying to run some quick POCs before migrating from @marcj/marshal to @deepkit/type.

One of them is a wrapper that we have for an external library (google-libphonenumber). It has a class member called parsedNumber which type comes from the external library.

So for some reason when you serialize the wrapper it returns nothing which causes an issue when afterwards it's deserialized as the parse Number member is undefined: https://github.com/juanda147/deepkit-test/blob/9c2344944e097e12bd9775e0377b48fdf67f440c/playground/DeepKit/Run.ts#L17

By using @marcj/marshal we just annotate the member using the @f.any() and it works just fine, so not sure what's the right way to handle those kind of scenarios with this new package.

Do we need to create the "mapping" using the annotateClass? if so that would be kinda convoluted since we would need kinda duplicate the type definition that lives in the given package, also not sure whether for the google-libphonenumber PhoneNumber class it would work as it only expose methods.

Thanks in advance!

marcj commented 7 months ago

@vany0114 can you show me how you annotated the member of PhoneNumber in Marshal? I was looking at its definition at https://www.npmjs.com/package/@types/google-libphonenumber?activeTab=code but it seems PhoneNumber doesn't have any class property members, so I wonder how you did that.

You can use annotateClass if PhoneNumber is really a class, but this only works if there are actually data members, which I did not find as mentioned.

Alternatively you can register your own class serializer handler, see https://deepkit.io/documentation/runtime-types/custom-serializer

import * as libphonenumber from 'google-libphonenumber';
import { serializer } from '@deepkit/type'; //default serializer used in cast/is/serialize/etc functions

serializer.serializeRegistry.registerClass(libphonenumber.PhoneNumber, (type, state) => {
    // ${state.accessor} is the reference to the runtime PhoneNumber instance. do whatever to convert it to JSON serialisable structure like numbers/string/objects.
    state.setContext({ phoneUtil: libphonenumber.PhoneNumberUtil.getInstance() });
    state.addSetter(`phoneUtil.format(${state.accessor}, libphonenumber.PhoneNumberFormat.NATIONAL);`);
});

serializer.deserializeRegistry.registerClass(libphonenumber.PhoneNumber, (type, state) => {
    // ${state.accessor} is the value set by the serialize code above (e.g. numbers/string/objects, whatever you used)
    state.setContext({ phoneUtil: libphonenumber.PhoneNumberUtil.getInstance() });
    state.addSetter(`phoneUtil.parse(${state.accessor}, region)`);
});

this way you can just use

import { PhoneNumber }  from 'google-libphonenumber';

class User {
   phone?: PhoneNumber;
}

and everything just works when doing cast<User>({phone: 1234566}).

if you also want to make is<User>({phone: 'invalidnumber'}) to work, you might want to add type guards:

serializer.typeGuards.getRegistry(1).registerClass(libphonenumber.PhoneNumber, (type, state) => {
    // ${state.accessor} is the reference to the runtime PhoneNumber instance, do whatever to check if is valid by setting false/true
    state.setContext({ phoneUtil: libphonenumber.PhoneNumberUtil.getInstance() });
    state.addSetter(`phoneUtil.isPossibleNumber(${state.accessor}) === true && phoneUtil.isValidNumber(${state.accessor})`);
});

if you go with the way to make the default serializer aware of your PhoneNumber class, then you can literally use this type everywhere and get proper serialization/validation out of the box, including with cast, is, assert, serialize, deserialize, validate plus if you use the deepkit framework as whole, all http requests are validated automatically with the same mechanisms:


router.get('/user/phone-number/:number', (number: PhoneNumberUtil) {
    number instanceof PhoneNumberUtil; // always true
});

# curl http://localhost/user/phone-number/12345456
# curl http://localhost/user/phone-number/invalid-number -> fails
marcj commented 7 months ago

regarding the custom serializer, we might have to check if it works with packages without runtimes types like google-libphonenumber though. it might be you have to define you own type alias in your own code where runtime types are enabled and then use this type everywhere, e.g.

# my-types.ts
import { PhoneNumber }  from 'google-libphonenumber';

export class Phone extends PhoneNumber {}

serializer.serializeRegistry.registerClass(Phone, (type, state) => {
   //...
});
import { Phone } from './my-types.js';

export class User {
   phone?: Phone;
}
vany0114 commented 7 months ago

@marcj this is how we annotate it using @marcj/marshal

image

And thanks for the detailed explanation, we'll give it a try!

BTW has @deepkit/type a delegate similar to @marcj/marshal @OnLoad()? we also use in some classes to initialize stuff after deserialization and I was wondering what's the right way to do so with this package.

marcj commented 7 months ago

No, @OnLooad() doesn't exist anymore. Maybe in the constructor is good enough? Or you can add additional calls to the serializer templates


serializer.deserializeRegistry.registerClass(libphonenumber.PhoneNumber, (type, state) => {
   // as described set the value via state.addSetter(`xy`);
});
serializer.deserializeRegistry.appendClass(libphonenumber.PhoneNumber, (type, state) => {
   //state.accessor is the value xy now from previous step
   state.addCode(`${state.accessor}.additionalCalls()`);
});
vany0114 commented 7 months ago

@marcj this is how it looks like after your suggestion about using a custom serializer, in this case, I just added the setter w/o doing anything else but just passing the current state. And you were right, I had to define the alias as well for it to work.

serializer.serializeRegistry.registerClass(GooglePhone, (_, state) => {
    state.addSetter(`${state.accessor}`);
});

serializer.deserializeRegistry.registerClass(GooglePhone, (_, state) => {
    state.addSetter(state.accessor);
});

@marcj Thanks again, we'll probably keep bugging you as we continue with the migration to @deepkit/type

juanda147 commented 6 months ago

Hi, thanks for all your help so far. I was wondering if there is a replace for @f.type(MyClass) and @f.asName('')

marcus-sa commented 6 months ago

@juanda147

  1. You just reference classes directly as a type
  2. There's a MapName type annotation. See https://deepkit.io/documentation/runtime-types/types
juanda147 commented 6 months ago

Hi Again, we've been facing an issue trying to deserialize a generic type. I added a sample here https://github.com/juanda147/deepkit-test/blob/main/playground/Marshal/DeserializeGeneric.ts . Our question is if we can get rid off the type parameter type: new (...args: any ) => S and let deepkit to resolve the generic type using reflection?

marcj commented 6 months ago

@juanda147

    import { ClassType } from '@deepkit/core';
    import { deserialize, resolveReceiveType } from '@deepkit/type';

    const testDeserialized = <T>(payload: Record<string, unknown>, type: ClassType<T>): T => {
        return deserialize(payload, undefined, undefined, undefined, resolveReceiveType(type));
    };

    class TestSource {
        name: string = '';
    }
    const payload = { name: 23 };
    const res = testDeserialized(payload, TestSource);
    console.log(res);
marcj commented 6 months ago

@juanda147 alternatively

    import { deserialize, resolveReceiveType, ReceiveType } from '@deepkit/type';

    const testDeserialized = <T>(payload: Record<string, unknown>, type?: ReceiveType<T>): T => {
        return deserialize(payload, undefined, undefined, undefined, resolveReceiveType(type));
    };

    class TestSource {
        name: string = '';
    }
    const payload = { name: 23 };
    const res = testDeserialized<TestSource>(payload);
    console.log(res);
juanda147 commented 6 months ago

Hi, we still have some issues with the serializing/deserializing objects. We've been trying all of your suggestions but on every iteration some different fails. We may get the object with a few properties, the entire object with all properties as undefined, below are some screenshots describing our situations:

With the below implementation it looks like this, ignoring all properties from Order class and leaving only the props from the AggregateRoot: image

Above object should look something like this: image

using serializer.serializeRegistry. If I'm serializing, why am I getting the object typed as the class? Here some properties are undefined for some odd reason.

const serializedOrder = serialize<Order>(testOrder, undefined, undefined, undefined, resolveReceiveType(Order));

 serializer.serializeRegistry.registerClass(Order, (_, state) => {
      state.addSetter(`${state.accessor}`);
    });

image

using the @marcj/marshal library, returns the object as needed, plus, no typed associated to it: const serializedOrder = classToPlain(Order, testOrder); image

marcj commented 5 months ago

I assume this code is wrong:

serializer.serializeRegistry.registerClass(Order, (_, state) => {
  state.addSetter(`${state.accessor}`);
});

it does basically nothing. It tells the serializer to use a custom serializer template for Order, but the given template just forwards the value as-is, without doing any serialization. So this code explains why you get Order instance as serialization result. The code is wrong and needs to be removed.

It's always better to provide fully functioning code examples/reproduction. This way I can precisely see what is going on, what is the expectation, and where the error is. With screenshots and snippets alone it's a guessing game.