19 update xdr files to latest protocol

[gianfra-t]

Added modifications to allow type building with the latest Stellar .x files.

Upon changing the Rakefile to point to the latest types, two main issues were faced. First, the generated .ts file is missing the constant declarations. A post-process script will append them to the file.

Second, some generated structs and enum definitions made references to non primitive types. This created a cyclical reference error upon compilation. For this reason those types must be wrapped in Box pointer. For instance, the type:

pub struct ScSpecTypeOption {
    pub value_type: Box<ScSpecTypeDef>
}

In this fix these types are flagged by name prefix (ScSpecType..), which is not a future-proof solution. Ideally, the types that require this Boxed reference should be detected automatically and not by name so the code is resilient to future Stellar types changes.

Also, some changes were made in order to solve an error when calling npm run generate. For this, the local package js-xdr is now compiled first, and this is used instead of the typescript project.

[ebma]

When running npm run build I get the following error:

/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:587
    return new TSError(diagnosticText, diagnosticCodes);
           ^
TSError: ⨯ Unable to compile TypeScript:
x2JavaScript/generated/stellar-xdr_generated.ts:1021:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1021   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1038:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1038   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1054:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1054   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1069:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1069   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1124:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1124   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1141:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1141   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1158:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1158   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1175:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1175   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1192:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1192   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1209:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1209   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1226:22 - error TS2304: Cannot find name 'SC_SPEC_DOC_LIMIT'.

1226   ["doc", xdr.string(SC_SPEC_DOC_LIMIT)],
                          ~~~~~~~~~~~~~~~~~
x2JavaScript/generated/stellar-xdr_generated.ts:1646:36 - error TS2304: Cannot find name 'SCSYMBOL_LIMIT'.

1646 xdr.typedef("ScSymbol", xdr.string(SCSYMBOL_LIMIT));
                                        ~~~~~~~~~~~~~~

    at createTSError (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:587:12)
    at reportTSError (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:591:19)
    at getOutput (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:921:36)
    at Object.compile (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:1189:32)
    at Module.m._compile (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:1295:42)
    at Module._extensions..js (node:internal/modules/cjs/loader:1300:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:1298:12)
    at Module.load (node:internal/modules/cjs/loader:1103:32)
    at Function.Module._load (node:internal/modules/cjs/loader:942:12)
    at Function.executeUserEntryPoint [as runMain] (node:internal/modules/run_main:83:12)

I guess the code in post-process.js should actually be executed at the beginning of the generate script and not at the end to avoid this error @gianfra-t ?

[gianfra-t]

Yes that's correct! Sorry, I will change the order.

[ebma]

Thanks, that fixed the error about constants. But now I get the following error for npm run build:

> rust-stellar-xdrgen@0.6.0 generate
> node post-process.js && npm run generate-all-types && npm run format

> rust-stellar-xdrgen@0.6.0 generate-all-types
> DESTINATION=../src/xdr MAIN_FILE_NAME=types.rs npx ts-node x2JavaScript/generated/stellar-xdr_generated.ts

/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/js-xdr/src/index.ts:1
import { DefinitionFactory, XdrType } from "../types/types";
^^^^^^

SyntaxError: Cannot use import statement outside a module
    at internalCompileFunction (node:internal/vm:73:18)
    at wrapSafe (node:internal/modules/cjs/loader:1166:20)
    at Module._compile (node:internal/modules/cjs/loader:1210:27)
    at Module._extensions..js (node:internal/modules/cjs/loader:1300:10)
    at Object.require.extensions.<computed> [as .ts] (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/node_modules/ts-node/src/index.ts:1286:43)
    at Module.load (node:internal/modules/cjs/loader:1103:32)
    at Function.Module._load (node:internal/modules/cjs/loader:942:12)
    at Module.require (node:internal/modules/cjs/loader:1127:19)
    at require (node:internal/modules/helpers:112:18)
    at Object.<anonymous> (/Users/marcel/Documents/substrate-stellar-sdk/autogenerator/x2JavaScript/generated/stellar-xdr_generated.ts:21:1)

Do you have any other local changes that are not pushed? I am on node -v v19.6.0

[gianfra-t]

That one is strange since I was able to run it properly in my local. Let me try to clone it fresh and see if I can replicate that error.

[ebma]

Okay 🤔 are you running it on macOS or a Linux OS?

[gianfra-t]

I am on Linux, but I can confirm now I get that error. Strange. I will look into it.

[gianfra-t]

Okay so I changed the script to a .ts one, nevertheless what solve the error in my case is to install typescript with npm install typescript ts-node --save in the autogenerate folder. Which is strange since it should be handled by package.json. Please let me know if it works for you, and if you know of a change that wont require to manually run that command!

[ebma]

So you are saying that npm install typescript ts-node --save in the /autogenerator folder fixed the issue for you? I tried and I still get the same error :(

npm install typescript ts-node --save would save those two into the dependencies though instead of devDependencies. Maybe that changes something for you? But as I said, for me it's still the same. Which node and npm version are you using?

[gianfra-t]

Yes that solved it for me! So, npm version 9.8.1 and node v20.6.1, could that be the issue?

[gianfra-t]

Yes, I think is unlikely this will break in the short term. We can create a new issue and wait for @TorstenStueber input on this. I am updating the README and testing one last time.