search

RicoSuter / NSwag

The Swagger/OpenAPI toolchain for .NET, ASP.NET Core and TypeScript.
http://NSwag.org
MIT License
6.67k stars 1.23k forks source link

Epic: Multiple file output for code generators #1398

Open RicoSuter opened 6 years ago

RicoSuter commented 6 years ago

Implemented in NJS, update NSwag generators, CLI, UI, etc.

vgb1993 commented 4 years ago

Hy Rico, what's the status of this epic?

In my company we are generating an Angular client from our Core API definition. Because all the services are generated inside the same file we can not code-split each service in a separate bundle. This is very inconvenient as you can imagine.

We would like to have each service in it's own file.

I'm not familiar with the Liquid template notation, but I've read in this issue that you could write to multiple files. Do you think this could be achived this way?

Thanks for your time

ttma1046 commented 4 years ago

@vgb1993 My approach is

  1. put the auto-generated giant service and just itself in one single NgModule. (e.g. api-autogenerated-module).
  2. ask different service/facade in different ngModules uses/DI that gaint service in api-autogenerated-module.
daiplusplus commented 4 years ago

I might also suggest running a sed (or T4 script if you're on Windows and have msbuild available) that takes the monolithic output and splits it up into multiple output files. This can be done with a simple regex.

vgb1993 commented 4 years ago

@Jehoel

I might also suggest running a sed (or T4 script if you're on Windows and have msbuild available) that takes the monolithic output and splits it up into multiple output files. This can be done with a simple regex.

Hey thanks for the reply. I already read about this option somewhere in this repo, but I was a bit skeptical. Do you have any examples? Also I have two questions about this approach:

Anyway I realy feel this feature would be much better implemented inside NswagStudio, don't you think? We like using the GUI, it's very user friendly. Is there any plan to implement the feature? This has been open since last year.

vgb1993 commented 4 years ago

@ttma1046

  1. put the auto-generated giant service and just itself in one single NgModule. (e.g. api-autogenerated-module).
  2. ask different service/facade in different ngModules uses/DI that gaint service in api-autogenerated-module.

I don't like the idea of writing the Service twice (autogenerated / facade) only to allow code splitting, this defeats the purpose of the generator.

Also, are you sure this allows code splitting? I did some tests some days ago and I beliebe if the services are in the same .ts file they always get bundled together.

Cheers

RicoSuter commented 4 years ago

Internally the feature is almost complete. It's just not exposed via CLI, here you can see that internally we already have multiple "files" (artifacts) which are then merged and written to a single file in the CLI

https://github.com/RicoSuter/NSwag/blob/master/src/NSwag.CodeGeneration/ClientGeneratorBase.cs#L89

vgb1993 commented 4 years ago

Hy @RicoSuter,

Yes, I can see the code artifacts. Good to see it's in process. Could you explain us what is left to implement and an approximate deadline? Also I wonder what would the final result look like?

Thanks for the reply and all the great work, NSwag is amazing. Let me know if if you need help with this, I could give you a hand.

joan-grau commented 4 years ago

I'm currently traped in quite the same situation, I've been using NSwag for a while and this feature sounds like a very usefull and convinient one.

Is there any way to access this splited-files generator? Do you have plans to releaseing or exposing this feature via CLI?

Also, If there is some help needed, do not hesitate on contact me, It would be a pleasure to contribute to this amazing project.

francisminu commented 4 years ago

@RicoSuter Can you please let us know when this is planned to be released? Also, if I need to use it now, is it possible?

rjs-picturepark commented 4 years ago

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

hemiaoio commented 4 years ago

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference nswag-ts-splitter

emisand commented 3 years ago

@RicoSuter any update on this epic? We have an API that results in a 4 MB nswag typescript file and such a large file is not usable for our application. The solution is to have one typescript file for each controller.

Mike-Becatti commented 3 years ago

@RicoSuter any update on this epic? We have an API that results in a 4 MB nswag typescript file and such a large file is not usable for our application. The solution is to have one typescript file for each controller.

@emisand - We could no longer wait on this badly needed feature . We ended up adding a swagger doc per controller. Then we have an nswag file for each swagger doc. You may have to go this route too.

JDelladecimas commented 3 years ago

I too would like to request having separate files for each model, service and an index.ts file that exports all. @RicoSuter Can you point me to the file that generates this one big typescript file and I could take a look at contributing.

dariooo512 commented 3 years ago

Any update on this? Is there any way I can help speed up the release of this feature?

dylanvdmerwe commented 3 years ago

Having a single .ts file get bundled as one really does bloat things.

This is with 11 services: image

Ultimately we want to lazy load the services (via modules) and include them as necessary so they are lazy loaded and not all bundles in the main bundle.

nobba75 commented 3 years ago

Bump, any update on this @RicoSuter? Thanks for your effort!

Ldoppea commented 3 years ago

Hi,

I'm also interested by this feature.

But I'm a bit lost about what exists or not:

My need is to split the resulting TS file when using TypeScriptClientGenerator as 1 file is too big and it breaks the Storybook.js' build. I give more details here : https://stackoverflow.com/questions/67116837/is-it-possible-to-make-storybook-js-working-with-very-large-auto-generated-types

Is there some solution existing for this scenario?

If not, is there anything I can do to help? I don't know how this project is implemented nor its philosophy but I can try if you give me some directions.

Christian-Oleson commented 3 years ago

I'll also chime in as this being quite important. Essentially, I pretty much am going to avoid using NSwagStudio in favor of other Open API Code generators that have this feature, which requires more work on my end to customize/extend, but in the long term will allow me to on modify the explicit files that are being changed.

bsell93 commented 3 years ago

https://github.com/RicoSuter/NSwag/issues/1398#issuecomment-824668608

I've had this same problem. I ended up landing on a solution. Although not desirable it has proven to be effective.

Essentially what I ended up with is using Regex to parse the output and separate it into appropriate files.

Hopefully this helps someone else. I know it's helped my team tremendously in code reviews 😂

Note: This is a .netcore backend with typescript/axios generated client.

GenerateClientFilesAsync (click to expand) ```c# private async Task GenerateClientFilesAsync() { var document = await OpenApiDocument.FromUrlAsync("http://localhost:5000/swagger/v1/swagger.json"); var settings = new TypeScriptClientGeneratorSettings { ClassName = "{controller}Api", ClientBaseClass = "ClientBase", Template = TypeScriptTemplate.Axios, UseGetBaseUrlMethod = true, UseTransformOptionsMethod = true, UseTransformResultMethod = true, }; settings.TypeScriptGeneratorSettings.ExtensionCode = _clientBaseImport; settings.TypeScriptGeneratorSettings.DateTimeType = TypeScriptDateTimeType.String; var generator = new TypeScriptClientGenerator(document, settings); var code = generator.GenerateFile(); code = RemoveBrokenGeneratedCode(code); var (newCodeWithoutEnums, enumNames) = RemoveEnumsAndGenerateClientEnumFile(code); code = newCodeWithoutEnums; var (newCodeWithoutModels, modelAndInterfaceNames) = RemoveModelsAndGenerateClientModelFile(code, enumNames); code = newCodeWithoutModels; GenerateClientApiFiles(code, modelAndInterfaceNames, enumNames); } ```
RemoveBrokenGeneratedCode (you may or may not need these... I did not, so I removed them) (click to expand) ```c# private string RemoveBrokenGeneratedCode(string code) { code = Regex.Replace(code, @"^(((?!protected).)*)Promise<(.*)>", "$1Promise>", RegexOptions.Multiline); code = Regex.Replace(code, @"\s*private instance: AxiosInstance;", ""); code = Regex.Replace(code, @"\s*this\.instance = instance \? instance : axios\.create\(\);", ""); code = code.Replace("constructor(baseUrl?: string, instance?: AxiosInstance)", "constructor(baseUrl?: string)"); code = Regex.Replace(code, @"(else if \(status !== 200 && status !== 204\) {\s*)const _responseText.*;\s*return.*;", "$1return response.data;"); return code; } ```
RemoveEnumsAndGenerateClientEnumFile (click to expand) ```c# private (string code, IEnumerable enumNames) RemoveEnumsAndGenerateClientEnumFile(string code) { var matches = Regex.Matches(code, @"(^export enum\s(.*)\s\{[^\}]+\})", RegexOptions.Multiline); var enumNames = matches.Select(m => m.Groups[2].Value).OrderBy(x => x); var enumString = String.Join('\n', new string[] { "/* tslint:disable */", "/* eslint-disable */" }.Concat(matches.Select(m => m.Value))); foreach (Match match in matches) { code = code.Replace(match.Value, ""); } WriteOutGeneratedFile("enums.ts", enumString); return (code, enumNames); } ```
RemoveModelsAndGenerateClientModelFile (click to expand) ```c# private (string code, IEnumerable modelAndInterfaceNames) RemoveModelsAndGenerateClientModelFile(string code, IEnumerable enumNames) { var interfaceMatches = Regex.Matches(code, @"(^export interface (\S+) [\S\s]*?^})", RegexOptions.Multiline); var modelMatches = Regex.Matches(code, @"(^export class (\S+) [^\n]*implements[\S\s]*?^})", RegexOptions.Multiline); var enumNamesString = String.Join(", ", enumNames); var importEnumString = $"import {{ {enumNamesString} }} from './enums';"; var modelString = String.Join('\n', new string[] { "/* tslint:disable */", "/* eslint-disable */", importEnumString } .Concat(interfaceMatches.Select(m => m.Value)) .Concat(modelMatches.Select(m => m.Value))); foreach (Match match in interfaceMatches) { code = code.Replace(match.Value, ""); } foreach (Match match in modelMatches) { code = code.Replace(match.Value, ""); } var modelAndInterfaceNames = modelMatches .Select(m => m.Groups[2].Value) .Concat(interfaceMatches.Select(m => m.Groups[2].Value)) .OrderBy(x => x); WriteOutGeneratedFile("models.ts", modelString); return (code, modelAndInterfaceNames); } ```
GenerateClientApiFiles (click to expand) ```c# private void GenerateClientApiFiles(string code, IEnumerable modelsAndInterfacesNames, IEnumerable enumNames) { var isAxiosErrorString = code.Split('\n').TakeLast(4); var apiMatches = Regex.Matches(code, @"(^export class (\S+) [^\n]*extends ClientBase[\S\s]*?^})", RegexOptions.Multiline); foreach (Match match in apiMatches) { string apiCode = match.Value; var enumsToImport = enumNames.Where(x => Regex.IsMatch(apiCode, $@"\b{x}\b")); var modelsToImport = modelsAndInterfacesNames.Where(x => Regex.IsMatch(apiCode, $@"\b{x}\b")); apiCode = apiCode.Replace("export class", "export default class"); var numberCommentAndImportLines = 11; if (modelsToImport.Any()) ++numberCommentAndImportLines; var fileTopImportsAndComments = code.Split('\n').Take(numberCommentAndImportLines); var apiString = string.Join('\n', fileTopImportsAndComments.Concat(new string[] { apiCode }).Concat(isAxiosErrorString)); if (enumsToImport.Any()) { var enumNamesString = String.Join(", ", enumsToImport); var importEnumNamesString = $"import {{ {enumNamesString} }} from './enums';"; apiString = apiString.Replace(_clientBaseImport, $"{_clientBaseImport}\n{importEnumNamesString}"); } if (modelsToImport.Any()) { var modelAndInterfaceNamesString = String.Join(", ", modelsToImport); var importModelAndInterfaceNamesString = $"import {{ {modelAndInterfaceNamesString} }} from './models';"; apiString = apiString.Replace(_clientBaseImport, $"{_clientBaseImport}\n{importModelAndInterfaceNamesString}"); } WriteOutGeneratedFile($"{match.Groups[2].Value}.ts", apiString); } } ```

The output of this looks something like this in my folder/file structure; 

|generated
|-- enums.ts
|-- FilesApi.ts
|-- models.ts
|-- OrganizationsApi.ts
|-- UsersApi.ts

You get the gist

daiplusplus commented 3 years ago

I have already implemented multiple-file-output in my local clone last year...

The problem is I wrote it for myself - using my own opinionated coding-style and well... it's far too different than @RicoSuter's style for them to want to simply accept a PR - so I'd have to spend a decent amount of time reworking it into the house-style through gritted teeth...

UPDATE: I don't just mean stuff that can be automated via .editorconfig + Reformat, but more fundamental stuff... actually, come to think about it, it isn't that bad... I'll take a look at my code again and get back to this thread shortly...

@RicoSuter I assume you'll be okay if my contributions make changes to the NSwag Studio UI and add more code-gen options and include my customized NSwag Liquid Templates as in-box alternatives for other people to use? I really like how I've gotten them to work and I'd like to share them - so if you include my new features in the product as an ego-stroking exercise for myself I'll get my multi-file-output support ready for PR 😸

daiplusplus commented 3 years ago

@bsell93 I just noticed you have HTML5 <summary>-expanding regions in your post, I didn't realise GitHub's Markdown supported that - so I thought your post was just daydreaming, not providing an actual solution... you might want to make it more obvious that your post does contain code.

FaizulHussain commented 3 years ago

Badly needing this feature to be OOB, file is getting huge here :(

llgarrido commented 3 years ago

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference nswag-ts-splitter

@hemiaoio I liked your quick solution as long as this issue does not have a definitive solution.

There are aspects that you have not yet addressed.

I want to publish a fork of your great work with my suggestions.

llgarrido commented 3 years ago

I have already implemented multiple-file-output in my local clone last year...

The problem is I wrote it for myself - using my own opinionated coding-style and well... it's far too different than @RicoSuter's style for them to want to simply accept a PR - so I'd have to spend a decent amount of time reworking it into the house-style through gritted teeth...

UPDATE: I don't just mean stuff that can be automated via .editorconfig + Reformat, but more fundamental stuff... actually, come to think about it, it isn't that bad... I'll take a look at my code again and get back to this thread shortly...

@RicoSuter I assume you'll be okay if my contributions make changes to the NSwag Studio UI and add more code-gen options and include my customized NSwag Liquid Templates as in-box alternatives for other people to use? I really like how I've gotten them to work and I'd like to share them - so if you include my new features in the product as an ego-stroking exercise for myself I'll get my multi-file-output support ready for PR 😸

How did you manage to split the files? From Liquid templates or C#?

Is this implementation in your github fork? Where is it?

Thanks

daiplusplus commented 3 years ago

@llgarrido

How did you manage to split the files? From Liquid templates or C#?

In C#

Is this implementation in your github fork? Where is it?

I wrote "my local clone" not "my fork" - the code exists only on my PC. I haven't pushed it anywhere.

Mike-Becatti commented 3 years ago

@Jehoel - I did it in C# by creating a document per controller. 

   var manager = (ApplicationPartManager)services.LastOrDefault(d => d.ServiceType == typeof(ApplicationPartManager)).ImplementationInstance;
   var feature = new ControllerFeature();
   manager.PopulateFeature(feature);

   // Get controllers, but exclude BaseController class from generating a swagger document.
   List<string> controllerNames = feature.Controllers.Where(c => !c.Name.Contains("BaseController")).Select(t => t.Name).ToList();

   controllerNames.ForEach(c =>
   {
      string controllerName = c.Replace("Controller", string.Empty);
      services.AddSwaggerDocument(settings => { settings.ApiGroupNames = new string[] { controllerName }; settings.DocumentName = controllerName; });
   });
daiplusplus commented 3 years ago

@AlphaCreative-Mike That's horrible :S - you shouldn't compromise the design of a system to workaround minor technical or workflow issues.

Mike-Becatti commented 3 years ago

@AlphaCreative-Mike That's horrible :S - you shouldn't compromise the design of a system to workaround minor technical or workflow issues.

We own both sides of the system. Sometimes doing what is practical overrides what should be done in principle. To each their own. What's your alternative? This issue has been open for 3 years. Are you going to keep waiting for the fix?

daiplusplus commented 3 years ago

@AlphaCreative-Mike

What's your alternative?

Years ago, I cloned NSwag locally, modified its source to generate multiple output files - and numerous other changes to suit my own opinionated style and architecture.

Are you going to keep waiting for the fix?

I already have my own fix. And I'd love to share it, but there are three problems:

Mike-Becatti commented 3 years ago

@Jehoel - That's awesome. If your changes get incorporated into this repo I'll pick them up and toss my 'solution' in the trash. Until then, my solution suits my needs.

lucaritossa commented 1 year ago

Hi everyone! I'm working in a project based on ASP.NET WebAPI and Angular v10 library. The project is growing every day and a few days ago I crashed my head to a strange issue building angular library: "Maximum call stack size exceeded"

I finally found that the problem is generated by the size of "api.generated.ts" (this is the name of the output file of NSwag generation). Not a very big file size, it's about 1.6MB and it's still a mystery for me why it happens.

For now I found a work-around forking NSwag, changing/adding some lines of ClientGeneratorBase and TypeScriptFileTemplateModel (see https://github.com/lucaritossa/NSwag/commit/872ead10166dada047ba78c8bcbe6ca3afdc2f4f), publishing a custom version of NSwag.MSBuild in our private nuget repository and using it in our solution. Some changes in nswag.config and angular File.liquid template complete the work-around.

Splitting api.generated.ts into 2 files

is possible having 2 nswag config file into the same .NET project. I prepared

With the split, a problem must be solved: api-client.generated.ts does not compile because missing import of dto classes, now defined to the other file.

To solve the problem I needed to instruct typescript File.liquid template of api-client to write down the "import { dto classes } from './api-dto.generated"

Here it is the snippet

{%-        if Framework.IsAngular -%}

{%-            if Framework.UseRxJs5 -%}
...
...
{%-            endif -%}

{%-            if GenerateDtoTypes == false -%}
import { 
{% for name in TypeNames %}
  {{ name }},{% endfor -%}

} from './api-dto.generated';
{%-            endif -%}

{%-        endif -%}

GenerateDtoTypes and TypeNames are 2 properties I forcibly added to TypeScriptFileTemplateModel

Finally, to avoid a massive refactoring of the import currently defined in hundreds of angular services/components based on 'api.generated' I defined this file with the export of the other two

export * from './api-dto.generated';
export * from './api-client.generated';

THIS IS A WORKAROUND, IS NOT the solution of this epic! Defining 2 nswag config slows down the compilation (precious seconds) and it is required a little change to liquid template (this disturbing me because more attention will be required when updating NSwag packages)

BUT, I ask to @RicoSuter if I can propose a PR of the changes https://github.com/lucaritossa/NSwag/commit/872ead10166dada047ba78c8bcbe6ca3afdc2f4f to avoid the custom version of NSwag.MSBuild I have currently in my private nuget repo and to help others in case they want to replicate my workaround.

nkosi23 commented 1 year ago

What is the status of this feature? Is help needed for anything to get it ready?

Our use case is that we do not want "confidential" services that are only used by the staff portal to be present in the client shipped to the general public. While the backend has proper security mechanisms to prevent unauthorized access, we do not want people taking a look at the source code to be able to know our business processes (function names, class names, DTO & Cie leak a lot of information).

williamleeadc commented 1 month ago

It's already July 5th, 2024, and I still haven't seen nswag being able to modularize the generation of front-end proxy classes!

patrickklaeren commented 1 month ago

What is the status of this feature? Is help needed for anything to get it ready?

Our use case is that we do not want "confidential" services that are only used by the staff portal to be present in the client shipped to the general public. While the backend has proper security mechanisms to prevent unauthorized access, we do not want people taking a look at the source code to be able to know our business processes (function names, class names, DTO & Cie leak a lot of information).

You can generate different outputs for your client and confidential client.

nkosi23 commented 1 month ago

@patrickklaeren Thanks for the feedback, I'll take a look! Do you have any top of the mind pointers in the doc?

hemiaoio commented 1 month ago

@RicoSuter Could you please let us know when you are planning on rolling out a release with this feature, or if you know of a good alternative on how to handle the splitting of the files?

I have implemented the generation of multiple TS files on this basis, but there are two problems that are not very urgent to deal with at present. They can be solved temporarily through the parent class file for your reference nswag-ts-splitter

@hemiaoio I liked your quick solution as long as this issue does not have a definitive solution.

There are aspects that you have not yet addressed.

  • You haven't taken into account the imports required for specific framework templates such as Angular.
  • You didn't identify classes from TypeScript union types, e.g. (string | number | undefined).
  • Provide the possibility to create the files with names in kebab-style.

I want to publish a fork of your great work with my suggestions.

Absolutely, I'm eagerly anticipating your piece.