Generate Swagger API

[Zalastax]

Swagger is the best way I know to visualize your REST interface. It works by generating a JSON-document describing the API. The Swagger UI reads this document and generates a documentation page where you can read about the API functions, required schemas, and try out valid requests. You have a REST interface generator so this should be quite straightforward to implement. I don't think it could work for freestanding routing without changes since the information about required types is never available to the router. The REST generator seem to be given an overhaul https://github.com/rejectedsoftware/vibe.d/issues/1040 so should I wait with looking into this?

[s-ludwig]

Since it should only depend on the interface of the API, there is no reason to make this dependent on any other PR. I'd probably implement it in a separate module anyway, so that there is no direct overlap.

It surely looks like an interesting addition. I think someone recently proposed to support WADL (WSDL?) recently, too. What is a little sad is that, although I was sure it had been implemented long ago, there doesn't seem to be __traits(comment) support in the language to get the Ddoc comment, so that we'd have to resort to UDAs for the descriptions. That or creating an external tool instead of a library module...

[rikkimax]

I've investigated __traits(comment) in the past. It's doable but honestly? There is no way it would get accepted. It would slow down the compiler. Memory increased wayy too much.

[s-ludwig]

I seemed to remember that most people were quite exited to get it at some point. But well, memory use would be an argument (although, why would memory use ever increase more than the size of the input source code, which should not really be worth to mention?). For me, the main worry at the time was that people may start to abuse this as a poor men's replacement for UDAs - that risk isn't really there anymore today obviously.

[rikkimax]

The problem is, to be able to get it via traits you need to enable keeping and parsing DDOC for every symbol. While parsing probably won't be necessary, it still would be a major issue. So not only adding the trait is required, it's also necessary to enable storing the comments in case of needing it always. Right now adding the trait is simple. Making it usable without -D flag. Not so much.

Realistically what is being suggested is well and truely out of scope of vibe.d. It should be a separate tool. Mostly likely as a plugin for dub or dmd itself. This is definitely web service framework territory and vibe.d is not that.

[s-ludwig]

I don't quite get it yet, the memory needed to store doc comments is never greater than the size of the source code, which should rarely be more than a few MBs. But I guess that it simply is difficult to implement this efficiently in the existing DMD code base (e.g. DMD always parses them as it stands)? Anyway, don't let's waste too much time with this, I'm not too sad about the lack of it even if it could still be extremely useful in some cases (Manu had the idea to use the DDOC comments to display parameter documentation in a game editor for example).

Realistically what is being suggested is well and truely out of scope of vibe.d. It should be a separate tool.

I agree that it will probably need to be that way anyhow for technical reasons, and in case of Swagger I'd also agree with you that it pushes the boundaries for vibe.d. But on the other hand, a simple automatically generated standard, machine readable documentation for the REST interface generator is something that would IMO definitely be inside of its scope. All information is already there, so it would be foolish not to use it.

Also, the whole vibe.web.* package already is a kind of (low level) web service framework. I don't plan going higher level with this within vibe.d, but that doesn't mean that no more features will be added at the same abstraction level. The library is also scheduled to finally get split up into sub packages soon, so that leaves the scope of vibe.d as a pure marketing decision in that sense, not a technical one.

[Zalastax]

I am supportive of having Swagger support in another repository if that's the direction you are moving. In that case I think that the whole vibe.web.* should also be moved out.
I had a look at the source and I think it should be possible to copy RestInterfaceClient and change it to generate a JSON-schema instead. Copy-pasting code is a warning-sign, but I'm not sure it would be any better to have a generator interface.
How do you handle bigger REST-interfaces? It looks like you need one big interface or create multiple RestInterfaceClient! instances. Essentially what I want is https://github.com/glennjones/hapi-swagger but for D. Vibe.d already have a lot of the needed features. For me the documentation for the REST mapping needs to be better and I want to be able to hook into the registerRestInterface to read the interface before information is lost. Having to register each interface with swagger by hand would be manageable but not ideal. I will help make this better if you tell me your goals and point me in the right direction.

[s-ludwig]

You can nest interfaces by defining methods/properties that return interface instances:

interface MainInterface {
    // GET /greeting
    string getGreeting();

    // * /sub/*
    @property SubAPI subApi();
}

interface SubAPI {
    // GET /sub/foo
    string getFoo();
}

[rikkimax]

All of this is aimed at @Zalastax

You are bringing up problems that I solved in Cmsed. Registering of interfaces. In fact with only one small change to dmd + a couple small changes to druntime it would be possible to make this fully customizable so that no importing or registering of interfaces is required. But on the note of moving vibe.web.* out. I agree. In fact vibe.http.* should go right with it.

While I may not like the idea of it going into e.g. vibe.web.*, it is not up to me. It just screams web service framework IMO. I know it's quite strong wording wise. I spent a very long time solving this issue for D and in the process creating new design patterns.

Feel free to contact me if you want to talk more about what I have in store in this arena.

[s-ludwig]

@rikkimax Sorry, but I'm still reading ;) If you haven't read my thoughts about this somewhere yet: vibe.d is intended to be(come) a web framework. The reason for this is that the low level functionality (async I/O + fibers etc.) will probably end up in Phobos/Druntime sooner or later. The original plan was to stop of the protocol level and layer other things on top, but once the I/O/concurrency part ends up in the D standard library, vibe.d would basically just disappear in that case.

[rikkimax]

@s-ludwig Yeah I had an inkling that was the long term goal and I am definitely looking forward to it. Just short term wise, it would help to separate the two out a bit I suspect. We really should have a chat regarding the future of D with web service frameworks. There is many different approaches. Might be worth combining forces on some of it. Feel free to jump on https://gitter.im/DNetDev/Public

[s-ludwig]

At least as a first step I'll define separate sub packages within vibe.d's package description. The next step will then be to separate the source files into either separate folders or even separate repositories (haven't decided on that, yet, but probably the latter, at least to some degree). However, the first step is still blocked for a while until it's safe to say that most people will have upgraded to the latest DUB release (which includes an important bug fix for sub packages).

I think it would indeed be interesting to think about different designs some more. I'm currently experimenting with an extension to the current approach in vibe.web that would be very nice in certain aspects. However, I'm still not sure how elegant that can be implemented under the hood (my current proof of concept implementation requires the use of a TaskLocal, which is... rather inelegant). I'll tell more once [or "if" ;)] that is solved.

[etcimon]

The reason for this is that the low level functionality (async I/O + fibers etc.) will probably end up in Phobos/Druntime sooner or later.

Can't wait to see who's going to be expected of doing it =)

with an extension to the current approach in vibe.web that would be very nice in certain aspects. However, I'm still not sure how elegant that can be implemented under the hood (my current proof of concept implementation requires the use of a TaskLocal, which is... rather inelegant).

I know a lot of people would be wishing for a virtual machine (lua or js), I know vibe.d is expected to "kill" dynamic programming but that's probably the most convincing "Task Local" approach.

[s-ludwig]

Can't wait to see who's going to be expected of doing it =)

It will definitely be a mammoth task. Getting libasync through a std.async review alone is going to take a whole lot of work (are you willing to do that, btw?). Implementing the higher level primitives on top is another thing, but adjusting the whole rest of Phobos/Druntime to be compatible with the async system (by avoiding blocking operations) will be the thing that will really be "interesting".

[rikkimax]

Neither Walter or Andrei know or properly understand web development. Sooner a few of us get together and decide an end goal for web development in D and have somebody step up to drive it the better. Either way, not really appropriate to discuss on this issue. So please do join the gitter.im channel unless of course there is another chat mechanism you guys prefer? And yes, this is more of a long term goal thinking then anything else.

[s-ludwig]

You are probably right about Andrei and Walter there, but this is not about the web part, only about the underlying I/O and concurrency parts. I'd be happy to join a web dev chat, but I need some time to get my todo list a little shorter, before starting the next project.

[etcimon]

It will definitely be a mammoth task. Getting libasync through a std.async review alone is going to take a whole lot of work (are you willing to do that, btw?). Implementing the higher level primitives on top is another thing, but adjusting the whole rest of Phobos/Druntime to be compatible with the async system (by avoiding blocking operations) will be the thing that will really be "interesting".

I'm more tempted to move to a higher-level and simply maintain my lower-level repos. I feel like putting this in phobos is going to stay at the bottom of my todo list, because it doesn't "block" me from developing my upcoming projects. You know, I'm not in academics, I need to make the bank roll eventually through high-level stuff too.

[rikkimax]

@s-ludwig I've talked to them both. They've openly said that they don't know it. We don't need to put it into place any time soon. But having an end goal here will be very beneficial. Even potentially rallying more people behind the more important tasks. Right now, we still don't truly know what we want to do here.

I know I'm pushing to have this done, but my feeling is that we could save a couple years of meaningless development by setting some goals. That is measurable.

[etcimon]

I know I'm pushing to have this done, but my feeling is that we could save a couple years of meaningless development by setting some goals. That is measurable.

Focus on low-level stuff and you won't be losing your time that's for sure. I'd suggest to concentrate on RFCs or libraries. We need a DNS resolver library like this one right now for example.

[s-ludwig]

@etcimon: Yeah, that's true, it's why I didn't make the push earlier either. But personal considerations aside, there needs to be proper integration with the standard library for the sake of a better language eco system. This is something that you too (or whoever would put work into this) could gain from one way or another, in the long term.

I'd personally probably start with a top to bottom approach, though. So first make sure that the high level primitives that the standard library works with (streams, mutexes, conditions etc.) can be made compatible with an AIO approach. Then integrate an actual implementation of AIO primitives.

@rikkimax: I think this should generally be approached slowly and from low level to high level. I don't think it will make sense to put a web framework into Phobos for example, at least for a long time to come.

In general, IMO all of these "kitchen sink" issues with the standard library are really not all that important anymore with a standard package manager. Integrating event loop driven operation so far seems to be the only issue that should be worked out soon, so that the code fragmentation in the community gets stopped. Everything else can - but doesn't have to - be integrated when a certain library has successfully proven itself as a normal package over a sufficiently large time frame.

[s-ludwig]

@etcimon: If you want to get business done, better stop right there and get started with an existing (C) solution ;)

But seriously, it's really great to have all that in pure D, and please don't stop with it, but you have to focus on you end product if you ever want to get something done. Because if the implementation/porting of all low level stuff won't kill you, maintaining those masses of code will for sure.

[rikkimax]

@s-ludwig I'm thinking of a 10 year plan here. So things like language changes are not out of the question to make it nice to the end developer. After all, how can we prioritise what is important now, if we have no clue where we are going.

[Zalastax]

@rikkimax I agree. A quantifiable end goal is great to have since it gives focus and purpose. Personally I would want to replace node.js when writing micro services. For that I need to be able to generate my API from the code. I also quite like the flexibility from node middleware where I can hook thins on and things stay compatible. @s-ludwig My opinion is that you should start with what you want as an end product and then fix the low level code to make that possible. Most of the time it's possible to hide the nasty details by making a nice façade.

[etcimon]

But seriously, it's really great to have all that in pure D, and please don't stop with it, but you have to focus on you end product if you ever want to get something done. Because if the implementation/porting of all low level stuff won't kill you, maintaining those masses of code will for sure.

If I don't follow the original author's refactoring, it won't be such an issue. Crypto algorithms are baked into standards and they don't change, the HTTP/2 is baked into standards and they don't change, and now I can finally have a framework that I know from the bottom up. ie. I don't have time to wait months for other people to add a feature it if it's blocking me. I can add the features I want with the blink of an eye, it's a win/win

But yeah, only time will tell :)

[etcimon]

Here's a nice little feature I've managed to put together

https://github.com/etcimon/vibe.d/blob/00f5ffec96cad553d6efbbb17d71c77e6d06952d/source/vibe/http/http2.d#L71

It guarantees you that no copy of the decrypted bytes from a TLS Transfer will be visible in memory beyond the time it takes to read them, and the most robust option guarantees they will never touch the hard drive nor the swap memory (unless the server load is too high)

This is probably a unique feature in the web framework sphere, because I've never seen it anywhere else before.. =)

[gedaiu]

I don't know if it's relevant or not... I wrote a small library with validation:

https://github.com/gedaiu/swaggarize

[MartinNowak]

I've investigated __traits(comment) in the past. It's doable but honestly? There is no way it would get accepted. It would slow down the compiler. Memory increased way too much.

That trait has been request a couple of times, mostly for similar reasons. One concern is that it could be misused to attach semantics to comments, e.g. store SQL queries in there. The comments could possibly be added by an external tool that actually parses the interfaces (using dmd:parser).

Generating REST clients from OpenAPI/Swagger descriptions would also be interesting.

[s-ludwig]

One concern is that it could be misused to attach semantics to comments, e.g. store SQL queries in there.

Does that particular concern have any real weight, though, considering that we already have UDAs? An external tool would come with the usual overhead in terms of build system and flexibility. It's an approach that doesn't appear very D-like.