Provide REST API explorer

alanz commented 8 years ago

The fledgeling HIE has a JSON API. This includes some introspection as to what plugins are available, and what commands they provide, together with their parameters.

Provide an application that can interact with the API, and present the results in a meaningful way.

i.e.

$ stack build
$ stack exec hie -- --repl -d

Then access http://localhost:8001/ as in

curl -v http://localhost:8081/req/base -X POST -H Content-Type:application/json --data-binary '{"ideParams":{},"ideCommand":"version","ideContext":{"ctxEndPos":null,"ctxCabal":null,"ctxStartPos":null,"ctxFile":null}}'

Possibly consider using http://swagger.io, especially as there is now servant-swagger

Or any other appropriate technology. It needs to be able to run locally though.

cc @kRITZCREEK

gracjan commented 8 years ago

Pulling in a web server is a big dependency. I would advise against it.

alanz commented 8 years ago

HIE's dependency footprint is going to be massive anyway.

Also, this can be a frontend plugin eventually, to be configured in per user if wanted.

See https://github.com/haskell/haskell-ide-engine/issues/25

kjameslubin commented 8 years ago

Hi all, @alanz. We are interested in getting involved with this project as we have time. Commenting here because I'm in favor of a REST API. It can help with a local remote symmetry in development. As my laptop's compile times are fairly long, there's a scenario in which I could see preferring to develop on a more powerful virtualized machine. It also makes it easy to integrate in build / stage / deploy pipelines, which would likely be cloud based.

alanz commented 8 years ago

Sounds good, shall I add you to the project?

kjameslubin commented 8 years ago

Please. No promises on pushing code soon, but we want an IDE! It was nearly at the point where we were considering rolling our own - not that the existing alternatives are bad, but they weren't enough of a gain over just using emacs. It would be much better to contribute to a community project.

alanz commented 8 years ago

Ok, great.

You know that this is not an IDE? That said, one of the first integrations is into emacs, and there is a lot to be done on that front still.

On Sun, Dec 6, 2015 at 10:59 PM, Kieren James-Lubin < notifications@github.com> wrote:

Please. No promises on pushing code soon, but we want an IDE! It was nearly at the point where we were considering rolling our own - not that the existing alternatives are bad, but they weren't enough of a gain over just using emacs. It would be much better to contribute to a community project.

— Reply to this email directly or view it on GitHub https://github.com/haskell/haskell-ide-engine/issues/19#issuecomment-162346989 .

kjameslubin commented 8 years ago

Yes - but I am still trying to understand exactly what it is. But I think the approach of not building an impenetrable monolith is completely right. Once I put a project into something like Eclipse, it's stuck there forever, and I have to find an Eclipse plugin to use e.g. CLI tools that I'm already using.

I think the vision could be flushed out a bit though - I have skimmed the docs but I'm not 100% clear what the organizing principle would be, in a sentence.

My priorities would be:

Organize and make it easy to use existing Haskell ecosystem tooling
Integrate with other popular tooling as much as possible (Git(hub), Docker, Travis..)
Separate concerns so that it is easy to pick and choose which pieces you want in your personal "IDE environment"

alanz commented 8 years ago

That sounds like a good agenda.

I would summarise the project as a set of mutually reinforcing goals

Be a focal point for the haskell ecosystem wrt IDEs / tooling
Provide a separation of concerns, so that experts in their various fields can focus on their specialities.

This means the people who can do IDE integration can do that, the tool writers do that, the guys sorting out the nitty-gritty details of managing a project on the disk do that.

On Sun, Dec 6, 2015 at 11:15 PM, Kieren James-Lubin < notifications@github.com> wrote:

Yes - but I am still trying to understand exactly what it is. But I think the approach of not building an impenetrable monolith is completely right. Once I put a project into something like Eclipse, it's stuck there forever, and I have to find an Eclipse plugin to use e.g. CLI tools that I'm already using.

I think the vision could be flushed out a bit though - I have skimmed the docs but I'm not 100% clear what the organizing principle would be, in a sentence.

My priorities would be:

Organize and make it easy to use existing Haskell ecosystem tooling

Integrate with other popular tooling as much as possible (Git(hub), Docker, Travis..)

Separate concerns so that it is easy to pick and choose which pieces you want in your personal "IDE environment"

— Reply to this email directly or view it on GitHub https://github.com/haskell/haskell-ide-engine/issues/19#issuecomment-162348173 .

kjameslubin commented 8 years ago

Sounds great to me, thanks.

tobiasgwaaler commented 8 years ago

Maybe this is old news for you, @alanz, but If we go the swagger-route, the actual API-browser could be the Swagger UI

alanz commented 8 years ago

Yes, I agree that swagger is probably the best approach here, in terms of documentation too.

On Mon, Dec 7, 2015 at 12:00 AM, Tobias G. Waaler notifications@github.com wrote:

Maybe this is old news for you, @alanz https://github.com/alanz, but If we go the swagger-route, the actual API-browser could be the Swagger UI http://swagger.io/swagger-ui/

— Reply to this email directly or view it on GitHub https://github.com/haskell/haskell-ide-engine/issues/19#issuecomment-162352656 .

tobiasgwaaler commented 8 years ago

I've looked into servant-swagger lately, and have a crude POC ready, so I'll assign myself to this issue unless @kjameslubin is also working on this?

kjameslubin commented 8 years ago

Please go ahead @tobiasgwaaler. I do have one comment, though - it would be nice to have a RAML based spec, as RAML is pretty lightweight and can be useful to generate live docs. See an example here - this is generated from a yesod-raml routes file and it mostly just works though we need to fix some stuff. But if you are going to actually do the work, don't let me get in the way. Can swagger take RAML as input? I seem to remember servant being able to output RAML fairly easily.

dmjio commented 8 years ago

Just an FYI. Servant swagger is going to release a 1.0 version soon to generically infer schema. Cc @fizruk

Sent from my iPhone

On Dec 13, 2015, at 2:01 PM, Tobias G. Waaler notifications@github.com wrote:

I've looked into servant-swagger lately, and have a (crude POC ready)[https://github.com/haskell/haskell-ide-engine/tree/servant-swagger], so I'll assign myself to this issue unless @kjameslubin is also working on this?

— Reply to this email directly or view it on GitHub.

tobiasgwaaler commented 8 years ago

That's great news, @dmjio! Will the 1.0 release break the code I've already written (which isn't a lot)? I guess my question is if I should wait for the 1.0 release or start working with the current version?

fizruk commented 8 years ago

@tobiasgwaaler yes, 1.0 release is going to bring a lot of changes, since it's spec model will change. It also is going to bring a considerable amount of automation (with generic Schemas).

More specifically servant-swagger is going to use swagger2 for a complete Swagger model and generic Schemas. swagger2 is where the work is going on right now, that's why you don't see any progress in servant-swagger repo. I expect to finish it in a couple of days. Once that's done servant-swagger will be migrated/rewritten (we don't expect it to take a lot of time, the first usable version may be out in a week or sooner).

If you're are impatient and want to start working with something, you can probably help us out by reviewing swagger2 (here's a small Hackage API example and here are examples of generically derived schemas). A lot of late changes are not released yet, so you might want to check out HEAD. There is some documentation, but more explanatory/tutorial docs are yet to be written. Any feedback before we roll this out would be much appreciated!

If you have any questions, you can find us on #servant IRC channel.

tobiasgwaaler commented 8 years ago

@fizruk thank you for informing me :) I'm looking forward to swagger2. If I get impatient I'll try to use it to generically derive the schema for haskell-ide-engine. I'll let you know if I find the time

fizruk commented 8 years ago

@tobiasgwaaler we've just released servant-swagger-0.1! Happy New Year! :christmas_tree:

cocreature commented 8 years ago

@tobiasgwaaler servant-swagger is released and #152 is merged so there should be nothing stopping you now :)

tobiasgwaaler commented 8 years ago

Wonderful, I'll take a look as soon as I get the chance. If anyone else is eager to work on this, please go ahead. Although I'm motivated, I'm not sure when I'll get the time :/

cocreature commented 8 years ago

no pressure

ankhers commented 8 years ago

@tobiasgwaaler Out of curiosity, have you gotten anywhere with this ticket? If not, I was going to spend a bit of time this evening to see if I can get something working.

tobiasgwaaler commented 8 years ago

Hi, @ankhers. I started working on this branch a while ago, but haven't done anything since the release of servant-swagger-0.1 and after the commands were moved to the type level. So I would say no, I haven't gotten anywhere ;) And I probably won't in a good while, so please go ahead! :+1:

alanz commented 8 years ago

@ankhers @tobiasgwaaler @cocreature Is anyone making progress on this? I have some time available and want to get my brain around the tech for my day job, so I can tackle it if not

tobiasgwaaler commented 8 years ago

@alanz I'm not working on this :)

alanz commented 8 years ago

FYI, I have started here https://github.com/alanz/haskell-ide-engine/tree/swagger2

I think I am making progress, but need to chase some instances through

alanz commented 8 years ago

I have constructed a function hieSwagger which results in a Swagger structure when called with our servant based API definition.

The problem is, I do not know how to create an instance of ToSchema for ParamValP, which is a data type with existentials in it. I suspect putting a ToSchema constraint on the existential may help.

@cocreature any ideas?

fizruk commented 8 years ago

@alanz ToSchema constraint won't help as the existentially quantified type variable does not actually affect the schema AFAICT.

One straightforward way to introduce ToSchema for an existential type like ParamValP is to provide an "untyped" variant and use a Generic-based instance of that. It may be the case that you already have such an untyped representation (I am not familiar with the codebase):

-- ------------------------------------------
-- This is copied for convenience
-- ------------------------------------------
data ParamType = PtText | PtFile | PtPos

data ParamValP = forall t. ParamValP { unParamValP :: ParamVal t }

data ParamVal (t :: ParamType) where
  ParamText :: T.Text -> ParamVal 'PtText
  ParamFile :: T.Text -> ParamVal 'PtFile
  ParamPos :: (Int,Int) -> ParamVal 'PtPos
-- ------------------------------------------

data UParamValP = UParamValP { unUParamValP :: UParamVal }
  deriving (Generic)

data UParamVal
  = UParamText T.Text
  | UParamFile T.Text
  | UParamPos (Int,Int)
  deriving (Generic)

instance ToSchema UParamVal where
  declareNamedSchema = genericDeclareNamedSchema defaultSchemaOptions
    { datatypeNameModifier   = drop 1   -- we drop the U prefix
    , constructorTagModifier = drop 1 } -- we drop the U prefix

instance ToSchema UParamValP where
  declareNamedSchema = genericDeclareNamedSchema defaultSchemaOptions
    { datatypeNameModifier  = drop 1    -- we drop the U prefix
    , fieldLabelModifier    = \s -> take 2 s ++ drop 3 s } -- we drop the U in unUParamValP

instance ToSchema (ParamVal t) where
  declareNamedSchema _ = declareNamedSchema (Proxy :: Proxy UParamVal)

instance ToSchema ParamValP where
  declareNamedSchema _ = declareNamedSchema (Proxy :: Proxy UParamValP)

Another straightforward approach would be to manually write the ToSchema instances:

instance ToSchema ParamValP where
  declareNamedSchema _ = do
    sParamVal <- declareSchemaRef (Proxy :: Proxy (ParamVal t))
    return $ NamedSchema (Just "ParamValP") $ mempty
      & type_ .~ SwaggerObject
      & properties . at "unParamValP" ?~ sParamVal
      & required .~ ["unParamValP"]

instance ToSchema (ParamVal t) where
  declareNamedSchema _ = do
    sText <- declareSchemaRef (Proxy :: Proxy T.Text)
    sPos <- declareSchemaRef (Proxy :: Proxy (Int, Int))
    return $ NamedSchema (Just "ParamVal") $ mempty
      & type_ .~ SwaggerObject
      & maxProperties ?~ 1
      & minProperties ?~ 1
      & properties . at "ParamText" ?~ sText
      & properties . at "ParamFile" ?~ sText
      & properties . at "ParamPos"  ?~ sPos

BTW, it should be possible to derive this instances with TH, so if you have lots of GADTs or existentials, feel free to file a feature request!

alanz commented 8 years ago

@fizruk thanks for the great pointers, I will give it a go some time this weekend

cocreature commented 8 years ago

Writing the instances manually seems like the best way to go.

alanz commented 8 years ago

The servant-swagger stuff seems to be going into an infinite loop which consumes all memory.

I think the best approach may be a custom generator based on the UntaggedPluginDescriptors, as is used in the hie-doc-generator.

fizruk commented 8 years ago

@alanz which part goes into an infinite loop? Can you file an issue on https://github.com/haskell-servant/servant-swagger/issues?

alanz commented 8 years ago

If you run this test: https://github.com/alanz/haskell-ide-engine/blob/swagger2/test/JsonSpec.hs#L204 it will terminate (and fail the test), but if you uncomment https://github.com/alanz/haskell-ide-engine/blob/swagger2/test/JsonSpec.hs#L236 (instead of the RNil on the line above) it fails to terminate.

alanz commented 8 years ago

@fizruk FYI I have found using the swagger2 package directly that using

declareResponse (Proxy :: Proxy Object)

for Data.Aeson.Object causes an infinite loop.

See http://lpaste.net/7300947608627838976

fizruk commented 8 years ago

@alanz I have just fixed the infinite loop in swagger2-2.0.2. Can you please try that?

alanz commented 8 years ago

@fizruk I just tested it, and it works, thanks for the update.

alanz commented 8 years ago

The swagger2 package provides

declareResponse :: ToSchema a => proxy a -> Declare (Definitions Schema) Response

I would like to declare a response schema for a CommandFunc, and have the following type signature

declareCmdResponse :: ToSchema a => CommandFunc a -> Declare (Definitions Schema) Response
declareCmdResponse _ = declareCmdResponse XXXX

Could this work, and any ideas of what XXX could be? my initial approach of trying

declareCmdResponse _ = declareCmdResponse (Proxy :: Proxy a)

does not typecheck. I am pretty sure I am missing something simple here.

fizruk commented 8 years ago

@alanz

declareCmdResponse :: ToSchema a => CommandFunc a -> Declare (Definitions Schema) Response
declareCmdResponse = declareResponse

This should work because proxy in proxy a in the type of declareResonse is a type variable. So it should work for CommandFunc a just like for Proxy a.

Unless CommandFunc is a type family, of course.

alanz commented 8 years ago

yes indeed, thanks. Was pretty sure it would be something straightforward. I am still getting my head around all this type level programming.

alanz commented 8 years ago

This is turning out to be trickier than I thought. Some observations so far

The swagger ecosystem is not geared toward passing parameters in the body of a POST. This means that even if the generated swagger JSON file is correct, the online tooling does not generate commands properly, which is the whole point of the exercise.
The Pos type being compound makes it difficult to pass as a parameter, there will have to be some kind of wrapping/unwrapping to its component parts, which does not play nice with the swagger2 package.

So I am contemplating making the following changes to the HTTP API (with knock-on effects)

Split Pos into a Line and a Col variable. This is probably good anyway, I can envisage line-oriented commands that can then be supported.
Pass the HTTP API parameters as normal query parameters, so the swagger ecosystem tooling can support them easily.

Comments?

haskell / haskell-ide-engine

Provide REST API explorer #19