Closed andyarvanitis closed 5 years ago
Hi @andyarvanitis! There has been no discussion about this yet, but so far the trend is that if there are commonly used compiler flags we try to take care of the more general use-case at the spago level rather than just passing them through (because the passthrough is not the best UX-wise and it introduces problems, e.g. see #353 or #216)
In this case I would be interested in investigating if we can have first-class support for alternate backends in spago, so e.g. in order to get the whole alternative build running you'd only have to add something like backend = "go"
to your spago.dhall
I don't know what's the build workflow for the various backends, what are your thoughts on this?
That sounds fine to me, and would make it even easier for people to use the alternate backends. The Go backend would probably be a better candidate (than C++) to start with, since its tooling is simpler. I'm still making some adjustments, but I can let you know how it works and what the requirements would be.
@andyarvanitis thanks, looking forward to it! I'll rename this issue to track support for alternate backends then 🙂
Following up on my previous comment, I think I have the golang tooling tweaking done (for now, at least). In the end, the only thing you currently need to do a go-backend build (typical case) involves one more step than what I showed in this issue earlier, namely:
$ spago build -- -g corefn
$ psgo
If spago
(via the project config file) can provide those purs
flags and then execute psgo
, the user would get a native build/executable.
FYI, psgo
has a command switch of --run
which doesn't build an executable, but instead runs the results of the compilation immediately. Maybe something to think about supporting for backends in general. Same goes for --no-build
, which only generates intermediate language (e.g. Go) source files.
Let me know what you think.
Sounds great! A possible UX for all of this could be:
backend
key to the spago.dhall
, e.g. backend = "go"
spago build
would call spago build -- -g corefn && psgo
psgo
supports --run
then we could make it so that spago run
calls spago build -- -g corefn && psgo --run
--no-build
flag. We have such flag already in Spago, but that skips the build entirely. One thing I'd like to avoid is adding flags for certain backends only in Spago, so maybe we can skip the --no-build
flag for now?If the above makes sense then I have a few questions:
psgo
toolchain? And in general ensuring things like minimum psgo
, purs
versions?psgo
?psgo
take care of that? (I'm afraid the go package management story has changed a lot since I last used it several years ago, so I have no context on this)And cc @nwolverson: would this be interesting for the Erlang backend too?
- adding an optional backend key to the spago.dhall, e.g. backend = "go"
- if that's set, spago build would call spago build -- -g corefn && psgo
- since psgo supports --run then we could make it so that spago run calls spago build -- -g corefn && psgo --run
These all sound great to me.
- I don't have UI ideas for using the --no-build flag. We have such flag already in Spago, but that skips the build entirely. One thing I'd like to avoid is adding flags for certain backends only in Spago, so maybe we can skip the --no-build flag for now?
Definitely. My mention of it was mainly for information; skipping for now sounds good.
- should Spago worry about fetching the psgo toolchain? And in general ensuring things like minimum psgo, purs versions?
If spago is not responsible for installing purs
, I'd say no for psgo
too (which it seems is the case?). As for ensuring versions, maybe so. I'm happy to do what's needed to support it. For now, I think it would be fine not to check/ensure versions, especially if spago support for alternate backends can be marked "experimental" or something.
- should we worry about bundling native go dependencies here or will psgo take care of that? (I'm afraid the go package management story has changed a lot since I last used it several years ago, so I have no context on this)
Good question. psgo
should be able to take care of it using Go's module system (which it now uses, and currently pulls in all needed go dependencies). It might help if I add support for something like psgo --package-set psc-0.13.0-20190626
if I start tagging the go foreign implementations of those packages.
@csicar congrats for the release of the Kotlin backend! 🙂
Cc'ing you to ask if you'd have any inputs about this thread, i.e. would the current description of "integration" work for pskt
too?
First of all, I think adding support for alternative backends is a great idea.
adding an optional backend key to the spago.dhall, e.g. backend = "go"
I think just using the name of the executable there might even be easier: eg. for the go backend you would write psgo
.
Spago could expect the commandline interface to be standardized: calling the executable without arguments compiles; calling with --run
runs the program.
For spago that would mean, that no special support for any backend would need to be build.
The proposed changes would also work for pskt
should Spago worry about fetching the psgo toolchain? And in general ensuring things like minimum psgo, purs versions?
I agree with @andyarvanitis there: I think spago shouldn't have to care about that
Some other ideas:
prelude
). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-setbtw: pskt now has the same cli interface as psgo, so no additional arguments need to be provided to compile
spago build -- --codegen corefn && pskt
@csicar thanks for the inputs, it all sounds great!
I have only one question:
Support side-loading foreign files: PsKt and (if I remember correctly also PsGo) side-load foreign files for some standard purescript libraries (like prelude). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-set
How would this work? Also why not just push the various .go
and .cpp
and .kt
files to the core libraries so there wouldn't be the need for sideloading but we could just use the "official" versions?
Though I think the above is a detail, so I'd say at this point we have a pretty clear plan. Since I'm going to be short on time myself in the next weeks I'll detail here what I think it should be done, so anyone can eventually pick it up and implement it:
backend
key to the Config
, of type Maybe Text
, where Nothing
is the default backend and if it's Just cmd
then cmd
is the alternate backend executablespago build
(and all the commands that build underneath) call spago build -- -g corefn && cmd
spago run
but passing the --run
flag to the backendIt's been a bit of a hard sell to get non-js foreign code into the official core libraries. I can understand some of their reluctance, though, especially with more backends coming onto the scene. The approach I started taking with the second generation C++ backend and then continued with the Go backend works around this. But I wouldn't say it's a perfect solution, and some versioning provisions are still needed. See the comments on Aug 14 above for some (very basic) discussion on the matter.
As for conforming to an "alternate backend CLI specification," I'm all for it, and would be happy to modify my stuff to accommodate it. Spago has been really great to work with!
Having the foreign files in the original repos is of course preferable. In early development of the Backend and for alpha quality backends it is probably not desirable to upstream the foreign files to the official packages.
Though I think the above is a detail
Yes, absolutely. I don't want that to get in the way of progress.
I'm looking forward to having --watch
available by default :)
Another question about the cli: I would also be interested in having a no build
option: The kotlin compiler can be awfully slow, that means that recompilation is sometimes undesirable. Also when developing for Android, compilation is performed by the ide anyway, so that step can be skipped.
I guess that's something to work out when defining the cli specification.
Am 18. September 2019 19:59:25 MESZ schrieb Fabrizio Ferrai notifications@github.com:
@csicar thanks for the inputs, it all sounds great!
I have only one question:
Support side-loading foreign files: PsKt and (if I remember correctly also PsGo) side-load foreign files for some standard purescript libraries (like prelude). If there was some support for that, the user experience would be improved. Especially cloning the correct version of the side-loaded repository for the installed package-set
How would this work? Also why not just push the various
.go
and.cpp
and.kt
files to the core libraries so there wouldn't be the need for sideloading but we could just use the "official" versions?Though I think the above is a detail, so I'd say at this point we have a pretty clear plan. Since I'm going to be short on time myself in the next weeks I'll detail here what I think it should be done, so anyone can eventually pick it up and implement it:
- add a
backend
key to theConfig
, of typeMaybe Text
, whereNothing
is the default backend and if it'sJust cmd
thencmd
is the alternate backend executable- have
spago build
(and all the commands that build underneath) callspago build -- -g corefn && cmd
- same for
spago run
but passing the--run
flag to the backend- write down an "alternate backend CLI specification" in here, so that we have an interface that any backend can conform to if they want to work with Spago
-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/spacchetti/spago/issues/355#issuecomment-532797168
I haven't tried upstreaming foreign code yet, but thst seems to confirm my suspicion.
My idea with side loading foreign files would be to clone a foreign package set with the same tag as the primary package set that is being used.
Say for example the 1.13
tag is used for the package set, then spago would also clone a foreign package set at tag 1.13
to some location in the current project - say .spago/foreigns/psgo/
. The idea isn't refined, but maybe a starting point.
@andyarvanitis could that be a solution?
Am 19. September 2019 05:30:33 MESZ schrieb Andy Arvanitis notifications@github.com:
It's been a bit of a hard sell to get non-js foreign code into the official core libraries. I can understand some of their reluctance, though, especially with more backends coming onto the scene. The approach I started taking with the second generation C++ backend and then continued with the Go backend works around this. But I wouldn't say it's a perfect solution, and some versioning provisions are still needed. See the comments on Aug 14 above for some (very basic) discussion on the matter.
As for conforming to an "alternate backend CLI specification," I'm all for it, and would be happy to modify my stuff to accommodate it. Spago has been really great to work with!
-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/spacchetti/spago/issues/355#issuecomment-532950240
Yes, I was thinking of something along those lines, but for the Golang backend, I'd probably need it to go through psgo
itself, due to the way the go
tools work. So something like a switch on psgo
specifying the tag, and then it would take care of fetching the files. An example would be
psgo --ffi-tag 1.13
. Do you think maybe pskt
could also do the fetching in a similar way? It wouldn't be great, but it would at least make it easier on spago
too.
That solution would also work for pskt.
One think isn't clear to me yet: How would psgo know for which packages it should download the foreign files? If for example the packages "function" and "effect" are installed, would you still download the foreign files for other packages in the foreign files repo?
Am 20. September 2019 07:01:36 MESZ schrieb Andy Arvanitis notifications@github.com:
Yes, I was thinking of something along those lines, but for the go backend, I'd probably need it to go through
psgo
itself, due to the way thego
tools work. So something like a switch onpsgo
specifying the tag, and then it would take care of fetching the files. An example would bepsgo --ffi-tag 1.13
. Do you think maybepskt
could also do the fetching in a similar way? It wouldn't be great, but it would at least make it easier onspago
too.-- You are receiving this because you were mentioned. Reply to this email directly or view it on GitHub: https://github.com/spacchetti/spago/issues/355#issuecomment-533406420
For now, the entire purescript-native/go-ffi
repo is pulled in (which is what I was talking about tagging). If the user wants to pull in other ffi files, they would have to add them in the go module loader/config files for their project. If they want to omit some of them (in the standard libraries) from their build, they would need to comment them out in the same files.
Yeah, it's the same for me: pskt-foreigns
contains all foreign files and all are compiled. Maybe that's fine for now, but I think installing only the packages requested by the user is a better solution long-term (at least for pskt).
Btw: --run
is now also supported by pskt
I'm working on the implementation here: https://github.com/spacchetti/spago/pull/426
Since the initial request was fixed by #426, I'll close this one and I spawned #434 and #435 to track support for the other things we discussed here.
Thanks for all the great input 😊
Sounds great, thanks!
You're welcome 🙂
@f-f I think support for spago test
is still missing, right? I don't think either of the tickets you spawned encapsulate this.
@i-am-the-slime right now spago test
is ultimately following through the run
workflow, which means it calls the backend with the test entry point (ie Test.Main
). This works out just fine for purerl
but some backends may benefit from an explicit test workflow
Yeah I'm totally in the dark wrt the need of other backends to have a custom test workflow, so please detail if you have more context - I think we'd be fine addind a separate path for test
(rather than going through run
), but we need to be careful about not hardcoding specific behaviour for this or that backend.
Has there been any discussion of being able to specify the options passed to
purs
in the config file? My particular use case (alternate backends) currently requiresspago build -- -g corefn
, which admittedly isn't bad, but it would be nice to just set it once for a project.