Support for piping data

[tryy3]

Purpose Right now the only way for joy to compile is if there is a file for it and joy heavily relies that there is a golang file before compiling because it uses packages like golang.org\x\tools\go\loader that reads from go source from files.

With a feature like this, you will be able to integrate joy with tools such as webpack where you just pass on the data instead of writing changes directly to a file. Now I don't know webpack specifically very well but I am sure there is a lot of tools that simply pass the data on through plugins/functions before finally writing, so having the ability to pipe data to the joy command will simplify the process for integrating with some tools.

Solution So to allow piping data with commands such as: joy < main.go

You either need to move away from the golang loader and do it yourself or come up with some workaround. There is probably better ways to do this, but I got bored and made a proposal as I am in need of this feature. https://github.com/tryy3/joy/commit/ba0325ef80d6f76c17e29785f7ce28e986c53a96

The way I do it is, it will try to detect if you are piping data and if you are, it will create a tmp folder and a tmp file in the joy src folder. Then simply continue as if you were running the command: joy main.go

Once everything is done it will also try to remove this folder.

Issues I didn't want to do a pull request yet as there is a few issues:

1. If you do other pipes it will assume you are trying to pipe and no commands will work For example on windows I was building the joy command and then running the joy command with this command: go install github.com/matthewmueller/joy/cmd/joy | D:\Codes\Golang\bin\joy.exe < D:/tmp/sandbox708857067/main.go

It works as expected but the issue is when you try to run it like: go install github.com/matthewmueller/joy/cmd/joy | D:\Codes\Golang\bin\joy.exe run D:/tmp/sandbox708857067/main.go

Where it ignored the "run" command because the kingpin parse is ran last, so in that case it will assume you are piping but it's not the sort of piping we expect.

2. Because I simply create a tmp folder it can create some issues, if you were to run 2 instances at same time it will try to access the same file. Read/Write permission issues.

Fixes Some of the problems can be fixed rather easy.

1. This problem can be fixed if you were to make a seperate kingpin command for piping, such as: joy pipe < main.go

This will fix the issue of detecting if you are piping or not as it will simply assume you are if you run the command. But the syntax might look strange as you need to remember to use the pipe command.

You can also look at the size of the data being piped and try to detect piping that way, but then problem like this can appear: https://stackoverflow.com/questions/22563616/determine-if-stdin-has-data-with-go#comment34759200_22564526 So that method isn't that reliable.

2. I realise now that this isn't an issue, because we can use the ioutil.TempDir() method to handle multiple instances running at same time.

I do want to note that using the OS temp folder isn't reliable, because Golang requires relative paths, it can create issues when you are using multiple partitions, I haven't tried this on Linux/Mac but I assume they wont have this issue. But on Windows, partitions have labels such as C: D:... So if you have $GOPATH on one drive and the file you want to compile on a different, it wont be able to create a relative path over partitions.

I did try to investigate this but so far I haven't found any "real" fixes except creating a tmp folder on same partitions, so in this case it would probably be best to create a tmp folder inside joy src just to be sure there wont be any relative path issues.

I want to end this long message with, this is just a proposal, it would be nice to have a feature like this but there is a lot of different ways to implement this feature, so it be nice if we could have a discussion regarding this before an implementation is merged.

[matthewmueller]

Thanks for taking the time to look into this! It sounds like you stumbled across my original issues with piping data.

I'm now a bit hesitant to add support for this because it doesn't work in all cases and I learned that the Go binary doesn't support this either: https://github.com/golang/go/issues/15540. It seems like they recommend writing to a temporary file.

That's really interesting regarding the windows issue. Does Go get around this issue by enforcing a $GOPATH? If you run go build -v main.go you'll see that it's built in tmp. Do you know how that works on windows?

[tryy3]

Hmm yeah this was a proposal and I rather have a discussion of how we are gonna aproach it, I think the best way to go is to do it with a temp file, but we should probably look into it more as my aproach has a few problems as mentioned.

Oh... that's interesting, I forgot that go creates a temp file and on windows it's built to the tmp file, I will investigate on how they make the relevant path work this weekend and I'll respond with what I find. If we can make the tmp file work with relevant, then we can just do like Go does, where you can create a tmp file and simply add some support for piping that way. I understand why Go doesn't want to support "proper" piping and I believe adding real piping support would be a challenge with a lot of problems, but creating a tmp file if we can get the relevant path working properly would be really nice. As the piping support can open up a lot of nice possibilities.

But I'll look into how Go does it under the weekend, I remember that when I have messed up the relevant path, Go have tried to look at packages in both my partitions, so it might be trying to look in $GOPATH first and then use some other environment or something, but I'll investigate it and see what I can come up with.

[matthewmueller]

Awesome, yah I also like being able to support stdin, but I think Rob's comment maybe good enough: https://github.com/golang/go/issues/15540#issuecomment-232490230

I think something like this: joy run /dev/stdin < main.go

Update doesn't work right now, but it could:

error building code: error finding mains: exit status 1
Stderr: can't load package: package ../../../../../../../dev/stdin: cannot find package "../../../../../../../dev/stdin" in:
    /dev/stdin

[tryy3]

Hmm... interesting, we should probably fix that error you found. But personally it would be nice with some other solution too, I personally mainly use windows for development, so I can't use /dev/stdin, unless I either put up a virtual desktop or use the inbuilt linux instance that was introduced in windows 10 not long ago, but either way it be nice with a cross platform solution. Making the /dev/stdin work would fix my original issue but it would make my testing on windows slightly harder.

Update I suppose we could just make it so people need to add support for this themselves. The main reason I need it right now is for playground, but since GO itself doesn't support piping, the process behind the scene wouldn't change, it still need to create a temp file as I do right now. And for the future I would like to look into webpack possibilities or maybe implement my own solution for similiar tools. Where you pass on data through plugins and one of the plugins "pipe" the data to joy, in which case you could just make it so the plugin create a temp file.

Meaning we could look into why /dev/stdin isn't working right now and then add a note/example in the README or wherever you want, that explains how to do piping related stuff?