Pr/mage build expt - Githubissues

owenwaller commented 1 month ago

First stab at creating a mage based build script, to replace the Taskfile based approach. The functionality offered in the magefile matches that of the Taskfile form PR #267

See:

https://github.com/vugu/vugu/pull/267

So far the mage approach have proved far superior.

All of the external tools - apart form docker have been eliminated. So this should now work on Linux, BSD, MacOS and Windows without change.
It is much more readable, maintainable and easily extendable approach approach
It's proved trivial to convert the existing Taskfile
We can lint the Magefiles, so we have some idea of quality and maintainability.

Can you please try executing this on your local machines?

If this works, then I propose this as a way forward over the Taskfile.

bradleypeabody commented 1 month ago

Thanks @owenwaller, I agree, this is awesome.

I agree that maintaining the build process in Go seems like a good idea and will lead to much simpler cross platform support. And mage is just as easy to install as task is, so why not.

When I ran mage allFast on my Mac with the latest Go version (1.22.2), I got the output show below. We can work together on debugging.

One thing I did want to mention - and we can discuss this separately as needed, is I still think that it best to keep the docker stuff as absolutely simple and minimal as possible. E.g. I'm not sure if we need to run the go linter via docker when we can instead do go install github.com/golangci/golangci-lint/cmd/golangci-lint@v1.57.2 (I realize their documentation says this is not recommended, but the Go module system is in my opinion far more reliable than most other software installation methods). The two things here that I think unavoidably need Docker are TinyGo (installation is too complex with lots of cgo stuff, etc.), and headless Chrome (same thing, too complex to install locally, especially cross platform). And the headless Chrome needs a server to run against - which I still think is much better to just put into one docker container along with the headless Chrome instance. I have run into number obscure docker networking issues, I do not consider it to be a reliable cross platform tool that just works in every case. We can keep testing the various scenarios, and if you have something that does work reliably cross platform then I'm not necessarily opposed to it. But if you're open to it, I'd like to see how to reduce our docker dependency to just TinyGo and Chrome+static file server.

Other than that concern, I think is approach is awesome and will work very well.

So far this is the issue I ran into locally:


% mage allFast
latest-alpine: Pulling from golangci/golangci-lint
Digest: sha256:90e1137558d1d226d8078cad24b7b34e26e86ff882245db06ea83780d91550d2
Status: Image is up to date for golangci/golangci-lint:latest-alpine
docker.io/golangci/golangci-lint:latest-alpine
level=info msg="[config_reader] Config search paths: [./ /app / /root]"
level=info msg="[lintersdb] Active 6 linters: [errcheck gosimple govet ineffassign staticcheck unused]"
level=info msg="[loader] Go packages loading at mode 575 (exports_file|imports|name|compiled_files|deps|files|types_sizes) took 7.708924891s"
level=info msg="[runner/filename_unadjuster] Pre-built 0 adjustments in 14.234579ms"
level=info msg="[linters_context/goanalysis] analyzers took 16.609179006s with top 10 stages: buildir: 8.041541744s, inspect: 621.849527ms, fact_deprecated: 600.070835ms, ctrlflow: 432.928044ms, printf: 407.406955ms, fact_purity: 348.659364ms, nilness: 267.314387ms, SA5012: 249.288638ms, typedness: 204.404315ms, isgenerated: 144.779768ms"
level=info msg="[runner] Issues before processing: 88, after processing: 0"
level=info msg="[runner] Processors filtering stat (out/in): cgo: 88/88, filename_unadjuster: 88/88, invalid_issue: 88/88, identifier_marker: 88/88, path_prettifier: 88/88, skip_files: 88/88, exclude: 88/88, exclude-rules: 19/88, skip_dirs: 88/88, autogenerated_exclude: 88/88, nolint: 0/19"
level=info msg="[runner] processing took 8.976787ms with stages: nolint: 4.982708ms, autogenerated_exclude: 2.744982ms, identifier_marker: 692.673µs, exclude-rules: 374.835µs, path_prettifier: 110.918µs, skip_dirs: 57.126µs, cgo: 4.543µs, invalid_issue: 3.75µs, filename_unadjuster: 1.751µs, max_same_issues: 792ns, fixer: 459ns, skip_files: 459ns, sort_results: 292ns, max_from_linter: 250ns, exclude: 250ns, uniq_by_line: 249ns, severity-rules: 208ns, diff: 166ns, source_code: 125ns, path_shortener: 84ns, path_prefixer: 84ns, max_per_file_from_linter: 83ns"
level=info msg="[runner] linters took 2.383283815s with stages: goanalysis_metalinter: 2.374244905s"
level=info msg="File cache stats: 0 entries of total size 0B"
level=info msg="Memory: 103 samples, avg is 191.9MB, max is 948.0MB"
level=info msg="Execution took 10.108345843s"
=== RUN   TestBuildEnvCachedComponent
--- PASS: TestBuildEnvCachedComponent (0.00s)
=== RUN   TestModCheckerChangeCounter
2024/04/28 15:02:26 changed(1) = true
2024/04/28 15:02:26 changed(2) = true
2024/04/28 15:02:26 changed(3) = false
2024/04/28 15:02:26 changed(4) = false
2024/04/28 15:02:26 changed(5) = true
2024/04/28 15:02:26 changed(6) = false
--- PASS: TestModCheckerChangeCounter (0.00s)
=== RUN   TestCompKey
2024/04/28 15:02:26 id=0x662ec7720cc6daf9
--- PASS: TestCompKey (0.00s)
=== RUN   TestModCheckerWidget
--- PASS: TestModCheckerWidget (0.00s)
=== RUN   TestModCheckerStruct
--- PASS: TestModCheckerStruct (0.00s)
=== RUN   TestModCheckerSliceArray
--- PASS: TestModCheckerSliceArray (0.00s)
=== RUN   TestModCheckerStrings
--- PASS: TestModCheckerStrings (0.00s)
=== RUN   TestModCheckerBool
--- PASS: TestModCheckerBool (0.00s)
=== RUN   TestModCheckerInt
--- PASS: TestModCheckerInt (0.00s)
=== RUN   TestModCheckerInt8
--- PASS: TestModCheckerInt8 (0.00s)
=== RUN   TestModCheckerInt16
--- PASS: TestModCheckerInt16 (0.00s)
=== RUN   TestModCheckerInt32
--- PASS: TestModCheckerInt32 (0.00s)
=== RUN   TestModCheckerInt64
--- PASS: TestModCheckerInt64 (0.00s)
=== RUN   TestModCheckerUint
--- PASS: TestModCheckerUint (0.00s)
=== RUN   TestModCheckerUint8
--- PASS: TestModCheckerUint8 (0.00s)
=== RUN   TestModCheckerUint16
--- PASS: TestModCheckerUint16 (0.00s)
=== RUN   TestModCheckerUint32
--- PASS: TestModCheckerUint32 (0.00s)
=== RUN   TestModCheckerUint64
--- PASS: TestModCheckerUint64 (0.00s)
PASS
ok      github.com/vugu/vugu    0.125s
?       github.com/vugu/vugu/cmd/vugufmt    [no test files]
?       github.com/vugu/vugu/cmd/vugugen    [no test files]
=== RUN   TestWasmCompiler
    compiler_test.go:19: Using temporary dir: /var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/TestWasmCompiler359386585
WasmCompiler: Successful build
WasmCompiler: build error: exit status 1; full output:
# TestWasmCompiler
./main.go:2:20: syntax error: unexpected valid at end of statement

    compiler_test.go:47: we correctly got a build error here: WasmCompiler: build error: exit status 1; full output:
        # TestWasmCompiler
        ./main.go:2:20: syntax error: unexpected valid at end of statement
WasmCompiler: Successful generate
WasmCompiler: Successful build
    compiler_test.go:74: Using temporary dir: /var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/TestWasmCompiler1843735203
WasmCompiler: Successful build
WasmCompiler: build error: exit status 1; full output:
# TestWasmCompiler
./main.go:2:19: syntax error: unexpected valid at end of statement

2024/04/28 15:02:28 MainWasmHandler: Execute error:
WasmCompiler: build error: exit status 1; full output:
# TestWasmCompiler
./main.go:2:19: syntax error: unexpected valid at end of statement
--- PASS: TestWasmCompiler (1.68s)
=== RUN   TestFileServer
    file-server_test.go:18: Using temporary dir: /var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/TestFileServer4054998540
--- PASS: TestFileServer (0.00s)
=== RUN   TestMux
--- PASS: TestMux (0.00s)
=== RUN   TestTinygoCompiler
TinygoCompiler: build error: exit status 1; cmd.args: [docker run --rm -v /:/src -e HOME=/tmp --user=501 -w /src/private/var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/TestTinygoCompiler716197059 tinygo/tinygo:latest tinygo build -o /src//var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/WasmCompiler3469945365 -target=wasm .], full output:
go: downloading github.com/vugu/html v0.0.0-20190914200101-c62dc20b8289
error: open /src//var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/WasmCompiler3469945365: no such file or directory

    tinygo-compiler_test.go:41: TinygoCompiler: build error: exit status 1; cmd.args: [docker run --rm -v /:/src -e HOME=/tmp --user=501 -w /src/private/var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/TestTinygoCompiler716197059 tinygo/tinygo:latest tinygo build -o /src//var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/WasmCompiler3469945365 -target=wasm .], full output:
        go: downloading github.com/vugu/html v0.0.0-20190914200101-c62dc20b8289
        error: open /src//var/folders/s5/lvbh5bhs3jjd3hrs1wy1j3vh0000gn/T/WasmCompiler3469945365: no such file or directory

--- FAIL: TestTinygoCompiler (11.74s)
FAIL
FAIL    github.com/vugu/vugu/devutil    13.536s
FAIL
Error: running "go test -v github.com/vugu/vugu/devutil" failed with exit code 1```

gedw99 commented 3 weeks ago

hey @owenwaller

https://github.com/flipt-io/flipt shows some neat tricks with Mage btw.

owenwaller commented 3 weeks ago

@gedw99

More so you are aware as @bradleypeabody comment is now quite a bit out of date.

We've had considerable discussion on Slack about using docker. The fair summary is it's likely to stay as it is so all the external tooling it called via docker. The approach has proved to be very robust including the docker networking.

The code base has also moved on considerably. See my PR branch (which is WIP and not yet merged)

https://github.com/owenwaller/vugu/tree/PR/mage_test001simple_refactor

The wasm-test-suite has been completely refactored and simplified, currently under new-test but that will change, and that build porcess is entirely driven by the magefile. I am at the minute simplifying the magfiles to both clarify and simplify them at the top level.

If you want to try it, pull the PR branch and do:

mage -v allfastnolint or mage -v allfast

The later runs the golangci-lint tool, the former doesn't.

I've not come across flipt so will investigate, but at a first glance at their magefile I don't see any tricks. Can you be more specific as to what you are referring to?

Also @bradleypeabody we need to be very careful here as flipt is GPL 3.0 licensed, and vugu is not, its MIT licensed. So I generally never take anything from GPL licensed code.

Owen

owenwaller commented 2 weeks ago

I am going to close this PR it has been superseded by the changes PR #272, that are in the v0.40 release.

vugu / vugu

Pr/mage build expt #269