Closed i10416 closed 2 years ago
Hi! I’d be happy to move to something more industry-standard than python scripts, but I have two major concerns:
Caused by: scala.MatchError: aarch64 (of class java.lang.String)
at BuildNative$.getArch(BuildNative.scala:11)
at BuildNative$.arch$lzycompute(BuildNative.scala:5)
at BuildNative$.arch(BuildNative.scala:5)
at BuildNative$.getLibName(BuildNative.scala:20)
[~/ws/jwm] brew install sbt
==> Auto-updated Homebrew!
Updated 1 tap (homebrew/core).
==> Updated Formulae
Updated 1 formula.
sbt: The x86_64 architecture is required for this software.
Error: sbt: An unsatisfied requirement failed this build.
A few questions I have for you:
This is python scripts performance numbers for comparison (I changed Log.cc and Example.java):
[~/ws/jwm] time ./script/run.py
Building libjwm_arm64.dylib
-- Configuring done
-- Generating done
-- Build files have been written to: /Users/tonsky/ws/jwm/macos/build
[2/2] Linking CXX shared library libjwm_arm64.dylib
Compiling 1 java files to examples/target/classes: ['examples/java/org/jetbrains/jwm/examples/Example.java']
./script/run.py 2.08s user 0.17s system 178% cpu 1.262 total
sbt from brew is not up to date. I found that the latest sbt supports m1 mac. You need to download universal package version from github
Excuse me for taking the time. I should check that in advance.
https://github.com/sbt/sbt/releases/
I am to blame for that scala.MatchError you give, not sbt compatibility. sbt itself has good portability.
project/BuildNative.scala
def getArch(): String = {
System.getProperty("os.arch").toLowerCase match {
// here should be "arm64" | "aarch64"
case "arm64" => "arm64"
case "amd64" | "x86_64" => "x64"
}
}
sbt takes about 1s > for stating up😖 but taking these facts
~
like ~compile
)into consideration, I think performance is decent enough.
If you keep running ~compile
, it does not take much time.
tl;dr see this build performance after editing Log.cc and Example.java https://github.com/HumbleUI/JWM/pull/155#issuecomment-925274099
run example application at examples/metrics
run
[info] compiling 14 Java sources toJWM-dev\examples\metrics\target
| => metrics / update 0s t\classes ...
| => metrics / Compile / compileIncremental 2s 0m
[info] running org.jetbrains.jwm.examples.Example
(These metrics below are measured in sbt shell on Windows. Linux is faster by around 2-5s and I think significant figures should be more precise.)
clean all(shared, win, macos, linux) including native build.
sbt:jwm> clean
[success] Total time: 1 s, completed Sep 23, 2021,
Compile for the first time
shared/c c
ompile
[info] compiling 39 Java sources to C:JWM-dev\shared\targ
get\classes ...
[success] Total time: 3 s, completed 2021/09/23
windows/compile
[info] compiling 2 Java sources to JWM-dev\windows\target\classes ...
[success] Total time: 1 s, completed 2021/09/23 0:54:40
linux/compile
[info] compiling 1 Java source to JWM-dev\linux\target\classes ...
[success] Total time: 1 s, completed 2021/09/23
macos/compile
[info] compiling 2 Java sources to JWM-dev\macos\target\classes ...
[success] Total time: 1 s, completed 2021/09/23
// clean all target/*
clean
[success] Total time: 0 s, completed 2021/09/23
compile all. note that windows/compile, macos/compile, linux/compile runs paralell.
compile
[info] compiling 39 Java sources to JWM-dev\shared\target\classes .
| => shared / update 0s ...
[success] Total time: 3 s, completed 2021/09/23
compile example application
sbt:metrics> compile
[success] Total time: 0 s, completed 2021/09/23
generate fat jar for example application
> assembly
[success] Total time: 12 s, completed 2021/09/23
package app
packageApp
[info] Wrote
[info] Windows Installer XML Toolset Compiler version 3.11.2.4516
[info] Copyright (c) .NET Foundation and contributors. All rights reserved.
[info] metrics.wxs
[info] Windows Installer XML Toolset Linker version 3.11.2.4516
[info] Copyright (c) .NET Foundation and contributors. All rights reserved.
use windows/packageBin from sbt-native-packager instead.
save package installer atJWM-dev\examples\metrics/target/windows
[success] Total time: 9 s, completed 2021/09/23
This is internally running the same script as Python.
Running cmake, ninja for the first time in the sbt shell
[info] -- The CXX compiler identification is MSVC 19.29.30133.0
[info] -- Detecting CXX compiler ABI info
[info] -- Detecting CXX compiler ABI info - done
[info] -- Check for working CXX compiler:
[info] -- Build files have been written to: JWM-dev/windows/build
| => root / cmake 1s [0m
Running ninja
[info] [1/21] Building CXX object JWM-dev\shared
| => root / ninja 7s d\src\main\cc\impl\Managed.cc.obj
[info] [2/21] Building CXX object JWM-dev\shared
| => root / ninja 7s d\src\main\cc\StringUTF16.cc.obj
[info] [3/21] Building CXX object JWM-dev\shared
| => root / ninja 7s d\src\main\cc\impl\RefCounted.cc.obj
[info] [4/21] Building CXX object JWM-dev\shared
| => root / ninja 7s d\src\main\cc\Window.cc.obj
[info] [16/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\WindowWin32.cc.obj
J
[info] [17/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Common.cc.obj
| => root / ninja 7s 0m
[info] [18/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Fence.cc.obj
[info] [19/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12Device.cc.obj
| => root / ninja 7s 0m
[info] [20/21] Building CXX object CMakeFiles\jwm.dir\src\main\cc\D3D12\DX12SwapChain.cc.obj
| => root / ninja 7s j
[info] [21/21] Linking CXX shared library jwm_x64.dll
Build JWM-dev\windows\build\jwm_x64.dll
copying artifact under resource directory
[success] Total time: 8 s, completed Sep 23, 2021
Running cmake, ninja from the second time onwards in the sbt shell
[info] [1/2] Building CXX object CMakeFiles\jwm.dir\src\main\cc\AppWin32.cc.obj
[info] [2/2] Linking CXX shared library jwm_x64.dll
Build workspace\JWM-dev\windows\build\jwm_x64.dll
copying artifact under resource directory
[success] Total time: 2 s, completed Sep 23, 2021
You said that “make it easier to run test and build scripts”. I would like to hear more, what is hard with current Python scripts? Can we maybe address those problems somehow?
General First, Python scripts are imperative. I prefer declarative style.
Although it is possible to do that via Python script, I think it is intuitive for developers to use build tools and package manager to manage projects.
Build tools have a standard way of doing stuffs and developers should obey the rules build tools impose on them to some extent, which reduces LoC and keeps code clean, but Python scripts does not.
For instance, build tools enable developers to easily specify dependency(local/remote), versions, etc... Many tasks such as managing version, generating fat.jar, releasing app to local repo or remote repo, packaging by jpackage installer, etc... become easier if you use build tools and its ecosystem (as long as you are on the rails).
In fact, this can be written in much more declarative way.
os.chdir(os.path.dirname(__file__) + "/../" + common.system)
rev = revision.revision()
artifact = "jwm-" + common.system + "-" + common.arch
with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.xml", "r+") as f:
pomxml = f.read()
f.seek(0)
f.write(pomxml.replace("0.0.0-SNAPSHOT", rev))
f.truncate()
with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.properties", "r+") as f:
pomprops = f.read()
f.seek(0)
f.write(pomprops.replace("0.0.0-SNAPSHOT", rev))
f.truncate()
with open(os.path.join('build', 'jwm.version'), 'w') as f:
f.write(rev)
...
# Restore poms
with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.xml", "w") as f:
f.write(pomxml)
with open("deploy/META-INF/maven/org.jetbrains.jwm/" + artifact + "/pom.properties", "w") as f:
f.write(pomprops)
ThisBuild / version := "0.0.0-SNAPSHOT"
// edit this and change is reflected to the all modules.
lazy val shared = project
.in(file("shared"))
.settings(
name := "jwm-shared",
)
lazy val shared = project
.in(file("windows"))
.settings(
name := "jwm-windows-x64",
)
lazy val shared = project
.in(file("linux"))
.settings(
name := "jwm-linux-x64",
)
lazy val shared = project
.in(file("macos"))
.settings(
name := "jwm-macos-x64",
)
This defines the versions for all projects.
sbt:jwm> version
[info] windows / version
[info] 0.0.0-SNAPSHOT
[info] linux / version
[info] 0.0.0-SNAPSHOT
[info] macos / version
[info] 0.0.0-SNAPSHOT
[info] version
[info] 0.0.0-SNAPSHOT
sbt:jwm> projectID
[info] shared / projectID
[info] org.jetbrains.jwm:jwm-shared:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver)
[info] windows / projectID
[info] org.jetbrains.jwm:jwm-windows-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver) 0J
[info] linux / projectID
[info] org.jetbrains.jwm:jwm-linux-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver)
[info] macos / projectID
[info] org.jetbrains.jwm:jwm-macos-x64:0.0.0-SNAPSHOT (e:info.versionScheme=early-semver)
Test With sbt (or any other build tools), developers can easily add and run tests by just adding code at test src directory.
Maintainability You can take advantage of type and IDE supports when writing scripts. I do not want to repeatedly run scripts to check its behavior. Besides, if someone is faced with build task error, in many cases, static typed languages can give more useful information than dynamic ones.
@tonsky
Note: there are some tweaks to be done especially for macos and linux related things.
Performance here (https://github.com/HumbleUI/JWM/pull/155#issuecomment-925088738) is measured on Windows. On linux, it is faster by 2.0-5.0s.
Performance comparison
I changed Log.cc and Example.java:
My environment
lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.3 LTS
Release: 20.04
Codename: focal
ninja --version
1.10.0
clang --version
clang version 10.0.0-4ubuntu1
Target: x86_64-pc-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
java --version
openjdk 16.0.1 2021-04-20
OpenJDK Runtime Environment (build 16.0.1+9-24)
OpenJDK 64-Bit Server VM (build 16.0.1+9-24, mixed mode, sharing)
Result
ninja build after edit Log.cc: 2s java sources, javadoc compile and publishLocal : 1s run example app after edit Example.java: < 1s
steps:
packArtifact;publishLocal
in jwm project sbt shellrun
in example app sbt shell[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/impl/package-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/org/jetbrains/jwm/impl/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/constant-values.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/overview-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/index.html...
[info] javadoc: Building index for all classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/allclasses-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/allpackages-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/index-all.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/overview-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/shared/target/api/help-doc.html...
[info] javadoc: 100 warnings
[info] Main Java API documentation successful.
[info] :: delivering :: org.jetbrains.jwm#jwm-shared;0.1.289 :: 0.1.289 :: release :: Thu Sep 23 04:41:50 JST 2021
[info] delivering ivy file to /home/i10416/workspace/JWM/shared/target/ivy-0.1.289.xml
[info] published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/poms/jwm-shared.pom
[info] published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/jars/jwm-shared.jar
[info] published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/srcs/jwm-shared-sources.jar
[info] published jwm-shared to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/docs/jwm-shared-javadoc.jar
[info] published ivy to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-shared/0.1.289/ivys/ivy.xml
[info] Wrote /home/i10416/workspace/JWM/linux/target/jwm-linux-x64-0.1.289.pom
[info] Main Java API documentation to /home/i10416/workspace/JWM/linux/target/api...
[info] compiling 1 Java source to /home/i10416/workspace/JWM/linux/target/classes ...
[info] javadoc: Loading source file WindowX11.java...
[info] javadoc: Constructing Javadoc information...
[info] javadoc: Building index for all the packages and classes...
[info] javadoc: Standard Doclet version 16.0.1+9-24
[info] javadoc: Building tree for all the packages and classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/WindowX11.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/package-summary.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/org/jetbrains/jwm/package-tree.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/overview-tree.html...
[info] javadoc: Building index for all classes...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/allclasses-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/allpackages-index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/index-all.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/index.html...
[info] javadoc: Generating /home/i10416/workspace/JWM/linux/target/api/help-doc.html...
[info] javadoc: 16 warnings
[info] Main Java API documentation successful.
[info] :: delivering :: org.jetbrains.jwm#jwm-linux-x64;0.1.289 :: 0.1.289 :: release :: Thu Sep 23 04:41:50 JST 2021
[info] delivering ivy file to /home/i10416/workspace/JWM/linux/target/ivy-0.1.289.xml
[info] published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/poms/jwm-linux-x64.pom
[info] published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/jars/jwm-linux-x64.jar
[info] published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/srcs/jwm-linux-x64-sources.jar
[info] published jwm-linux-x64 to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/docs/jwm-linux-x64-
[info] published ivy to /home/i10416/.ivy2/local/org.jetbrains.jwm/jwm-linux-x64/0.1.289/ivys/ivy.xml
[success] Total time: 3 s, completed Sep 23, 2021
I agree that using a build tool is an industry standard. You need a really good reason to NOT go with the off-the-shelf tooling.
I also appreciate you building the prototype and making a strong case for it. Under normal circumstances (e.g. standard Java project), I would wholeheartedly agree.
This project is, unfortunately, not a typical project.
First, it’s a 50/50 mix of Java and C++. Most of the build tools and IDEs are great at one language/ecosystem and offer escape hatches for everything else. E.g. in SBT you get Java compilation for free, but C++ is still a call to external tools and you have to write a script to run it.
Second, we also make use of Lombok, which adds an extra step to Java stages of the pipeline. Integrations exists for all major build systems, of course, and it’s great when they work, but it’s very hard to understand when they don’t work.
Third, project is very simple. There are no transitive dependencies, very few classes. Big build tools are great for huge projects, because you pay the overhead once and then reap the benefits. In our case, the overhead is bigger than the benefits. I don’t want to learn all the intricacies of SBT, Gradle or Maven just to make a simple javac or CMake call.
Fourth, our build is fairly non-standard. We are building jars out of native code, which is not a use-case Java build tools were build for. I have plans to merge all platforms in one jar eventually, which would mean merging multiple files built on different machines. It’s not a hard task, but it’s hard to teach e.g. Maven to do it. Build tools are good because they are rigid, they force you into their model. What we need in this project is flexibility.
Finally, I am very performance focused. I love the ability to change something and see the results immediately. Waiting event for a few extra seconds after every change is noticeable for me.
Now, having all those in mind, let’s go over build systems we’ve tried and see why we arrived at Python. Hopefully it’ll make more sense.
Gradle. Pros: flexible. Cons: java-focused, imperative, have to learn new language, slow.
Maven: Pros: declarative. Cons: java-focused, not flexible at all, hard to extend, slow.
Make: Pros: declarative, fast, language-agnostic (C-focused?). Cons: wasn’t able to make per-file recompilation of java files, relies on bash (see below).
Bash: Pros: flexible, language-agnostic, fast. Cons: imperative, not cross-platform, very hard to code.
Clojure (via Babashka): Pros: language-agnostic, cross-platform, fast, flexible, real programming language. Cons: have to learn new language, not enough batteries, imperative.
Python: Pros: flexible, language-agnostic, cross-platform, very popular language, batteries included (no dependencies needed), real programming language, fast. Cons: imperative.
SBT (I haven’t tried it, so it’s just my guesstimate): Pros: declarative, flexible? Cons: have to learn new language, slow, java-focused.
I am mostly happy with Python. I know it’s a non-obvious choice and I know IDE support is lacking, but I am happy with the pros it gives and the simplicity of the whole setup. I like the ability just do what I need directly (e.g. copy this file, call this program with those arguments) instead of guessing what hints I need to give to a build system to convince it to do what I want it to do.
E.g. with Maven, it was very hard to convince it to include some files but don’t include some other files, to NOT to run tests when I didn’t need it to, or generate javadocs on delomboked source files.
I’m sorry, but I have to give it a hard pass. I appreciate you doing the work, but I hope you can see my side of the things and where this decision comes from.
And again, if you have some specific problems with Python scripts, let me know and we could (hopefully) fix them. Including usability/convenience problems.
I like the ability just do what I need directly (e.g. copy this file, call this program with those arguments) instead of guessing what hints I need to give to a build system to convince it to do what I want it to do.
Thank you for elaborate your reasons. Most of them make sense for me. Thus, I will withdraw PR (or feel free to close by yourself). It is reasonable to run small native c-oriented tools from Python scripts in this project. I feel comfortable using Scala, but Python seems much suitable 😢
I’m sorry, but I have to give it a hard pass. I appreciate you doing the work, but I hope you can see my side of the things and where this decision comes from.
I don't mind 👍
if you have some specific problems with Python scripts,
This may not be about Python scripts and be nits pick, but I think it would be better to separate debug-purpose app from example apps like below. It feels neat if scripts for some remaining tasks such as packaging app or building native-image are managed under that dir.
P.S. If you felt sbt slow when you used it before, try sbt --client option introduced from v1.4.0. P.S. You can use Scala, real language, in sbt!
SBT (I haven’t tried it, so it’s just my guesstimate): Pros: declarative, flexible?
Thank you for taking time! jpackage task #104 has been almost done in this PR, thus I will rewrite it in Python and make another PR soon.
I Introduced sbt in order to make it easier to run test and build scripts.
// compile sbt compile
// clean shared/build, shared/target,/build,/target
sbt clean
// clean linux/build, linux/target sbt linux/clean
// publish to local ivy sbt publishLocal
// publish to local m2 sbt publishM2
// at examples/metrics
// run example application sbt run
// generate fat jar sbt assembly
// package example application. generate linux pacakge at target/linux ,msi at target/windows sbt packageApp // note: For some reasons, java 16's jpackage does not work on windows, so I use sbt-native-packager feature.