google / starlark-go

Starlark in Go: the Starlark configuration language, implemented in Go
BSD 3-Clause "New" or "Revised" License
2.27k stars 204 forks source link

The int optimization poses problems for users running with ulimit #394

Closed aarzilli closed 1 year ago

aarzilli commented 2 years ago

Reference: https://github.com/go-delve/delve/issues/2907

The code in int_posix64.go assumes that the program has the ability to make a 4G allocation, this is not necessarily the case if the system is using ulimit to limit the amount of memory that processes can allocate. Given that this is an optimization the interpreter should be able to continue functioning even if the allocation fails (possibly attempting a smaller allocation first?)

adonovan commented 2 years ago

I see two possible fixes:

  1. Disable the optimization statically via a go build flag that is set by the delve build.
  2. Disable the optimization dynamically if mmap fails and a global flag is set that permits fallback to a slow implementation that always allocates a big.Int. This approach would require adjusting callers of Int.get to not assume that the "big" arm of the union is actually a large number. We'd need to measure the cost of the extra smallints > 0 check in Int.get and makeSmallInt.
TommyJerryMairo commented 2 years ago

Hi @adonovan, thanks for your reply. Are there any instructions on how could I disable the optimization statically via the go build flag?

adonovan commented 2 years ago

Something like this:

diff --git a/starlark/int_generic.go b/starlark/int_generic.go
--- a/starlark/int_generic.go
+++ b/starlark/int_generic.go
@@ -1,4 +1,4 @@
-//+build !linux,!darwin,!dragonfly,!freebsd,!netbsd,!solaris darwin,arm64 !amd64,!arm64,!mips64x,!ppc64x
+//+build !linux,!darwin,!dragonfly,!freebsd,!netbsd,!solaris darwin,arm64 !amd64,!arm64,!mips64x,!ppc64x StarlarkIntNoMmap

diff --git a/starlark/int_posix64.go b/starlark/int_posix64.go
--- a/starlark/int_posix64.go
+++ b/starlark/int_posix64.go
@@ -1,5 +1,6 @@
 //+build linux darwin dragonfly freebsd netbsd solaris
 //+build amd64 arm64,!darwin mips64x ppc64x
+//+build !StarlarkIntNoMmap
$ go build -tags=StarlarkIntNoMmap