Requests a 'large" amount of virtual memory addresses, which may fail on some operating systems.

[ghost]

I am trying to use a piece of software that relies on starlark-go on a system that rigourously enforces ulimits (in particular on virtual memory). Which means that I have to patch out the the line 62 in int_posix64.go, which tries to allocate 4Gb of memory, which the machine does not possess.

var smallints = reserveAddresses(1 << 32)

func reserveAddresses(len int) uintptr {
    b, err := unix.Mmap(-1, 0, len, unix.PROT_READ, unix.MAP_PRIVATE|unix.MAP_ANON)
    if err != nil {
        log.Fatalf("mmap: %v", err)
    }
    return uintptr(unsafe.Pointer(&b[0]))
}

Please, consider implementing proper memory management, instead of "just allocating 4Gb of RAM".

[adonovan]

This function doesn't allocate 4GB of RAM, it allocates 4GB of virtual address space. They're just locations in memory that a Go program can point to, but there's no actual RAM there. Loading or storing through those pointers would cause a segmentation fault.

[ghost]

You cannot assume that the OS memory manager will give you so much virtual address space in general.

In particular, this code always returns "mmap: cannot allocate memory" on OpenBSD (which is mentioned at the top of the file) machines with less that 4Gb of ram.

[ghost]

You can simulate the same behaviour on Linux with ulimit -S -v 1000000

[adonovan]

You cannot assume that the OS memory manager will give you so much virtual address space in general.

The OS memory manager may deny the request, either because VM is exhausted (which is exceedingly unlikely: 2^64 is a large number) or because the request would exceed the current ulimit. I read than OpenBSD imposes a default ulimit for virtual memory of about 1.5GB, in contrast to Linux and Darwin (no limit). (Could you confirm what value you see from ulimit -v? Thanks.) In any case, the behavior really has nothing to do with the amount of actual RAM.

If OpenBSD's default ulimit is the problem, can you confirm whether ulimit -v 5000000 is an effective workaround? If so, then I am inclined to make the error message on OpenBSD more informative. Another possibility is to make OpenBSD use the generic int representation, at a significant performance cost.

[ghost]

Could you confirm what value you see from ulimit -v? Thanks

>ulimit -v
1052672

It's GNU bash, so the value is in kb.

can you confirm whether ulimit -v 5000000 is an effective workaround?

>ulimit -S -v 5000000
-bash: ulimit: virtual memory: cannot modify limit: Invalid argument
>ulimit -H -v 5000000
-bash: ulimit: virtual memory: cannot modify limit: Invalid argument

I think the limit for ordinary users is 1052672 and something about 1552672 for the staff login class, and it is not possible to go above that.

Another possibility is to make OpenBSD use the generic int representation, at a significant performance cost.

I am by no means a guru of OpenBSD, but speed has never been its selling point. Security and portability were. If this issue is allowed to stay for a day or two, I can email the community and ask for suggestions, but I am inclined to say that a generic representation would be a better choice.

[adonovan]

If this issue is allowed to stay for a day or two, I can email the community and ask for suggestions, but I am inclined to say that a generic representation would be a better choice.

Sure, no rush. To use the generic representation (which is used on all 32-bit machines and non-POSIX OSs), all you need to do is alter these two boolean formulas, one being the negation of the other:

https://github.com/google/starlark-go/blob/master/starlark/int_posix64.go#L1-L2 https://github.com/google/starlark-go/blob/master/starlark/int_generic.go#L1

[falsifian]

Why doesn't this code just use invalid pointers for small int32 values? E.g. odd addresses, assuming the alignment of big.Int is an even number. Then you don't need to reserve address space.

[adonovan]

Why doesn't this code just use invalid pointers for small int32 values? E.g. odd addresses, assuming the alignment of big.Int is an even number. Then you don't need to reserve address space.

That's a nice idea. Care to send a PR?

[falsifian]

I don't use skylark (or go) so am probably not the best person to do it. Maybe this will work:

• Add an assertion somewhere in int_posix64.go that Alignof(big.Int) is even.
• Change get() to something like

  // get returns the (small, big) arms of the union.
  func (i Int) get() (int64, *big.Int) {
  ptr := uintptr(i.impl)
  if ptr % 2 == 1 {
      return int64(ptr / 2) + math.MinInt32
  }
  return 0, (*big.Int)(i.impl)
  }

• Change makeSmallInt() to something like

  // Precondition: math.MinInt32 <= x && x <= math.MaxInt32
  func makeSmallInt(x int64) Int {
  return Int{intImpl(uintptr(x-math.MinInt32) * 2 + 1)}
  }

• Possibly rename int_posix64.go to something more generic, because this shouldn't only work on posix.