ioutil.ReadFile fails on the growing files

[krasin]

$ hg identify
0b98ba2443b0+ tip

What steps will reproduce the problem?
1. Start downloading a large file from the internet, like:

wget http://mirror.lstn.net/ubuntu-releases//oneiric/ubuntu-11.10-desktop-i386.iso

2. Call ioutil.ReadFile on this file. I have used the following test program:

package main

import (
    "io/ioutil"
    "os"
)

func main() {
    ioutil.ReadFile(os.Args[1])
}

What is the expected output?
Expected: nothing on the screen, exit code 0

What do you see instead?

$ ./read_file ~/gzip-bug/pgdvd042010.iso 
panic: runtime error: makeslice: cap out of range

goroutine 1 [running]:
bytes.(*Buffer).ReadFrom(0xf840021000, 0xf84000e210, 0xf840011018, 0x7ffffe00, 0x0, ...)
    /home/krasin/go/src/pkg/bytes/buffer.go:136 +0x2ab
io/ioutil.readAll(0xf84000e210, 0xf840011018, 0x200, 0xf84000e210, 0xf840011018, ...)
    /home/krasin/go/src/pkg/io/ioutil/ioutil.go:19 +0xc5
io/ioutil.ReadFile(0x7fffd87dc4c3, 0xf800000025, 0x0, 0x0, 0x0, ...)
    /home/krasin/go/src/pkg/io/ioutil/ioutil.go:47 +0x1ac
main.main()
    /home/krasin/read_file.go:9 +0x3e

Which compiler are you using (5g, 6g, 8g, gccgo)?

Which operating system are you using?

Which revision are you using?  (hg identify)

Please provide any additional information below.

[bytbox]

Comment 1:

Cannot reproduce (*), either with wget or /dev/urandom. (Linux jagadai 3.1.9-2-ARCH #1
SMP PREEMPT Sat Jan 14 09:11:37 CET 2012 x86_64 Intel(R) Core(TM)2 Duo CPU P8700 @
2.53GHz GenuineIntel GNU/Linux) What OS are you on?
This is a strange error to be getting, since it /should/ imply that an attempt was made
to create a slice larger than 2GB. I notice that the url you give in that command does
not match the filename given in the panic - is the file you're actually downloading
larger than 2GB? 
(*) I can reproduce if I run read_file on a file >2GB.

[krasin]

Comment 2:

Oh! Right, I was actually downloading Project Guternberg iso file which is ~15 GB. Thank
you for the correction!

[krasin]

Comment 3:

OS: Ubuntu Linux 11.10 x86-64

[krasin]

Comment 4:

And yes, you're right -- it's not related to the file being downloading, it's related to
the size of the file.
Please, close this issue as invalid.

[bytbox]

Comment 5:

That makes more sense - ReadFile is trying to create a 15GB slice, and can't (even if
your machine is 32 bytes).
Your solution is to manually call Read() multiple times, and process the file
bit-by-bit. (Note that if you run read_file while the file is still being downloaded, it
will /not/ wait to finish until the whole file is downloaded. Ordinary files can't be
used as FIFOs.)
Perhaps the size limitations on slices should be documented in the spec? (Or better,
removed? It does feel somewhat artificial.)

[krasin]

Comment 6:

After few additional minutes of thinking, I would prefer an error to be returned from
ReadFile instead of panic.
I agree with the rest of your notes.

[robpike]

Comment 7:

This issue was closed by revision 696bf79350b5cb0e977def1fc98ba6d6c8bd829.

Status changed to Fixed.