v hangs with sync.pool and parallel http.fetch commands

./v doctor sh: 1: cannot open /proc/1/mountinfo: No such file OS: linux, Debian GNU/Linux 10 (buster) (chroot) Processor: 6 cpus, 64bit, little endian, Intel(R) Xeon(R) E-2286G CPU @ 4.00GHz CC version: cc (Debian 8.3.0-6) 8.3.0

getwd: /home/rosensbc/v2 vmodules: /home/rosensbc/.vmodules vroot: /home/rosensbc/v2 vexe: /home/rosensbc/v2/v vexe mtime: 2021-05-12 12:30:08 is vroot writable: true is vmodules writable: true V full version: V 0.2.2 8361f71

Git version: git version 2.20.1 Git vroot status: weekly.2021.19-26-g8361f714 (49 commit(s) behind V master) .git/config present: true thirdparty/tcc status: thirdparty-linux-amd64 7543de81

What did you do? Parallel http requests with http.fetch().

mut fetcher_pool := pool.new_pool_processor(
  callback: multi_worker_process
)
fetcher_pool.set_shared_context(StockContext{
  api_connection: api_connection,
  products_s: products_s,
  saved_taxes: saved_taxes
})
for {
        page++
        response := api_connection.get('product/?page=$page&limit=500') // it always hangs here
        r := json.decode(ShopResponse, response) or {
            exit(1)
        }
        if r.meta.total < 1 {
            break
        }
        fetcher_pool.work_on_items(r.data) // here many parallel http.fetch commands are run
    }

What did you expect to see? Program not to hang.

What did you see instead? Program stops computing after some time when the api is unreachable for a tiny amount of time (don't know why no connection can be made - maybe because db/dns lookup happens on the CPU so blocking CPU computing globally makes it hang). If I do a strace it continues to compute.

strace -p 29609
strace: Process 29609 attached
futex(0x56530f507770, FUTEX_WAIT_BITSET_PRIVATE|FUTEX_CLOCK_REALTIME, 0, NULL, FUTEX_BITSET_MATCH_ANY) = -1 EAGAIN (Resource temporarily unavailable))
socket(AF_INET, SOCK_STREAM, IPPROTO_IP) = 3
setsockopt(3, SOL_SOCKET, SO_REUSEADDR, [1], 4) = 0
fcntl(3, F_GETFL)                       = 0x2 (flags O_RDWR)
fcntl(3, F_SETFL, O_RDWR|O_NONBLOCK)    = 0
stat("/etc/resolv.conf", {st_mode=S_IFREG|0644, st_size=38, ...}) = 0
openat(AT_FDCWD, "/etc/hosts", O_RDONLY|O_CLOEXEC) = 4
lseek(4, 0, SEEK_CUR)                   = 0
fstat(4, {st_mode=S_IFREG|0644, st_size=376, ...}) = 0
read(4, "\n127.0.0.1 localhost.localdomain"..., 4096) = 376
lseek(4, 0, SEEK_CUR)                   = 376
read(4, "", 4096)                       = 0
close(4)                                = 0
connect(3, {sa_family=AF_INET, sin_port=htons(80), sin_addr=inet_addr("192.168.132.81")}, 16) = -1 EINPROGRESS

I guess we need to add a timeout setting to http.FetchConfig.

vlang / v

v hangs with sync.pool and parallel http.fetch commands #10148