search

peak / s5cmd

Parallel S3 and local filesystem execution tool.
MIT License
2.66k stars 237 forks source link

runtime: program exceeds 10000-thread limit #400

Open jslanier opened 2 years ago

jslanier commented 2 years ago

This appears to be a golang error that occurs when I am using Wasabi as the endpoint (instead of AWS S3). Apparently this can be changed by https://pkg.go.dev/runtime/debug#SetMaxThreads

I just do not know where to make this change in the code base. Can anyone point me in the right direction?

I have the endpoint-url set as an env variable

Here is the error: s5cmd cp --acl "public-read" --storage-class REDUCED_REDUNDANCY $localTilePath $newAWSPath runtime: program exceeds 10000-thread limit fatal error: thread exhaustion

jslanier commented 2 years ago

Fixed this by importing go debug package and then modifying main.go to raise MaxThreads

`package main

import ( "context" "os" "os/signal" "syscall" "runtime/debug" "github.com/peak/s5cmd/command" )

func main() { ctx, cancel := context.WithCancel(context.Background()) debug.SetMaxThreads(75000) go func() { ch := make(chan os.Signal, 1) signal.Notify(ch, os.Interrupt, syscall.SIGTERM) <-ch cancel() signal.Stop(ch) }()

if err := command.Main(ctx, os.Args); err != nil {
    os.Exit(1)
}

}`