Problems with large files

[marcusbuffett]

I'm trying to proxy to a local web server, but it looks like ktunnel doesn't behave well with large files. Here's some output from curl:

❯ curl https://proxy.mbuffett.com/_next/static/chunks/main.js | wc -c
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  8 4295k    8  377k    0     0  73862      0  0:00:59  0:00:05  0:00:54 73862
curl: (18) transfer closed with 4012193 bytes remaining to read
386672

❯ curl localhost:8085/_next/static/chunks/main.js | wc -c            
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 4295k  100 4295k    0     0   209M      0 --:--:-- --:--:-- --:--:--  209M
4398865

Please, no judgement for the 4MB JS file :smile: I do see these logs from ktunnel about buffer sizes:

INFO[2020-07-30 07:59:34] increasing buffer size to 65536              
INFO[2020-07-30 07:59:34] increasing buffer size to 131072

Wonder if there's anything I can do about this limitation? Thanks!

And awesome program, exactly what I was looking for and easy to set up

[omrikiei]

Hi @marcusbuffett, thank you for the kind words! I'm 98% positive that the failure is related to buffer limitations in GRPC, so I limited the buffer size and should release it under v1.3.0 sometimes this week. however, if you'd like to test it out before that would be great! you can do so by cloning and building from branch v_1_3, and explicitly call the server version with the -i flag for example:

./ktunnel expose -i quay.io/omrikiei/ktunnel:v_1_3

[marcusbuffett]

No dice, unfortunately :/ I tried with a 400kb file:

❯ fallocate -l 400000 400k.txt
❯ curl http://proxy.mbuffett.com/static/400k.txt --output /dev/null
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
 32  390k   32  127k    0     0   4237      0  0:01:34  0:00:30  0:01:04     0

curl gets stuck there and stays at the same % received. Then eventually gives:

curl: (18) transfer closed with 269227 bytes remaining to read

[omrikiei]

hi @marcusbuffett, I'm finding it difficult to make time to troubleshoot and debug this issue, so it might take longer. I'm sorry :(

[marcusbuffett]

No worries at all, I’m happily using ktunnel for other purposes currently

[jaredallard]

I'm also running into this. Happy to help, but not 100% sure where to go 😢

[jaredallard]

I fixed this here: https://github.com/jaredallard/ktunnel/tree/jaredallard/feat/context

However, I put a lot of other changes in this branch, so I need to clean it up to make an actual PR.

[omrikiei]

@marcusbuffett, with @jaredallard 's contribution now merged and released in v1.3.0 - could you kindly check if the issue persists?

[omrikiei]

Just managed to test with 1.8G file - it works. so closing the issue..