Closed ohmer1 closed 4 years ago
Currently looking into this. The root issue is that we are using bufio.Scanner
with the default MaxScanTokenSize
which is 64kiB. I can reproduce this in our test suite by adding a package with an unrealistically large extended description, which is technically not illegal.
This can be addressed several ways:
MaxScanTokenSize
and call Scanner.Buffer
with it upon the creation of Parser
in NewParser
Scanner.Buffer
to set it with the user-provided valuebufio.ErrTooLong
and dynamically increase buffer sizeOption 3 is not ideal as we run into the risk of the buffer growing infinitely large.
Option 2 gives the most control, but requires a lot of elbow grease since we'll need to change a bunch of function signatures to expose MaxScanTokenSize all the way to Cacher
(and we might also need to touch Mirror
since it also calls apt.ExtractFileInfo
. Plus, it introduces a configuration knob which might seem obscure to the user since it exposes an implementation detail which may not necessarily be relevant to them.
Option 1 doesn't give any control, but has the advantage that it does not break any of the current function signatures. It is also the most lightweight to implement.
For the time being, I'm leaning towards option 1 with an internal MaxScanTokenSize of 1 MiB, which seems to be a reasonable enough maximum size. If the buffer size issue is found to be prevalent enough, we can always consider building upon option 1 to implement the other options.
The fix works great, thanks!
I was using apt-cacher-ng but like you, I found it was unstable and I had intermittent issues. I'm trying to use go-apt-cacher as a satellite cache server. Our main repository is hosted with aptly.
I commented the existing mapping found in your sample config file and added mine:
The cacher is working fine. The first time a client ask for a package, it downloads it from the main repository and cache it locally. When another client ask for the same file again, the cacher get it from the local cache and doesn't get it from the central repository. All fine!
Problem is when I stop the cacher and try to start again. I get this error:
The only thing I found to "fix" that is to remove the "meta" directory content. But by doing that, this seem to invalidate my local cache.