regular IncompleteRead on workers

[jelmer]

See e.g. https://janitor.debian.net/cupboard/pkg/pacemaker/eb48a0cd-94b4-46a6-bc40-a159d6ed8f44/ or https://jenkins.debian.net/job/janitor-worker/1033/console:

Started by timer
Running as SYSTEM
[EnvInject] - Loading node environment variables.
Building remotely on osuosl-build167-amd64.debian.net (osuosl167 amd64) in workspace /srv/jenkins/pseudo-hosts/osuosl-build167-amd64/workspace/janitor-worker
[janitor-worker] $ /bin/sh -xe /tmp/jenkins10539654540650077313.sh
+ /srv/jenkins/bin/jenkins_master_wrapper.sh
Opening branch at https://salsa.debian.org/ha-team/pacemaker.git/
Using cached branch https://janitor.debian.net/git/pacemaker/
Resuming from branch https://janitor.debian.net/git/pacemaker/
Elapsed time: 0:05:10.851644
Traceback (most recent call last):
  File "/usr/lib/python3.7/http/client.py", line 554, in _get_chunk_left
    chunk_left = self._read_next_chunk_size()
  File "/usr/lib/python3.7/http/client.py", line 521, in _read_next_chunk_size
    return int(line, 16)
ValueError: invalid literal for int() with base 16: b''

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/http/client.py", line 586, in _readinto_chunked
    chunk_left = self._get_chunk_left()
  File "/usr/lib/python3.7/http/client.py", line 556, in _get_chunk_left
    raise IncompleteRead(b'')
http.client.IncompleteRead: IncompleteRead(0 bytes read)

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/lib/python3.7/runpy.py", line 193, in _run_module_as_main
    "__main__", mod_spec)
  File "/usr/lib/python3.7/runpy.py", line 85, in _run_code
    exec(code, run_globals)
  File "/srv/janitor/debian-janitor/janitor/pull_worker.py", line 436, in <module>
    sys.exit(asyncio.run(main()))
  File "/usr/lib/python3.7/asyncio/runners.py", line 43, in run
    return loop.run_until_complete(main)
  File "/usr/lib/python3.7/asyncio/base_events.py", line 584, in run_until_complete
    return future.result()
  File "/srv/janitor/debian-janitor/janitor/pull_worker.py", line 396, in main
    possible_transports=possible_transports))
  File "/usr/lib/python3.7/concurrent/futures/thread.py", line 57, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/srv/janitor/debian-janitor/janitor/pull_worker.py", line 173, in run_worker
    possible_transports=possible_transports) as (ws, result):
  File "/usr/lib/python3.7/contextlib.py", line 112, in __enter__
    return next(self.gen)
  File "/srv/janitor/debian-janitor/janitor/worker.py", line 864, in process_package
    pick_additional_colocated_branches(main_branch))) as ws:
  File "/srv/janitor/debian-janitor/silver-platter/silver_platter/proposal.py", line 300, in __enter__
    dir=self._dir, path=self._path)
  File "/srv/janitor/debian-janitor/silver-platter/silver_platter/utils.py", line 86, in create_temp_sprout
    raise e
  File "/srv/janitor/debian-janitor/silver-platter/silver_platter/utils.py", line 70, in create_temp_sprout
    stacked=use_stacking)
  File "/srv/janitor/debian-janitor/breezy/breezy/git/dir.py", line 177, in sprout
    mapping=source_branch.mapping)
  File "/srv/janitor/debian-janitor/breezy/breezy/git/interrepo.py", line 780, in fetch_objects
    determine_wants, graphwalker, f.write)
  File "/srv/janitor/debian-janitor/breezy/breezy/git/remote.py", line 451, in fetch_pack
    progress)
  File "/srv/janitor/debian-janitor/dulwich/dulwich/client.py", line 1810, in fetch_pack
    progress)
  File "/srv/janitor/debian-janitor/dulwich/dulwich/client.py", line 747, in _handle_upload_pack_tail
    SIDE_BAND_CHANNEL_PROGRESS: progress}
  File "/srv/janitor/debian-janitor/dulwich/dulwich/client.py", line 530, in _read_side_band64k_data
    for pkt in proto.read_pkt_seq():
  File "/srv/janitor/debian-janitor/dulwich/dulwich/protocol.py", line 269, in read_pkt_seq
    pkt = self.read_pkt_line()
  File "/srv/janitor/debian-janitor/dulwich/dulwich/protocol.py", line 220, in read_pkt_line
    pkt_contents = read(size-4)
  File "/srv/janitor/debian-janitor/breezy/breezy/transport/http/__init__.py", line 1914, in read
    return self._actual.read(amt)
  File "/usr/lib/python3.7/http/client.py", line 457, in read
    n = self.readinto(b)
  File "/usr/lib/python3.7/http/client.py", line 491, in readinto
    return self._readinto_chunked(b)
  File "/usr/lib/python3.7/http/client.py", line 602, in _readinto_chunked
    raise IncompleteRead(bytes(b[0:total_bytes]))
http.client.IncompleteRead: IncompleteRead(4679 bytes read)

[jelmer]

This is due to git-store getting overloaded, which is due to dulwich sending full packs at the moment.

[jelmer]

This is mostly mitigated now, because we retry. But it would still be good to fix.