JSONDecodeError whenever communicating with Buildsense after acquire-auth

[stuhood]

(originally reported as https://github.com/pantsbuild/pants/issues/14222)

Describe the bug

Hi! I've acquired auth but Pants seem to be having a hard time talking with Buildsense.

Any time I run any Pants command it spews out a JSON error:

./pants filter --target-type=python_sources,python_tests :: | xargs ./pants lint
16:06:44.23 [INFO] Completed: Lint with isort - isort succeeded.
Skipped 47 files

16:06:44.87 [INFO] View on BuildSense: https://app.toolchain.com/organizations/grapl-security/repos/grapl/builds/pants_run_2022_01_20_16_06_43_209_0e550a27bfda4f199e846d8ccfd06ecc/
Exception in thread buildsense-reporter:
Traceback (most recent call last):
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/threading.py", line 926, in _bootstrap_inner
    self.run()
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/threading.py", line 870, in run
    self._target(*self._args, **self._kwargs)
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/reporter.py", line 338, in _report_loop
    operation = self._build_state.send_report()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/state.py", line 151, in send_report
    batch_sent = self._maybe_send_batched_build()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/state.py", line 248, in _maybe_send_batched_build
    batched_build = self._local_store.get_upload_batch()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 86, in get_upload_batch
    return self._create_batch()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 97, in _create_batch
    batch, files_in_batch = self._get_batched_data()
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 118, in _get_batched_data
    batch_data[key] = {build_file.stem: json.loads(build_file.read_bytes()) for build_file in files_batch}  # type: ignore[assignment]
  File "/home/wimax/.cache/pants/named_caches/pex_root/venvs/s/6ae700a8/venv/lib/python3.7/site-packages/toolchain/pants/buildsense/local_store.py", line 118, in <dictcomp>
    batch_data[key] = {build_file.stem: json.loads(build_file.read_bytes()) for build_file in files_batch}  # type: ignore[assignment]
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/__init__.py", line 348, in loads
    return _default_decoder.decode(s)
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/decoder.py", line 337, in decode
    obj, end = self.raw_decode(s, idx=_w(s, 0).end())
  File "/home/wimax/.pyenv/versions/3.7.10/lib/python3.7/json/decoder.py", line 355, in raw_decode
    raise JSONDecodeError("Expecting value", s, err.value) from None
json.decoder.JSONDecodeError: Expecting value: line 1 column 1 (char 0)

16:06:47.78 [INFO] Completed: Lint with Flake8 - Flake8 succeeded.
16:06:47.80 [INFO] Completed: Lint with Black - Black succeeded.
All done! ✨ 🍰 ✨
185 files would be left unchanged.

✓ Black succeeded.
✓ Flake8 succeeded.
✓ isort succeeded.

The token is fresh and has these permissions:

Pants version Which version of Pants are you using? 2.8.0

OS Are you encountering the bug on MacOS, Linux, or both? linux

Additional info

[asherf]

This is the batch logic that tried to upload build data that we failed to upload in previous buildsenee runs. I am looking into this issue

[stuhood]

I haven't seen this recently... worth resolving?

[asherf]

No. I think there is an actual issue here. I will take a look this week

[asherf]

At the very least , the plugin should be resilient to this kind of error