This PR fixes a bug in databroker where it attempts and fails to deserialize partially streamed JSON objects due to chunking over http interfering with the chunking of JSON objects in the data layer.
Description
databroker.documents() interface fails to list all the documents in certain cases, especially when there are large serialized JSON documents involved. A test case test_large_document is written to test the large JSON deserialization bug, and it is patched by modifying the databroker client's document() interface to maintain a buffer (tail) until a complete JSON object is streamed.
Motivation and Context
This bug was encountered when testing the prefect workflows at NSLS-II CHX beamline. Databroker failed to list the documents associated with certail Bluesky runs, and raises a JSONDecodeError.
How Has This Been Tested?
A unit test is written in databroker/tests/test_broker.py which uses the demo server at https://tiled-demo.blueskyproject.io to list documents associated with a hard-coded uid knnown to reproduce the error without the patch.
This PR fixes a bug in databroker where it attempts and fails to deserialize partially streamed JSON objects due to chunking over http interfering with the chunking of JSON objects in the data layer.
Description
databroker.documents()interface fails to list all the documents in certain cases, especially when there are large serialized JSON documents involved. A test casetest_large_documentis written to test the large JSON deserialization bug, and it is patched by modifying the databroker client'sdocument()interface to maintain a buffer (tail) until a complete JSON object is streamed.Motivation and Context
This bug was encountered when testing the prefect workflows at NSLS-II CHX beamline. Databroker failed to list the documents associated with certail Bluesky runs, and raises a
JSONDecodeError.How Has This Been Tested?
A unit test is written in
databroker/tests/test_broker.pywhich uses the demo server at https://tiled-demo.blueskyproject.io to list documents associated with a hard-coded uid knnown to reproduce the error without the patch.