default `paginate_batch_size=1000` on `MetadataDbClient.retrieve_docdb_records` returns 500 error on first page

[dyf]

results = self.doc_db_client.retrieve_docdb_records(
    filter_query={"name": {"$regex": "^SmartSPIM_.*stitched_.*"}},
)
len(results) # 483, with logging errors

results = self.doc_db_client.retrieve_docdb_records(
    filter_query={"name": {"$regex": "^SmartSPIM_.*stitched_.*"}},
    paginate_batch_size=500
)
len(results) # 1483, no errors

[helen-m-lin]

After investigation, we found that this is due to smart spim records being arbitrarily large and exceeding the 6MB payload limit in Lambda, even when compressed. We can add a check for the response size and improve the error handling (detailed error and suggest user to use a projection or decrease page size).

[dyf]

@helen-m-lin agree with this. Let's also change the default page size to 500.