Closed theathorn closed 5 months ago
This is also the case for prod2. Start: 2022/02/18 14:08:37 End: 2022/02/18 14:14:03 /aws/lambda/azul-service-prod2
fields @timestamp, @message | filter @requestId like "b2cf13ec-c762-4f3d-abb4-dfd932af2ad9" | sort @timestamp desc | limit 2000
@timestamp | @message |
---|---|
2022-02-18 22:11:02.400 | END RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 |
2022-02-18 22:11:02.400 | REPORT RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Duration: 5584.74 ms Billed Duration: 5585 ms Memory Size: 2048 MB Max Memory Used: 229 MB |
2022-02-18 22:11:02.400 | RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Error: Runtime exited with error: exit status 1 Runtime.ExitError |
2022-02-18 22:11:01.540 | [DEBUG] 2022-02-18T22:11:01.540Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Returning 200 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key"}. See next line for the first 1024 characters of the body. {"hits": [{"protocols": [{"lastModifiedDate": "2019-05-16T16:16:30.693000Z", "submissionDate": "2019-05-16T16:16:24.783000Z", "updateDate": "2019-05-16T16:16:30.693000Z", "libraryConstructionApproach": ["10X v2 sequencing"], "nucleicAcidSource": ["single cell"]}, {"lastModifiedDate": "2019-05-16T16:16:31.087000Z", "submissionDate": "2019-05-16T16:16:24.801000Z", "updateDate": "2019-05-16T16:16:31.087000Z", "instrumentManufacturerModel": ["Illumina HiSeq 4000"], "pairedEnd": [false]}], "entryId": "74b6d569-3b11-42ef-b6b1-a0454522b4a0", "sources": [{"sourceId": "43f89fcc-8cd3-40d2-a2a3-aa43b2ce86c5", "sourceSpec": "tdr:datarepo-699bbe9b:snapshot/hca_prod_74b6d5693b1142efb6b1a0454522b4a0__20220117_dcp2_20220124_dcp12:/1"}], "projects": [{"lastModifiedDate": "2021-10-20T09:03:20.617000Z", "submissionDate": "2019-05-16T16:09:33.130000Z", "updateDate": "2021-10-20T09:03:20.617000Z", "projectId": "74b6d569-3b11-42ef-b6b1-a0454522b4a0", "projectTitle": "1.3 Million Brain Cells from E18 Mice", "projectShortname": "1M |
2022-02-18 22:10:57.126 | [INFO] 2022-02-18T22:10:57.125Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 POST https://search-azul-index-prod2-f7axew6smhdyuga32dcjapmlla.us-east-1.es.amazonaws.com:443/azul_v2_prod2_dcp13_projects_aggregate/_search [status:200 request:0.274s] |
2022-02-18 22:10:56.816 | [INFO] 2022-02-18T22:10:56.816Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Received GET request for '/index/projects', with query {"catalog": "dcp13", "size": "195"} and headers {"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "accept-encoding": "gzip, deflate, br", "accept-language": "en-US,en;q=0.5", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-country": "US", "host": "service.azul2.data.humancellatlas.org", "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "none", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0", "via": "2.0 d6db3d1c73f3e08aa70e81d7382400fe.cloudfront.net (CloudFront)", "x-amz-cf-id": "Dkwt4Om6FzufduXQZ5sHe3GQef-YsmGDC78jtvuLzNrSPXCkab4WBA==", "x-amzn-trace-id": "Root=1-62101970-342c5da056e88b3f001c6cc7", "x-forwarded-for": "76.89.218.122, 130.176.174.106", "x-forwarded-port": "443", "x-forwarded-proto": "https"}. |
2022-02-18 22:10:56.816 | [INFO] 2022-02-18T22:10:56.816Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Did not authenticate request. |
2022-02-18 22:10:56.812 | START RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Version: $LATEST |
@hannes-ucsc: "Spike to find out if other endpoints also break below a page size of 1000 (the current maximum). As part of the spike determine the size of each project in an unfiltered projects endpoint response for both catalogues, dcp12 and dcp13. With that information we will make the page size limit specific to each endpoint."
With the size set to 1000
dcp13
catalog has 502 errors on projects and bundles endpoint
dcp12
catalog has a 504 timeout error on bundles endpoint
2022-02-24 17:33:36,707 INFO MainThread: 502 https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=1000
2022-02-24 17:33:48,236 INFO MainThread: 200 https://service.azul.data.humancellatlas.org/index/files?catalog=dcp13&size=1000
2022-02-24 17:33:55,543 INFO MainThread: 200 https://service.azul.data.humancellatlas.org/index/samples?catalog=dcp13&size=1000
2022-02-24 17:34:13,048 INFO MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=1000
2022-02-24 17:34:21,543 INFO MainThread: 200 https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp12&size=1000
2022-02-24 17:34:30,607 INFO MainThread: 200 https://service.azul.data.humancellatlas.org/index/files?catalog=dcp12&size=1000
2022-02-24 17:34:35,488 INFO MainThread: 200 https://service.azul.data.humancellatlas.org/index/samples?catalog=dcp12&size=1000
2022-02-24 17:35:04,919 INFO MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=1000
For the dcp13
catalog the largest size for the unfiltered projects endpoint is 190, using 191 causes errors
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=191'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:15:52 GMT
Via: 1.1 c7dc1e4970209c6ac0b27019856131a4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: IrVzqYGVKN68vZPXVmYvnm5kf4wMITFEvoz6YZMmHwONxHQVJ8ZMaQ==
X-Amz-Cf-Pop: PHX50-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OEwWnE6VoAMFavg=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 2b46b48e-2479-4b4e-a803-d9ac3beeaa63
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=190'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 741584
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:16:11 GMT
Via: 1.1 a68097bdd514e6082da61e488096dbfa.cloudfront.net (CloudFront)
X-Amz-Cf-Id: MDYk3_oxGAiEch3JDAOpO_evtJjqkxbcQW_jwnkh-wWzHx4VQJs04Q==
X-Amz-Cf-Pop: PHX50-C1
X-Amzn-Trace-Id: Root=1-62182dd5-73ac01006d1c32c72ea2c7c9;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OEwZUGHVoAMFrtw=
x-amzn-RequestId: 6990e034-0f78-42ed-8db7-f1dbec00dc21
(.venv) CBSEspecter:azul amar$
Catalog dcp13
bundles endpoint can take a max size of 461, using 462 causes errors.
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=462'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:42:02 GMT
Via: 1.1 4d573cd6a6c0581b92beae04909c3dce.cloudfront.net (CloudFront)
X-Amz-Cf-Id: h0nKsaK-yDTU79zPO-HiMW6fNbaaTH08un6kRw3cUgYt0CnoSloXcQ==
X-Amz-Cf-Pop: SFO5-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OE0K8H1boAMFgHQ=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 2ec1659f-6387-4b40-82a1-796ec7443dcb
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=461'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 716000
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:42:15 GMT
Via: 1.1 28663e5849ed20a9d037ca8066957990.cloudfront.net (CloudFront)
X-Amz-Cf-Id: VrpEVOYvXx4XQ6pygF4oP9f3_YLyFjNmNRcM6_OSI892DjP6aWRJ-g==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-621833ee-2a51b8a074c1e3752121772d;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OE0NOGVNoAMFRVA=
x-amzn-RequestId: 2231ed6b-3efb-4a29-bad6-18031013c255
For the dcp12
catalog the unfiltered projects endpoint can return with a size of 1000 without an error
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp12&size=1000'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 717039
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:18:27 GMT
Via: 1.1 c361075814ee65e6523d12c5af3e7994.cloudfront.net (CloudFront)
X-Amz-Cf-Id: SemmnnfcqFY-lj4IiQpURQMYnQO5dXHAKfdoQn-7VOmkaX1Gya82kg==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-62182e5d-3f5bcf4b6fa69e4c07db031f;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OEwutEjtIAMFYUg=
x-amzn-RequestId: 297a56be-94ab-48c4-95d3-ceb8650f4261
Catalog dcp12
bundles
502 endpoint errors start occurring with a size of 966
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=966'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:47:46 GMT
Via: 1.1 89d40f9555bb19bc571952b32ca87398.cloudfront.net (CloudFront)
X-Amz-Cf-Id: yO9yjR-MVxyamc8y8RY6aoAaq8415QRPZuh0guVIC_MLu3sHDl29pA==
X-Amz-Cf-Pop: SFO5-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OE1AoH3hoAMF_bA=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 9f0e4d27-7d14-416f-ab35-442bde49ae50
(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=965'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 556724
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:48:11 GMT
Via: 1.1 b5546ff55405d525045e7263ba6db012.cloudfront.net (CloudFront)
X-Amz-Cf-Id: DSQOfpTVQzAMEuY8jTTlbq39O3CIJSCywqHpn9-3gwVo35IwL42MgA==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-62183552-1f9f5bff04106f416bd99141;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OE1E3EiyIAMFs8Q=
x-amzn-RequestId: 6c127f4e-a63f-4d34-969f-9cd8135e907b
504 Timeout errors start to occur with the size of 994, but it is inconsistant
2022-02-24 17:59:54,498 INFO MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=993
2022-02-24 18:00:24,163 INFO MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=994
2022-02-24 18:00:52,639 INFO MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=995
2022-02-24 18:01:21,714 INFO MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=996
2022-02-24 18:01:51,004 INFO MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=997
2022-02-24 18:02:20,159 INFO MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=998
2022-02-24 18:02:49,342 INFO MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=999
@hannes-ucsc to evaluate findings and come up with next steps.
It appears you missed part of the spike @amarjandu
As part of the spike determine the size of each project in an unfiltered projects endpoint response for both catalogues, dcp12 and dcp13.
{projectId: bytes}
for dcp12
and dcp13
from the the /index/projects
endpoint
{'dcp12': {'005d611a-14d5-4fbf-846e-571a1f874f70': 16833,
'027c51c6-0719-469f-a7f5-640fe57cbece': 11532,
'03c6fce7-789e-4e78-a27a-664d562bb738': 15989,
'04ad400c-58cb-40a5-bc2b-2279e13a910b': 174737,
'05657a59-9f9d-4bb9-b77b-24be13aa5cea': 11461,
'05be4f37-4506-429b-b112-506444507d62': 7558,
'07073c12-8006-4710-a00b-23abdb814904': 18757,
'074a9f88-729a-455d-bca5-0ce80edf0cea': 11530,
'0792db34-8047-4e62-802c-9177c9cd8e28': 24401,
'08b794a0-519c-4516-b184-c583746254c5': 9950,
'091cf39b-01bc-42e5-9437-f419a66c8a45': 14467,
'0c09fade-e079-4fde-8e60-6725b8c1d84b': 13559,
'0c3b7785-f74d-4091-8616-a68757e4c2a8': 12857,
'0d4b87ea-6e9e-4569-82e4-1343e0e3259f': 13145,
'0fd8f918-62d6-4b8b-ac35-4c53dd601f71': 10077,
'116965f3-f094-4769-9d28-ae675c1b569c': 10836,
'16ed4ad8-7319-46b2-8859-6fe1c1d73a82': 32319,
'1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': 12947,
'1cd1f41f-f81a-486b-a05b-66ec60f81dcf': 10878,
'1ce3b3dc-02f2-44a8-96da-d6d107b27a76': 7980,
'1defdada-a365-44ad-9b29-443b06bd11d6': 11990,
'2043c65a-1cf8-4828-a656-9e247d4e64f1': 17666,
'2084526b-a66f-4c40-bb89-6fd162f2eb38': 30836,
'2086eb05-10b9-432b-b7f0-169ccc49d270': 11795,
'20f37aaf-caa1-40e6-9123-be6ce8feb2d6': 16495,
'21ea8ddb-525f-4f1f-a820-31f0360399a2': 9325,
'23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': 10091,
'248fcf03-16c6-4a41-b6cc-aad4d894ca42': 11551,
'24c654a5-caa5-440a-8f02-582921f2db4a': 13382,
'2a64db43-1b55-4639-aabb-8dba0145689d': 9149,
'2a72a4e5-66b2-405a-bb7c-1e463e8febb0': 18103,
'2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': 14635,
'2af52a13-65cb-4973-b513-39be38f2df3f': 6657,
'2b38025d-a5ea-4c0f-b22e-367824bcaf4c': 61708,
'2d846095-8a33-4f3c-97d4-585bafac13b4': 12269,
'2ef3655a-973d-4d69-9b41-21fa4041eed7': 11173,
'2f676143-80c2-4bc6-b7b4-2613fe0fadf0': 13729,
'3089d311-f9ed-44dd-bb10-397059bad4dc': 20946,
'31887183-a72c-4308-9eac-c6140313f39c': 78347,
'34cba5e9-ecb1-4d81-bf08-48987cd63073': 9795,
'376a7f55-b876-4f60-9cf3-ed7bc83d5415': 12421,
'379ed69e-be05-48bc-af5e-a7fc589709bf': 15653,
'38449aea-70b5-40db-84b3-1e08f32efe34': 15205,
'38e44dd0-c3df-418e-9256-d0824748901f': 17806,
'3a694703-3084-4ece-9abe-d935fd5f6748': 15402,
'3c27d2dd-b180-4b2b-bf05-e2e418393fd1': 12084,
'3cfcdff5-dee1-4a7b-a591-c09c6e850b11': 14874,
'3e329187-a9c4-48ec-90e3-cc45f7c2311c': 8140,
'4037007b-0eff-4e6d-b7bd-8dd8eec80143': 14503,
'403c3e76-6814-4a2d-a580-5dd5de38c7ff': 46734,
'414acced-eba0-440f-b721-befbc5642bef': 11901,
'41fb1734-a121-4616-95c7-3b732c9433c7': 10866,
'42d4f8d4-5422-4b78-adae-e7c3c2ef511c': 15585,
'455b46e6-d8ea-4611-861e-de720a562ada': 20894,
'4a95101c-9ffc-4f30-a809-f04518a23803': 25576,
'4af795f7-3e1d-4341-b867-4ac0982b9efd': 11229,
'4bec484d-ca7a-47b4-8d48-8830e06ad6db': 11277,
'4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': 18090,
'4e6f083b-5b9a-4393-9890-2a83da8188f1': 18908,
'50151324-f3ed-4358-98af-ec352a940a61': 135225,
'504e0cee-1688-40fa-b936-361c4a831f87': 8135,
'5116c081-8be7-49c5-8ce0-73b887328aa9': 22242,
'51f02950-ee25-4f4b-8d07-59aa99bb3498': 17446,
'520afa10-f9d2-4e93-ab7a-26c4c863ce18': 7484,
'52b29aa4-c8d6-42b4-807a-b35be94469ca': 9791,
'52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': 8173,
'53c53cd4-8127-4e12-bc7f-8fe1610a715c': 22727,
'54aaa409-dc28-48c5-be26-d368b4a5d5c6': 20920,
'559bb888-7829-41f2-ace5-2c05c7eb81e9': 21304,
'56e73ccb-7ae9-4fae-a738-acfb69936d7a': 13966,
'577c946d-6de5-4b55-a854-cd3fde40bff2': 10747,
'58028aa8-0ed2-49ca-b60f-15e2ed5989d5': 16299,
'591af954-cdcd-4839-96d3-a0d1b1e885ac': 33292,
'5b328561-4a97-40ac-b7ad-6a90fc59d374': 134408,
'5b5f05b7-2482-468d-b76d-8f68c04a7a47': 11148,
'5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': 7105,
'5eafb94b-02d8-423e-81b8-3673da319ca0': 9157,
'5ee710d7-e2d5-4fe2-818d-15f5e31dae32': 9328,
'602628d7-c038-48a8-aa97-ffbb2cb44c9d': 17394,
'6072616c-8794-4b20-8f52-fb15992ea5a4': 8531,
'60ea42e1-af49-42f5-8164-d641fdb696bc': 7018,
'63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': 8009,
'65858543-530d-48a6-a670-f972b34dfe10': 21843,
'67a3de09-45b9-49c3-a068-ff4665daa50e': 12239,
'68df3629-d2d2-4eed-b0ab-a10e0f019b88': 8489,
'6c040a93-8cf8-4fd5-98de-2297eb07e9f6': 17331,
'7027adc6-c9c9-46f3-84ee-9badc3a4f53b': 18126,
'71436067-ac41-4ace-be1b-2fbcc2cb02fa': 7081,
'71eb5f6d-cee0-4297-b503-b1125909b8c7': 41059,
'74493e98-44fc-48b0-a58f-cc7e77268b59': 27268,
'74b6d569-3b11-42ef-b6b1-a0454522b4a0': 7784,
'75dbbce9-0cde-489c-88a7-93e8f92914a3': 9835,
'769a08d1-b8a4-4f1e-95f7-6071a9827555': 21688,
'783c9952-a4ae-4106-a6ce-56f20ce27f88': 11876,
'7880637a-35a1-4047-b422-b5eac2a2a358': 27821,
'78b2406d-bff2-46fc-8b61-20690e602227': 150223,
'79b13a2a-9ca1-42a4-97bd-70208a11bea6': 6192,
'7ac8822c-4ef0-4194-adf0-74290611b1c6': 23137,
'7adede6a-0ab7-45e6-9b67-ffe7466bec1f': 15108,
'7b947aa2-43a7-4082-afff-222a3e3a4635': 10092,
'7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': 29179,
'8185730f-4113-40d3-9cc3-929271784c2b': 25695,
'83f5188e-3bf7-4956-9544-cea4f8997756': 33488,
'842605c7-375a-47c5-9e2c-a71c2c00fcad': 11175,
'8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': 150775,
'8787c238-89ef-4636-a57d-3167e8b54a80': 8683,
'87d52a86-bdc7-440c-b84d-170f7dc346d9': 11126,
'88ec040b-8705-4f77-8f41-f81e57632f7d': 170912,
'8999b456-6fa6-438b-ab17-b62b1d8ec0c3': 10315,
'8a40ff19-e614-4c50-b23b-5c9e1d546bab': 110744,
'8ab8726d-81b9-4bd2-acc2-4d50bee786b4': 13879,
'8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': 29628,
'8c3c290d-dfff-4553-8868-54ce45f4ba7f': 27803,
'8d566d35-d8d3-4975-a351-be5e25e9b2ea': 69372,
'8dacb243-e918-4bd2-bb9a-aac6dc424161': 15434,
'90bd6933-40c0-48d4-8d76-778c103bf545': 14363,
'94023a08-611d-4f22-a8c9-90956e091b2e': 8532,
'946c5add-47d1-402a-97bb-a5af97e8bce7': 9506,
'955dfc2c-a8c6-4d04-aa4d-907610545d11': 10064,
'95f07e6e-6a73-4e1b-a880-c83996b3aa5c': 15066,
'962bd805-eb89-4c54-bad2-008e497d1307': 20180,
'99101928-d9b1-4aaf-b759-e97958ac7403': 9587,
'996120f9-e84f-409f-a01e-732ab58ca8b9': 12720,
'9d97f01f-9313-416e-9b07-560f048b2350': 63322,
'a004b150-1c36-4af6-9bbd-070c06dbc17d': 10666,
'a29952d9-925e-40f4-8a1c-274f118f1f51': 9935,
'a39728aa-70a0-4201-b0a2-81b7badf3e71': 22783,
'a60803bb-f7db-45cf-b529-95436152a801': 47589,
'a80a63f2-e223-4890-81b0-415855b89abc': 14909,
'a9301beb-e9fa-42fe-b75c-84e8a460c733': 601838,
'a96b71c0-78a7-42d1-88ce-83c78925cfeb': 8101,
'a991ef15-4d4a-4b80-a93e-c538b4b54127': 10359,
'a9c022b4-c771-4468-b769-cabcf9738de3': 17361,
'abe1a013-af7a-45ed-8c26-f3793c24a1f4': 17286,
'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': 63898,
'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': 23889,
'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': 14456,
'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': 8821,
'b4a7d12f-6c2f-40a3-9e35-9756997857e3': 11626,
'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': 20938,
'b7259878-436c-4274-bfff-ca76f4cb7892': 14386,
'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': 8996,
'b963bd4b-4bc1-4404-8425-69d74bc636b8': 29180,
'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': 35193,
'bd7104c9-a950-490e-9472-7d41c6b11c62': 26788,
'c1810dbc-16d2-45c3-b45e-3e675f88d87b': 32597,
'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': 28214,
'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': 146188,
'c4077b3c-5c98-4d26-a614-246d12c2e5d7': 29707,
'c41dffbf-ad83-447c-a0e1-13e689d9b258': 22433,
'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': 49294,
'c5f46615-68de-4cf4-bbc2-a0ae10f08243': 19482,
'c6ad8f9b-d26a-4811-b2ba-93d487978446': 9965,
'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': 31423,
'c893cb57-5c9f-4f26-9312-21b85be84313': 14034,
'cc95ff89-2e68-4a08-a234-480eca21ce79': 15763,
'ccd1f1ba-74ce-469b-9fc9-f6faea623358': 7146,
'ccef38d7-aa92-4010-9621-c4c7b1182647': 17268,
'cddab57b-6868-4be4-806f-395ed9dd635a': 17802,
'ce33dde2-382d-448c-b6ac-bfb424644f23': 10592,
'ce7b12ba-664f-4f79-8fc7-3de6b1892183': 17589,
'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': 15343,
'd2111fac-3fc4-4f42-9b6d-32cd6a828267': 8897,
'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': 49703,
'd3a4ceac-4d66-4984-9704-2570c0647a56': 12795,
'd3ac7c1b-5302-4804-b611-dad9f89c049d': 29543,
'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': 57138,
'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': 21298,
'da2747fa-2921-42e0-afd4-39ef57b2b88b': 23818,
'daf9d982-7ce6-43f6-ab51-272577290606': 81839,
'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': 7508,
'dc1a41f6-9e09-42a6-959e-3be23db6da56': 77281,
'dd7f2436-0c56-4709-bd17-e526bba4cc15': 18355,
'df88f39f-01a8-4b5b-92f4-3177d6c0f242': 12926,
'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': 143939,
'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': 8297,
'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': 6695,
'e57dc176-ab98-446b-90c2-89e0842152fd': 14580,
'e5d45579-1f5b-48c3-b568-320d93e7ca72': 17096,
'e77fed30-959d-4fad-bc15-a0a5a85c21d2': 6588,
'e8808cc8-4ca0-4096-80f2-bba73600cba6': 14324,
'eaefa1b6-dae1-4414-953b-17b0427d061e': 13700,
'ede2e0b4-6652-464f-abbc-0b2d964a25a0': 20005,
'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': 164365,
'ef1e3497-515e-4bbe-8d4c-10161854b699': 40945,
'efea6426-510a-4b60-9a19-277e52bfa815': 15699,
'f0f89c14-7460-4bab-9d42-22228a91f185': 21954,
'f2fe82f0-4454-4d84-b416-a885f3121e59': 84683,
'f48e7c39-cc67-4055-9d79-bc437892840c': 10165,
'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': 5724,
'f81efc03-9f56-4354-aabb-6ce819c3d414': 8088,
'f83165c5-e2ea-4d15-a5cf-33f3550bffde': 36002,
'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': 13457,
'f8aa201c-4ff1-45a4-890e-840d63459ca2': 42435,
'faeedcb0-e046-4be7-b1ad-80a3eeabb066': 52939},
'dcp13': {'005d611a-14d5-4fbf-846e-571a1f874f70': 16833,
'027c51c6-0719-469f-a7f5-640fe57cbece': 11532,
'03c6fce7-789e-4e78-a27a-664d562bb738': 17807,
'04ad400c-58cb-40a5-bc2b-2279e13a910b': 174737,
'05657a59-9f9d-4bb9-b77b-24be13aa5cea': 13261,
'05be4f37-4506-429b-b112-506444507d62': 9362,
'065e6c13-ad6b-46a3-8075-c3137eb03068': 137608,
'06c7dd8d-6cc6-4b79-b795-8805c47d36e1': 34866,
'07073c12-8006-4710-a00b-23abdb814904': 18757,
'074a9f88-729a-455d-bca5-0ce80edf0cea': 11530,
'0792db34-8047-4e62-802c-9177c9cd8e28': 24401,
'08b794a0-519c-4516-b184-c583746254c5': 9950,
'091cf39b-01bc-42e5-9437-f419a66c8a45': 14467,
'0c09fade-e079-4fde-8e60-6725b8c1d84b': 13559,
'0c3b7785-f74d-4091-8616-a68757e4c2a8': 12857,
'0d4b87ea-6e9e-4569-82e4-1343e0e3259f': 13145,
'0fd8f918-62d6-4b8b-ac35-4c53dd601f71': 10077,
'10201832-7c73-4033-9b65-3ef13d81656a': 142535,
'116965f3-f094-4769-9d28-ae675c1b569c': 10836,
'16ed4ad8-7319-46b2-8859-6fe1c1d73a82': 32319,
'1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': 12947,
'1cd1f41f-f81a-486b-a05b-66ec60f81dcf': 10878,
'1ce3b3dc-02f2-44a8-96da-d6d107b27a76': 9788,
'1dddae6e-3753-48af-b20e-fa22abad125d': 79837,
'1defdada-a365-44ad-9b29-443b06bd11d6': 11990,
'1eb69a39-b5b2-41ec-afae-5fe37f272756': 11780,
'2043c65a-1cf8-4828-a656-9e247d4e64f1': 17666,
'2084526b-a66f-4c40-bb89-6fd162f2eb38': 30836,
'2086eb05-10b9-432b-b7f0-169ccc49d270': 11795,
'20f37aaf-caa1-40e6-9123-be6ce8feb2d6': 16495,
'21ea8ddb-525f-4f1f-a820-31f0360399a2': 9325,
'23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': 13399,
'248fcf03-16c6-4a41-b6cc-aad4d894ca42': 11551,
'24c654a5-caa5-440a-8f02-582921f2db4a': 13382,
'2a64db43-1b55-4639-aabb-8dba0145689d': 9149,
'2a72a4e5-66b2-405a-bb7c-1e463e8febb0': 19931,
'2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': 14635,
'2af52a13-65cb-4973-b513-39be38f2df3f': 6657,
'2b38025d-a5ea-4c0f-b22e-367824bcaf4c': 61708,
'2d846095-8a33-4f3c-97d4-585bafac13b4': 14101,
'2ef3655a-973d-4d69-9b41-21fa4041eed7': 11173,
'2f676143-80c2-4bc6-b7b4-2613fe0fadf0': 13729,
'3089d311-f9ed-44dd-bb10-397059bad4dc': 20946,
'31887183-a72c-4308-9eac-c6140313f39c': 78347,
'34cba5e9-ecb1-4d81-bf08-48987cd63073': 9795,
'376a7f55-b876-4f60-9cf3-ed7bc83d5415': 12421,
'379ed69e-be05-48bc-af5e-a7fc589709bf': 15653,
'38449aea-70b5-40db-84b3-1e08f32efe34': 15205,
'38e44dd0-c3df-418e-9256-d0824748901f': 17806,
'3a694703-3084-4ece-9abe-d935fd5f6748': 17815,
'3c27d2dd-b180-4b2b-bf05-e2e418393fd1': 12084,
'3cfcdff5-dee1-4a7b-a591-c09c6e850b11': 14874,
'3e329187-a9c4-48ec-90e3-cc45f7c2311c': 8140,
'4037007b-0eff-4e6d-b7bd-8dd8eec80143': 14503,
'403c3e76-6814-4a2d-a580-5dd5de38c7ff': 46734,
'414acced-eba0-440f-b721-befbc5642bef': 11901,
'41fb1734-a121-4616-95c7-3b732c9433c7': 10866,
'42d4f8d4-5422-4b78-adae-e7c3c2ef511c': 15585,
'455b46e6-d8ea-4611-861e-de720a562ada': 20894,
'4a95101c-9ffc-4f30-a809-f04518a23803': 25576,
'4af795f7-3e1d-4341-b867-4ac0982b9efd': 11229,
'4bec484d-ca7a-47b4-8d48-8830e06ad6db': 11277,
'4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': 18090,
'4e6f083b-5b9a-4393-9890-2a83da8188f1': 18908,
'50151324-f3ed-4358-98af-ec352a940a61': 135225,
'504e0cee-1688-40fa-b936-361c4a831f87': 8135,
'5116c081-8be7-49c5-8ce0-73b887328aa9': 22242,
'51f02950-ee25-4f4b-8d07-59aa99bb3498': 17446,
'520afa10-f9d2-4e93-ab7a-26c4c863ce18': 9294,
'52b29aa4-c8d6-42b4-807a-b35be94469ca': 9791,
'52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': 8173,
'53c53cd4-8127-4e12-bc7f-8fe1610a715c': 22727,
'54aaa409-dc28-48c5-be26-d368b4a5d5c6': 20920,
'559bb888-7829-41f2-ace5-2c05c7eb81e9': 21304,
'56e73ccb-7ae9-4fae-a738-acfb69936d7a': 13966,
'577c946d-6de5-4b55-a854-cd3fde40bff2': 10747,
'58028aa8-0ed2-49ca-b60f-15e2ed5989d5': 16485,
'591af954-cdcd-4839-96d3-a0d1b1e885ac': 33292,
'5b328561-4a97-40ac-b7ad-6a90fc59d374': 134408,
'5b5f05b7-2482-468d-b76d-8f68c04a7a47': 11148,
'5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': 7105,
'5eafb94b-02d8-423e-81b8-3673da319ca0': 9157,
'5ee710d7-e2d5-4fe2-818d-15f5e31dae32': 9328,
'602628d7-c038-48a8-aa97-ffbb2cb44c9d': 17394,
'6072616c-8794-4b20-8f52-fb15992ea5a4': 8531,
'60ea42e1-af49-42f5-8164-d641fdb696bc': 7018,
'63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': 8009,
'65858543-530d-48a6-a670-f972b34dfe10': 21843,
'67a3de09-45b9-49c3-a068-ff4665daa50e': 14771,
'68df3629-d2d2-4eed-b0ab-a10e0f019b88': 8489,
'6c040a93-8cf8-4fd5-98de-2297eb07e9f6': 17331,
'6f89a7f3-8d4a-4344-aa4f-eccfe7e91076': 13836,
'7027adc6-c9c9-46f3-84ee-9badc3a4f53b': 18126,
'71436067-ac41-4ace-be1b-2fbcc2cb02fa': 7081,
'71eb5f6d-cee0-4297-b503-b1125909b8c7': 41059,
'74493e98-44fc-48b0-a58f-cc7e77268b59': 27268,
'74b6d569-3b11-42ef-b6b1-a0454522b4a0': 7784,
'75dbbce9-0cde-489c-88a7-93e8f92914a3': 9835,
'769a08d1-b8a4-4f1e-95f7-6071a9827555': 21688,
'783c9952-a4ae-4106-a6ce-56f20ce27f88': 11876,
'7880637a-35a1-4047-b422-b5eac2a2a358': 27821,
'78b2406d-bff2-46fc-8b61-20690e602227': 151894,
'78d7805b-fdc8-472b-8058-d92cf886f7a4': 48912,
'79b13a2a-9ca1-42a4-97bd-70208a11bea6': 6192,
'7ac8822c-4ef0-4194-adf0-74290611b1c6': 23137,
'7adede6a-0ab7-45e6-9b67-ffe7466bec1f': 15108,
'7b947aa2-43a7-4082-afff-222a3e3a4635': 10092,
'7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': 29179,
'8185730f-4113-40d3-9cc3-929271784c2b': 25695,
'83f5188e-3bf7-4956-9544-cea4f8997756': 33488,
'842605c7-375a-47c5-9e2c-a71c2c00fcad': 11175,
'8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': 152438,
'85a9263b-0887-48ed-ab1a-ddfa773727b6': 25211,
'8787c238-89ef-4636-a57d-3167e8b54a80': 8683,
'87d52a86-bdc7-440c-b84d-170f7dc346d9': 11126,
'88ec040b-8705-4f77-8f41-f81e57632f7d': 172954,
'8999b456-6fa6-438b-ab17-b62b1d8ec0c3': 10315,
'8a40ff19-e614-4c50-b23b-5c9e1d546bab': 110744,
'8ab8726d-81b9-4bd2-acc2-4d50bee786b4': 13879,
'8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': 29628,
'8c3c290d-dfff-4553-8868-54ce45f4ba7f': 31036,
'8d566d35-d8d3-4975-a351-be5e25e9b2ea': 69372,
'8dacb243-e918-4bd2-bb9a-aac6dc424161': 15434,
'90bd6933-40c0-48d4-8d76-778c103bf545': 14363,
'94023a08-611d-4f22-a8c9-90956e091b2e': 8532,
'946c5add-47d1-402a-97bb-a5af97e8bce7': 9506,
'955dfc2c-a8c6-4d04-aa4d-907610545d11': 10064,
'95f07e6e-6a73-4e1b-a880-c83996b3aa5c': 15066,
'962bd805-eb89-4c54-bad2-008e497d1307': 20180,
'99101928-d9b1-4aaf-b759-e97958ac7403': 12155,
'996120f9-e84f-409f-a01e-732ab58ca8b9': 12720,
'9c20a245-f2c0-43ae-82c9-2232ec6b594f': 12240,
'a004b150-1c36-4af6-9bbd-070c06dbc17d': 10666,
'a29952d9-925e-40f4-8a1c-274f118f1f51': 9935,
'a39728aa-70a0-4201-b0a2-81b7badf3e71': 22783,
'a60803bb-f7db-45cf-b529-95436152a801': 47589,
'a80a63f2-e223-4890-81b0-415855b89abc': 14909,
'a9301beb-e9fa-42fe-b75c-84e8a460c733': 601838,
'a96b71c0-78a7-42d1-88ce-83c78925cfeb': 8101,
'a991ef15-4d4a-4b80-a93e-c538b4b54127': 10359,
'a9c022b4-c771-4468-b769-cabcf9738de3': 17361,
'abe1a013-af7a-45ed-8c26-f3793c24a1f4': 17286,
'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': 63898,
'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': 23889,
'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': 14456,
'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': 8821,
'b4a7d12f-6c2f-40a3-9e35-9756997857e3': 11626,
'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': 20938,
'b7259878-436c-4274-bfff-ca76f4cb7892': 14386,
'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': 8996,
'b963bd4b-4bc1-4404-8425-69d74bc636b8': 29180,
'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': 35193,
'bd7104c9-a950-490e-9472-7d41c6b11c62': 26788,
'c1810dbc-16d2-45c3-b45e-3e675f88d87b': 32597,
'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': 28214,
'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': 146188,
'c4077b3c-5c98-4d26-a614-246d12c2e5d7': 29707,
'c41dffbf-ad83-447c-a0e1-13e689d9b258': 22433,
'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': 49294,
'c5f46615-68de-4cf4-bbc2-a0ae10f08243': 19482,
'c6ad8f9b-d26a-4811-b2ba-93d487978446': 9965,
'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': 31423,
'c893cb57-5c9f-4f26-9312-21b85be84313': 14034,
'cc95ff89-2e68-4a08-a234-480eca21ce79': 15763,
'ccd1f1ba-74ce-469b-9fc9-f6faea623358': 8938,
'ccef38d7-aa92-4010-9621-c4c7b1182647': 22121,
'cd61771b-661a-4e19-b269-6e5d95350de6': 237934,
'cddab57b-6868-4be4-806f-395ed9dd635a': 17802,
'ce33dde2-382d-448c-b6ac-bfb424644f23': 10592,
'ce7b12ba-664f-4f79-8fc7-3de6b1892183': 17589,
'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': 15343,
'd2111fac-3fc4-4f42-9b6d-32cd6a828267': 8897,
'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': 49703,
'd3a4ceac-4d66-4984-9704-2570c0647a56': 12795,
'd3ac7c1b-5302-4804-b611-dad9f89c049d': 29543,
'd6225aee-8f0e-4b20-a20c-682509a9ea14': 29005,
'd71c76d3-3670-4774-a9cf-034249d37c60': 54815,
'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': 57138,
'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': 21298,
'da2747fa-2921-42e0-afd4-39ef57b2b88b': 23818,
'daf9d982-7ce6-43f6-ab51-272577290606': 81839,
'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': 7508,
'dbd836cf-bfc2-41f0-9834-41cc6c0b235a': 8890,
'dc1a41f6-9e09-42a6-959e-3be23db6da56': 77281,
'dd7ada84-3f14-4765-b7ce-9b64642bb3dc': 11503,
'dd7f2436-0c56-4709-bd17-e526bba4cc15': 18355,
'df88f39f-01a8-4b5b-92f4-3177d6c0f242': 12926,
'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': 143939,
'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': 8297,
'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': 6695,
'e57dc176-ab98-446b-90c2-89e0842152fd': 14580,
'e5d45579-1f5b-48c3-b568-320d93e7ca72': 17096,
'e77fed30-959d-4fad-bc15-a0a5a85c21d2': 6588,
'e8808cc8-4ca0-4096-80f2-bba73600cba6': 17634,
'eaefa1b6-dae1-4414-953b-17b0427d061e': 13700,
'ede2e0b4-6652-464f-abbc-0b2d964a25a0': 20005,
'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': 164365,
'ef1e3497-515e-4bbe-8d4c-10161854b699': 40945,
'efea6426-510a-4b60-9a19-277e52bfa815': 15699,
'f0f89c14-7460-4bab-9d42-22228a91f185': 21954,
'f2fe82f0-4454-4d84-b416-a885f3121e59': 84683,
'f48e7c39-cc67-4055-9d79-bc437892840c': 10165,
'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': 7527,
'f81efc03-9f56-4354-aabb-6ce819c3d414': 8088,
'f83165c5-e2ea-4d15-a5cf-33f3550bffde': 36002,
'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': 13457,
'f8aa201c-4ff1-45a4-890e-840d63459ca2': 42435,
'faeedcb0-e046-4be7-b1ad-80a3eeabb066': 52939}}
@hannes-ucsc : " @amarjandu to continue spike to convert the above data structure to a single dictionary with keys being the project IDs and the values being a list containing the two sizes for dcp12 and dcp13, respectively."
projectId: [bytes dcp12, bytes dcp13]
{'005d611a-14d5-4fbf-846e-571a1f874f70': [16833, 16833],
'027c51c6-0719-469f-a7f5-640fe57cbece': [11532, 11532],
'03c6fce7-789e-4e78-a27a-664d562bb738': [15989, 17807],
'04ad400c-58cb-40a5-bc2b-2279e13a910b': [174737, 174737],
'05657a59-9f9d-4bb9-b77b-24be13aa5cea': [11461, 13261],
'05be4f37-4506-429b-b112-506444507d62': [7558, 9362],
'065e6c13-ad6b-46a3-8075-c3137eb03068': [None, 137608],
'06c7dd8d-6cc6-4b79-b795-8805c47d36e1': [None, 34866],
'07073c12-8006-4710-a00b-23abdb814904': [18757, 18757],
'074a9f88-729a-455d-bca5-0ce80edf0cea': [11530, 11530],
'0792db34-8047-4e62-802c-9177c9cd8e28': [24401, 24401],
'08b794a0-519c-4516-b184-c583746254c5': [9950, 9950],
'091cf39b-01bc-42e5-9437-f419a66c8a45': [14467, 14467],
'0c09fade-e079-4fde-8e60-6725b8c1d84b': [13559, 13559],
'0c3b7785-f74d-4091-8616-a68757e4c2a8': [12857, 12857],
'0d4b87ea-6e9e-4569-82e4-1343e0e3259f': [13145, 13145],
'0fd8f918-62d6-4b8b-ac35-4c53dd601f71': [10077, 10077],
'10201832-7c73-4033-9b65-3ef13d81656a': [None, 142535],
'116965f3-f094-4769-9d28-ae675c1b569c': [10836, 10836],
'16ed4ad8-7319-46b2-8859-6fe1c1d73a82': [32319, 32319],
'1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': [12947, 12947],
'1cd1f41f-f81a-486b-a05b-66ec60f81dcf': [10878, 10878],
'1ce3b3dc-02f2-44a8-96da-d6d107b27a76': [7980, 9788],
'1dddae6e-3753-48af-b20e-fa22abad125d': [None, 79837],
'1defdada-a365-44ad-9b29-443b06bd11d6': [11990, 11990],
'1eb69a39-b5b2-41ec-afae-5fe37f272756': [None, 11780],
'2043c65a-1cf8-4828-a656-9e247d4e64f1': [17666, 17666],
'2084526b-a66f-4c40-bb89-6fd162f2eb38': [30836, 30836],
'2086eb05-10b9-432b-b7f0-169ccc49d270': [11795, 11795],
'20f37aaf-caa1-40e6-9123-be6ce8feb2d6': [16495, 16495],
'21ea8ddb-525f-4f1f-a820-31f0360399a2': [9325, 9325],
'23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': [10091, 13399],
'248fcf03-16c6-4a41-b6cc-aad4d894ca42': [11551, 11551],
'24c654a5-caa5-440a-8f02-582921f2db4a': [13382, 13382],
'2a64db43-1b55-4639-aabb-8dba0145689d': [9149, 9149],
'2a72a4e5-66b2-405a-bb7c-1e463e8febb0': [18103, 19931],
'2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': [14635, 14635],
'2af52a13-65cb-4973-b513-39be38f2df3f': [6657, 6657],
'2b38025d-a5ea-4c0f-b22e-367824bcaf4c': [61708, 61708],
'2d846095-8a33-4f3c-97d4-585bafac13b4': [12269, 14101],
'2ef3655a-973d-4d69-9b41-21fa4041eed7': [11173, 11173],
'2f676143-80c2-4bc6-b7b4-2613fe0fadf0': [13729, 13729],
'3089d311-f9ed-44dd-bb10-397059bad4dc': [20946, 20946],
'31887183-a72c-4308-9eac-c6140313f39c': [78347, 78347],
'34cba5e9-ecb1-4d81-bf08-48987cd63073': [9795, 9795],
'376a7f55-b876-4f60-9cf3-ed7bc83d5415': [12421, 12421],
'379ed69e-be05-48bc-af5e-a7fc589709bf': [15653, 15653],
'38449aea-70b5-40db-84b3-1e08f32efe34': [15205, 15205],
'38e44dd0-c3df-418e-9256-d0824748901f': [17806, 17806],
'3a694703-3084-4ece-9abe-d935fd5f6748': [15402, 17815],
'3c27d2dd-b180-4b2b-bf05-e2e418393fd1': [12084, 12084],
'3cfcdff5-dee1-4a7b-a591-c09c6e850b11': [14874, 14874],
'3e329187-a9c4-48ec-90e3-cc45f7c2311c': [8140, 8140],
'4037007b-0eff-4e6d-b7bd-8dd8eec80143': [14503, 14503],
'403c3e76-6814-4a2d-a580-5dd5de38c7ff': [46734, 46734],
'414acced-eba0-440f-b721-befbc5642bef': [11901, 11901],
'41fb1734-a121-4616-95c7-3b732c9433c7': [10866, 10866],
'42d4f8d4-5422-4b78-adae-e7c3c2ef511c': [15585, 15585],
'455b46e6-d8ea-4611-861e-de720a562ada': [20894, 20894],
'4a95101c-9ffc-4f30-a809-f04518a23803': [25576, 25576],
'4af795f7-3e1d-4341-b867-4ac0982b9efd': [11229, 11229],
'4bec484d-ca7a-47b4-8d48-8830e06ad6db': [11277, 11277],
'4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': [18090, 18090],
'4e6f083b-5b9a-4393-9890-2a83da8188f1': [18908, 18908],
'50151324-f3ed-4358-98af-ec352a940a61': [135225, 135225],
'504e0cee-1688-40fa-b936-361c4a831f87': [8135, 8135],
'5116c081-8be7-49c5-8ce0-73b887328aa9': [22242, 22242],
'51f02950-ee25-4f4b-8d07-59aa99bb3498': [17446, 17446],
'520afa10-f9d2-4e93-ab7a-26c4c863ce18': [7484, 9294],
'52b29aa4-c8d6-42b4-807a-b35be94469ca': [9791, 9791],
'52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': [8173, 8173],
'53c53cd4-8127-4e12-bc7f-8fe1610a715c': [22727, 22727],
'54aaa409-dc28-48c5-be26-d368b4a5d5c6': [20920, 20920],
'559bb888-7829-41f2-ace5-2c05c7eb81e9': [21304, 21304],
'56e73ccb-7ae9-4fae-a738-acfb69936d7a': [13966, 13966],
'577c946d-6de5-4b55-a854-cd3fde40bff2': [10747, 10747],
'58028aa8-0ed2-49ca-b60f-15e2ed5989d5': [16299, 16485],
'591af954-cdcd-4839-96d3-a0d1b1e885ac': [33292, 33292],
'5b328561-4a97-40ac-b7ad-6a90fc59d374': [134408, 134408],
'5b5f05b7-2482-468d-b76d-8f68c04a7a47': [11148, 11148],
'5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': [7105, 7105],
'5eafb94b-02d8-423e-81b8-3673da319ca0': [9157, 9157],
'5ee710d7-e2d5-4fe2-818d-15f5e31dae32': [9328, 9328],
'602628d7-c038-48a8-aa97-ffbb2cb44c9d': [17394, 17394],
'6072616c-8794-4b20-8f52-fb15992ea5a4': [8531, 8531],
'60ea42e1-af49-42f5-8164-d641fdb696bc': [7018, 7018],
'63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': [8009, 8009],
'65858543-530d-48a6-a670-f972b34dfe10': [21843, 21843],
'67a3de09-45b9-49c3-a068-ff4665daa50e': [12239, 14771],
'68df3629-d2d2-4eed-b0ab-a10e0f019b88': [8489, 8489],
'6c040a93-8cf8-4fd5-98de-2297eb07e9f6': [17331, 17331],
'6f89a7f3-8d4a-4344-aa4f-eccfe7e91076': [None, 13836],
'7027adc6-c9c9-46f3-84ee-9badc3a4f53b': [18126, 18126],
'71436067-ac41-4ace-be1b-2fbcc2cb02fa': [7081, 7081],
'71eb5f6d-cee0-4297-b503-b1125909b8c7': [41059, 41059],
'74493e98-44fc-48b0-a58f-cc7e77268b59': [27268, 27268],
'74b6d569-3b11-42ef-b6b1-a0454522b4a0': [7784, 7784],
'75dbbce9-0cde-489c-88a7-93e8f92914a3': [9835, 9835],
'769a08d1-b8a4-4f1e-95f7-6071a9827555': [21688, 21688],
'783c9952-a4ae-4106-a6ce-56f20ce27f88': [11876, 11876],
'7880637a-35a1-4047-b422-b5eac2a2a358': [27821, 27821],
'78b2406d-bff2-46fc-8b61-20690e602227': [150223, 151894],
'78d7805b-fdc8-472b-8058-d92cf886f7a4': [None, 48912],
'79b13a2a-9ca1-42a4-97bd-70208a11bea6': [6192, 6192],
'7ac8822c-4ef0-4194-adf0-74290611b1c6': [23137, 23137],
'7adede6a-0ab7-45e6-9b67-ffe7466bec1f': [15108, 15108],
'7b947aa2-43a7-4082-afff-222a3e3a4635': [10092, 10092],
'7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': [29179, 29179],
'8185730f-4113-40d3-9cc3-929271784c2b': [25695, 25695],
'83f5188e-3bf7-4956-9544-cea4f8997756': [33488, 33488],
'842605c7-375a-47c5-9e2c-a71c2c00fcad': [11175, 11175],
'8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': [150775, 152438],
'85a9263b-0887-48ed-ab1a-ddfa773727b6': [None, 25211],
'8787c238-89ef-4636-a57d-3167e8b54a80': [8683, 8683],
'87d52a86-bdc7-440c-b84d-170f7dc346d9': [11126, 11126],
'88ec040b-8705-4f77-8f41-f81e57632f7d': [170912, 172954],
'8999b456-6fa6-438b-ab17-b62b1d8ec0c3': [10315, 10315],
'8a40ff19-e614-4c50-b23b-5c9e1d546bab': [110744, 110744],
'8ab8726d-81b9-4bd2-acc2-4d50bee786b4': [13879, 13879],
'8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': [29628, 29628],
'8c3c290d-dfff-4553-8868-54ce45f4ba7f': [27803, 31036],
'8d566d35-d8d3-4975-a351-be5e25e9b2ea': [69372, 69372],
'8dacb243-e918-4bd2-bb9a-aac6dc424161': [15434, 15434],
'90bd6933-40c0-48d4-8d76-778c103bf545': [14363, 14363],
'94023a08-611d-4f22-a8c9-90956e091b2e': [8532, 8532],
'946c5add-47d1-402a-97bb-a5af97e8bce7': [9506, 9506],
'955dfc2c-a8c6-4d04-aa4d-907610545d11': [10064, 10064],
'95f07e6e-6a73-4e1b-a880-c83996b3aa5c': [15066, 15066],
'962bd805-eb89-4c54-bad2-008e497d1307': [20180, 20180],
'99101928-d9b1-4aaf-b759-e97958ac7403': [9587, 12155],
'996120f9-e84f-409f-a01e-732ab58ca8b9': [12720, 12720],
'9c20a245-f2c0-43ae-82c9-2232ec6b594f': [None, 12240],
'9d97f01f-9313-416e-9b07-560f048b2350': [63322, None],
'a004b150-1c36-4af6-9bbd-070c06dbc17d': [10666, 10666],
'a29952d9-925e-40f4-8a1c-274f118f1f51': [9935, 9935],
'a39728aa-70a0-4201-b0a2-81b7badf3e71': [22783, 22783],
'a60803bb-f7db-45cf-b529-95436152a801': [47589, 47589],
'a80a63f2-e223-4890-81b0-415855b89abc': [14909, 14909],
'a9301beb-e9fa-42fe-b75c-84e8a460c733': [601838, 601838],
'a96b71c0-78a7-42d1-88ce-83c78925cfeb': [8101, 8101],
'a991ef15-4d4a-4b80-a93e-c538b4b54127': [10359, 10359],
'a9c022b4-c771-4468-b769-cabcf9738de3': [17361, 17361],
'abe1a013-af7a-45ed-8c26-f3793c24a1f4': [17286, 17286],
'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': [63898, 63898],
'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': [23889, 23889],
'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': [14456, 14456],
'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': [8821, 8821],
'b4a7d12f-6c2f-40a3-9e35-9756997857e3': [11626, 11626],
'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': [20938, 20938],
'b7259878-436c-4274-bfff-ca76f4cb7892': [14386, 14386],
'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': [8996, 8996],
'b963bd4b-4bc1-4404-8425-69d74bc636b8': [29180, 29180],
'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': [35193, 35193],
'bd7104c9-a950-490e-9472-7d41c6b11c62': [26788, 26788],
'c1810dbc-16d2-45c3-b45e-3e675f88d87b': [32597, 32597],
'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': [28214, 28214],
'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': [146188, 146188],
'c4077b3c-5c98-4d26-a614-246d12c2e5d7': [29707, 29707],
'c41dffbf-ad83-447c-a0e1-13e689d9b258': [22433, 22433],
'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': [49294, 49294],
'c5f46615-68de-4cf4-bbc2-a0ae10f08243': [19482, 19482],
'c6ad8f9b-d26a-4811-b2ba-93d487978446': [9965, 9965],
'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': [31423, 31423],
'c893cb57-5c9f-4f26-9312-21b85be84313': [14034, 14034],
'cc95ff89-2e68-4a08-a234-480eca21ce79': [15763, 15763],
'ccd1f1ba-74ce-469b-9fc9-f6faea623358': [7146, 8938],
'ccef38d7-aa92-4010-9621-c4c7b1182647': [17268, 22121],
'cd61771b-661a-4e19-b269-6e5d95350de6': [None, 237934],
'cddab57b-6868-4be4-806f-395ed9dd635a': [17802, 17802],
'ce33dde2-382d-448c-b6ac-bfb424644f23': [10592, 10592],
'ce7b12ba-664f-4f79-8fc7-3de6b1892183': [17589, 17589],
'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': [15343, 15343],
'd2111fac-3fc4-4f42-9b6d-32cd6a828267': [8897, 8897],
'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': [49703, 49703],
'd3a4ceac-4d66-4984-9704-2570c0647a56': [12795, 12795],
'd3ac7c1b-5302-4804-b611-dad9f89c049d': [29543, 29543],
'd6225aee-8f0e-4b20-a20c-682509a9ea14': [None, 29005],
'd71c76d3-3670-4774-a9cf-034249d37c60': [None, 54815],
'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': [57138, 57138],
'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': [21298, 21298],
'da2747fa-2921-42e0-afd4-39ef57b2b88b': [23818, 23818],
'daf9d982-7ce6-43f6-ab51-272577290606': [81839, 81839],
'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': [7508, 7508],
'dbd836cf-bfc2-41f0-9834-41cc6c0b235a': [None, 8890],
'dc1a41f6-9e09-42a6-959e-3be23db6da56': [77281, 77281],
'dd7ada84-3f14-4765-b7ce-9b64642bb3dc': [None, 11503],
'dd7f2436-0c56-4709-bd17-e526bba4cc15': [18355, 18355],
'df88f39f-01a8-4b5b-92f4-3177d6c0f242': [12926, 12926],
'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': [143939, 143939],
'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': [8297, 8297],
'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': [6695, 6695],
'e57dc176-ab98-446b-90c2-89e0842152fd': [14580, 14580],
'e5d45579-1f5b-48c3-b568-320d93e7ca72': [17096, 17096],
'e77fed30-959d-4fad-bc15-a0a5a85c21d2': [6588, 6588],
'e8808cc8-4ca0-4096-80f2-bba73600cba6': [14324, 17634],
'eaefa1b6-dae1-4414-953b-17b0427d061e': [13700, 13700],
'ede2e0b4-6652-464f-abbc-0b2d964a25a0': [20005, 20005],
'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': [164365, 164365],
'ef1e3497-515e-4bbe-8d4c-10161854b699': [40945, 40945],
'efea6426-510a-4b60-9a19-277e52bfa815': [15699, 15699],
'f0f89c14-7460-4bab-9d42-22228a91f185': [21954, 21954],
'f2fe82f0-4454-4d84-b416-a885f3121e59': [84683, 84683],
'f48e7c39-cc67-4055-9d79-bc437892840c': [10165, 10165],
'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': [5724, 7527],
'f81efc03-9f56-4354-aabb-6ce819c3d414': [8088, 8088],
'f83165c5-e2ea-4d15-a5cf-33f3550bffde': [36002, 36002],
'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': [13457, 13457],
'f8aa201c-4ff1-45a4-890e-840d63459ca2': [42435, 42435],
'faeedcb0-e046-4be7-b1ad-80a3eeabb066': [52939, 52939]}
@hannes-ucsc to review.
Just the diff:
{
"03c6fce7-789e-4e78-a27a-664d562bb738": [
15989,
17807
],
"05657a59-9f9d-4bb9-b77b-24be13aa5cea": [
11461,
13261
],
"05be4f37-4506-429b-b112-506444507d62": [
7558,
9362
],
"065e6c13-ad6b-46a3-8075-c3137eb03068": [
null,
137608
],
"06c7dd8d-6cc6-4b79-b795-8805c47d36e1": [
null,
34866
],
"10201832-7c73-4033-9b65-3ef13d81656a": [
null,
142535
],
"1ce3b3dc-02f2-44a8-96da-d6d107b27a76": [
7980,
9788
],
"1dddae6e-3753-48af-b20e-fa22abad125d": [
null,
79837
],
"1eb69a39-b5b2-41ec-afae-5fe37f272756": [
null,
11780
],
"23587fb3-1a4a-4f58-ad74-cc9a4cb4c254": [
10091,
13399
],
"2a72a4e5-66b2-405a-bb7c-1e463e8febb0": [
18103,
19931
],
"2d846095-8a33-4f3c-97d4-585bafac13b4": [
12269,
14101
],
"3a694703-3084-4ece-9abe-d935fd5f6748": [
15402,
17815
],
"520afa10-f9d2-4e93-ab7a-26c4c863ce18": [
7484,
9294
],
"58028aa8-0ed2-49ca-b60f-15e2ed5989d5": [
16299,
16485
],
"67a3de09-45b9-49c3-a068-ff4665daa50e": [
12239,
14771
],
"6f89a7f3-8d4a-4344-aa4f-eccfe7e91076": [
null,
13836
],
"78b2406d-bff2-46fc-8b61-20690e602227": [
150223,
151894
],
"78d7805b-fdc8-472b-8058-d92cf886f7a4": [
null,
48912
],
"8559a8ed-5d8c-4fb6-bde8-ab639cebf03c": [
150775,
152438
],
"85a9263b-0887-48ed-ab1a-ddfa773727b6": [
null,
25211
],
"88ec040b-8705-4f77-8f41-f81e57632f7d": [
170912,
172954
],
"8c3c290d-dfff-4553-8868-54ce45f4ba7f": [
27803,
31036
],
"99101928-d9b1-4aaf-b759-e97958ac7403": [
9587,
12155
],
"9c20a245-f2c0-43ae-82c9-2232ec6b594f": [
null,
12240
],
"9d97f01f-9313-416e-9b07-560f048b2350": [
63322,
null
],
"ccd1f1ba-74ce-469b-9fc9-f6faea623358": [
7146,
8938
],
"ccef38d7-aa92-4010-9621-c4c7b1182647": [
17268,
22121
],
"cd61771b-661a-4e19-b269-6e5d95350de6": [
null,
237934
],
"d6225aee-8f0e-4b20-a20c-682509a9ea14": [
null,
29005
],
"d71c76d3-3670-4774-a9cf-034249d37c60": [
null,
54815
],
"dbd836cf-bfc2-41f0-9834-41cc6c0b235a": [
null,
8890
],
"dd7ada84-3f14-4765-b7ce-9b64642bb3dc": [
null,
11503
],
"e8808cc8-4ca0-4096-80f2-bba73600cba6": [
14324,
17634
],
"f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0": [
5724,
7527
]
}
So it appears that the solution is to limit /index/{project,bundles}
to a page size of 100 while leaving the other endpoints at 1000.
Index: src/azul/plugins/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/plugins/__init__.py b/src/azul/plugins/__init__.py
--- a/src/azul/plugins/__init__.py (revision b036a13ac9d74a9c12368b8cb84e6053b728e27b)
+++ b/src/azul/plugins/__init__.py (date 1647278502944)
@@ -68,7 +68,8 @@
Translation = Mapping[str, str]
-class ServiceConfig(NamedTuple):
+@attr.s(frozen=True, auto_attribs=True, kw_only=True)
+class ServiceConfig:
# Except otherwise noted the attributes were previously held in a JSON file
# called `request_config.json`
translation: Translation
@@ -79,8 +80,9 @@
autocomplete_mapping_config: Mapping[str, Mapping[str, Union[str, Sequence[str]]]]
# This used to be defined in a text file called `order_config`
order_config: Sequence[str]
-
- source_id_facet = 'sourceId'
+ source_id_facet: str
+ # Upper bound on pages size per endpoint i.e. outer entity type
+ page_size_limits: Mapping[str, int]
T = TypeVar('T', bound='Plugin')
Index: src/azul/plugins/metadata/hca/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/plugins/metadata/hca/__init__.py b/src/azul/plugins/metadata/hca/__init__.py
--- a/src/azul/plugins/metadata/hca/__init__.py (revision b036a13ac9d74a9c12368b8cb84e6053b728e27b)
+++ b/src/azul/plugins/metadata/hca/__init__.py (date 1647278420808)
@@ -165,6 +165,7 @@
}
def service_config(self) -> ServiceConfig:
+ source_id_facet = 'sourceId'
return ServiceConfig(
# FIXME: Detect invalid values in translation
# https://github.com/DataBiosphere/azul/issues/3071
@@ -232,13 +233,14 @@
"entryId": "entity_id",
- ServiceConfig.source_id_facet: "sources.id",
+ source_id_facet: "sources.id",
"sourceSpec": "sources.spec",
"aggregateSubmissionDate": "contents.aggregate_dates.submission_date",
"aggregateUpdateDate": "contents.aggregate_dates.update_date",
"aggregateLastModifiedDate": "contents.aggregate_dates.last_modified_date",
},
+ source_id_facet=source_id_facet,
autocomplete_translation={
"files": {
"entity_id": "entity_id"
@@ -401,7 +403,13 @@
"biologicalSex",
"genusSpecies",
"protocol"
- ]
+ ],
+ page_size_limits={
+ 'projects': 100,
+ 'samples': 1000,
+ 'files': 1000,
+ 'bundles': 100
+ }
)
def aggregate_class(self) -> Type[Aggregate]:
The pushing down of the source_id_facet should be done in a separate commit.
[edit]
Make sure that the limits are documented in the Swagger UI.
Extract the page size configuration limit into the plugin, similar to, if not piggy-backing onto, what we we did for https://github.com/DataBiosphere/azul/issues/4262.
[/edit]
For demo, attempt to reproduce. The expected response is 400, not 500. Request each /index/…
endpoint in anvilprod
and prod
with the maximum page size for that endpoint (should yield 200) and the maximum page size + 1 (should yield 400).
Request: https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=210 Result: {"message": "Internal server error"}
From the Data Browser team: "We read all the projects in order to 1) get the diff between catalogs and 2) build a sitemap that lists all projects. We are modifying the differ and sitemap code to use a low size (e.g. 50) and pagination but I wanted to bring this to the team's attention as the large size used to work until just recently".
[edit by @hannes-ucsc] Proposed solution here: https://github.com/DataBiosphere/azul/issues/3886#issuecomment-1067089108 [/edit]