search

DataBiosphere / azul

Metadata indexer and query service used for AnVIL, HCA, LungMAP, and CGP
Apache License 2.0
6 stars 2 forks source link

Request for 210 projects returns "Internal server error" #3886

Closed theathorn closed 5 months ago

theathorn commented 2 years ago

Request: https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=210 Result: {"message": "Internal server error"}

From the Data Browser team: "We read all the projects in order to 1) get the diff between catalogs and 2) build a sitemap that lists all projects. We are modifying the differ and sitemap code to use a low size (e.g. 50) and pagination but I wanted to bring this to the team's attention as the large size used to work until just recently".

[edit by @hannes-ucsc] Proposed solution here: https://github.com/DataBiosphere/azul/issues/3886#issuecomment-1067089108 [/edit]

achave11-ucsc commented 2 years ago

This is also the case for prod2. Start: 2022/02/18 14:08:37 End: 2022/02/18 14:14:03 /aws/lambda/azul-service-prod2

fields @timestamp, @message | filter @requestId like "b2cf13ec-c762-4f3d-abb4-dfd932af2ad9" | sort @timestamp desc | limit 2000

@timestamp @message
2022-02-18 22:11:02.400 END RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9
2022-02-18 22:11:02.400 REPORT RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Duration: 5584.74 ms Billed Duration: 5585 ms Memory Size: 2048 MB Max Memory Used: 229 MB
2022-02-18 22:11:02.400 RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Error: Runtime exited with error: exit status 1
Runtime.ExitError
2022-02-18 22:11:01.540 [DEBUG] 2022-02-18T22:11:01.540Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Returning 200 response with headers {"Access-Control-Allow-Origin": "*", "Access-Control-Allow-Headers": "Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key"}. See next line for the first 1024 characters of the body.
{"hits": [{"protocols": [{"lastModifiedDate": "2019-05-16T16:16:30.693000Z", "submissionDate": "2019-05-16T16:16:24.783000Z", "updateDate": "2019-05-16T16:16:30.693000Z", "libraryConstructionApproach": ["10X v2 sequencing"], "nucleicAcidSource": ["single cell"]}, {"lastModifiedDate": "2019-05-16T16:16:31.087000Z", "submissionDate": "2019-05-16T16:16:24.801000Z", "updateDate": "2019-05-16T16:16:31.087000Z", "instrumentManufacturerModel": ["Illumina HiSeq 4000"], "pairedEnd": [false]}], "entryId": "74b6d569-3b11-42ef-b6b1-a0454522b4a0", "sources": [{"sourceId": "43f89fcc-8cd3-40d2-a2a3-aa43b2ce86c5", "sourceSpec": "tdr:datarepo-699bbe9b:snapshot/hca_prod_74b6d5693b1142efb6b1a0454522b4a0__20220117_dcp2_20220124_dcp12:/1"}], "projects": [{"lastModifiedDate": "2021-10-20T09:03:20.617000Z", "submissionDate": "2019-05-16T16:09:33.130000Z", "updateDate": "2021-10-20T09:03:20.617000Z", "projectId": "74b6d569-3b11-42ef-b6b1-a0454522b4a0", "projectTitle": "1.3 Million Brain Cells from E18 Mice", "projectShortname": "1M
2022-02-18 22:10:57.126 [INFO] 2022-02-18T22:10:57.125Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 POST https://search-azul-index-prod2-f7axew6smhdyuga32dcjapmlla.us-east-1.es.amazonaws.com:443/azul_v2_prod2_dcp13_projects_aggregate/_search [status:200 request:0.274s]
2022-02-18 22:10:56.816 [INFO] 2022-02-18T22:10:56.816Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Received GET request for '/index/projects', with query {"catalog": "dcp13", "size": "195"} and headers {"accept": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,*/*;q=0.8", "accept-encoding": "gzip, deflate, br", "accept-language": "en-US,en;q=0.5", "cloudfront-forwarded-proto": "https", "cloudfront-is-desktop-viewer": "true", "cloudfront-is-mobile-viewer": "false", "cloudfront-is-smarttv-viewer": "false", "cloudfront-is-tablet-viewer": "false", "cloudfront-viewer-country": "US", "host": "service.azul2.data.humancellatlas.org", "sec-fetch-dest": "document", "sec-fetch-mode": "navigate", "sec-fetch-site": "none", "sec-fetch-user": "?1", "upgrade-insecure-requests": "1", "user-agent": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:97.0) Gecko/20100101 Firefox/97.0", "via": "2.0 d6db3d1c73f3e08aa70e81d7382400fe.cloudfront.net (CloudFront)", "x-amz-cf-id": "Dkwt4Om6FzufduXQZ5sHe3GQef-YsmGDC78jtvuLzNrSPXCkab4WBA==", "x-amzn-trace-id": "Root=1-62101970-342c5da056e88b3f001c6cc7", "x-forwarded-for": "76.89.218.122, 130.176.174.106", "x-forwarded-port": "443", "x-forwarded-proto": "https"}.
2022-02-18 22:10:56.816 [INFO] 2022-02-18T22:10:56.816Z b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Did not authenticate request.
2022-02-18 22:10:56.812 START RequestId: b2cf13ec-c762-4f3d-abb4-dfd932af2ad9 Version: $LATEST
melainalegaspi commented 2 years ago

@hannes-ucsc: "Spike to find out if other endpoints also break below a page size of 1000 (the current maximum). As part of the spike determine the size of each project in an unfiltered projects endpoint response for both catalogues, dcp12 and dcp13. With that information we will make the page size limit specific to each endpoint."

amarjandu commented 2 years ago

With the size set to 1000 dcp13 catalog has 502 errors on projects and bundles endpoint dcp12 catalog has a 504 timeout error on bundles endpoint

2022-02-24 17:33:36,707 INFO    MainThread: 502 https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=1000
2022-02-24 17:33:48,236 INFO    MainThread: 200 https://service.azul.data.humancellatlas.org/index/files?catalog=dcp13&size=1000
2022-02-24 17:33:55,543 INFO    MainThread: 200 https://service.azul.data.humancellatlas.org/index/samples?catalog=dcp13&size=1000
2022-02-24 17:34:13,048 INFO    MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=1000
2022-02-24 17:34:21,543 INFO    MainThread: 200 https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp12&size=1000
2022-02-24 17:34:30,607 INFO    MainThread: 200 https://service.azul.data.humancellatlas.org/index/files?catalog=dcp12&size=1000
2022-02-24 17:34:35,488 INFO    MainThread: 200 https://service.azul.data.humancellatlas.org/index/samples?catalog=dcp12&size=1000
2022-02-24 17:35:04,919 INFO    MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=1000

For the dcp13 catalog the largest size for the unfiltered projects endpoint is 190, using 191 causes errors

(.venv) CBSEspecter:azul amar$ http  --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=191'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:15:52 GMT
Via: 1.1 c7dc1e4970209c6ac0b27019856131a4.cloudfront.net (CloudFront)
X-Amz-Cf-Id: IrVzqYGVKN68vZPXVmYvnm5kf4wMITFEvoz6YZMmHwONxHQVJ8ZMaQ==
X-Amz-Cf-Pop: PHX50-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OEwWnE6VoAMFavg=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 2b46b48e-2479-4b4e-a803-d9ac3beeaa63

(.venv) CBSEspecter:azul amar$ http  --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp13&size=190'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 741584
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:16:11 GMT
Via: 1.1 a68097bdd514e6082da61e488096dbfa.cloudfront.net (CloudFront)
X-Amz-Cf-Id: MDYk3_oxGAiEch3JDAOpO_evtJjqkxbcQW_jwnkh-wWzHx4VQJs04Q==
X-Amz-Cf-Pop: PHX50-C1
X-Amzn-Trace-Id: Root=1-62182dd5-73ac01006d1c32c72ea2c7c9;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OEwZUGHVoAMFrtw=
x-amzn-RequestId: 6990e034-0f78-42ed-8db7-f1dbec00dc21

(.venv) CBSEspecter:azul amar$

Catalog dcp13 bundles endpoint can take a max size of 461, using 462 causes errors.

(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=462'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:42:02 GMT
Via: 1.1 4d573cd6a6c0581b92beae04909c3dce.cloudfront.net (CloudFront)
X-Amz-Cf-Id: h0nKsaK-yDTU79zPO-HiMW6fNbaaTH08un6kRw3cUgYt0CnoSloXcQ==
X-Amz-Cf-Pop: SFO5-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OE0K8H1boAMFgHQ=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 2ec1659f-6387-4b40-82a1-796ec7443dcb

(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp13&size=461'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 716000
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:42:15 GMT
Via: 1.1 28663e5849ed20a9d037ca8066957990.cloudfront.net (CloudFront)
X-Amz-Cf-Id: VrpEVOYvXx4XQ6pygF4oP9f3_YLyFjNmNRcM6_OSI892DjP6aWRJ-g==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-621833ee-2a51b8a074c1e3752121772d;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OE0NOGVNoAMFRVA=
x-amzn-RequestId: 2231ed6b-3efb-4a29-bad6-18031013c255

For the dcp12 catalog the unfiltered projects endpoint can return with a size of 1000 without an error

(.venv) CBSEspecter:azul amar$ http  --headers 'https://service.azul.data.humancellatlas.org/index/projects?catalog=dcp12&size=1000'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 717039
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:18:27 GMT
Via: 1.1 c361075814ee65e6523d12c5af3e7994.cloudfront.net (CloudFront)
X-Amz-Cf-Id: SemmnnfcqFY-lj4IiQpURQMYnQO5dXHAKfdoQn-7VOmkaX1Gya82kg==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-62182e5d-3f5bcf4b6fa69e4c07db031f;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OEwutEjtIAMFYUg=
x-amzn-RequestId: 297a56be-94ab-48c4-95d3-ceb8650f4261

Catalog dcp12 bundles 502 endpoint errors start occurring with a size of 966

(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=966'
HTTP/1.1 502 Bad Gateway
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 56
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:47:46 GMT
Via: 1.1 89d40f9555bb19bc571952b32ca87398.cloudfront.net (CloudFront)
X-Amz-Cf-Id: yO9yjR-MVxyamc8y8RY6aoAaq8415QRPZuh0guVIC_MLu3sHDl29pA==
X-Amz-Cf-Pop: SFO5-C1
X-Cache: Error from cloudfront
x-amz-apigw-id: OE1AoH3hoAMF_bA=
x-amzn-ErrorType: InternalServerErrorException
x-amzn-RequestId: 9f0e4d27-7d14-416f-ab35-442bde49ae50

(.venv) CBSEspecter:azul amar$ http --headers 'https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=965'
HTTP/1.1 200 OK
Access-Control-Allow-Headers: Authorization,Content-Type,X-Amz-Date,X-Amz-Security-Token,X-Api-Key
Access-Control-Allow-Origin: *
Connection: keep-alive
Content-Encoding: gzip
Content-Length: 556724
Content-Type: application/json
Date: Fri, 25 Feb 2022 01:48:11 GMT
Via: 1.1 b5546ff55405d525045e7263ba6db012.cloudfront.net (CloudFront)
X-Amz-Cf-Id: DSQOfpTVQzAMEuY8jTTlbq39O3CIJSCywqHpn9-3gwVo35IwL42MgA==
X-Amz-Cf-Pop: SFO5-C1
X-Amzn-Trace-Id: Root=1-62183552-1f9f5bff04106f416bd99141;Sampled=0
X-Cache: Miss from cloudfront
x-amz-apigw-id: OE1E3EiyIAMFs8Q=
x-amzn-RequestId: 6c127f4e-a63f-4d34-969f-9cd8135e907b

504 Timeout errors start to occur with the size of 994, but it is inconsistant

2022-02-24 17:59:54,498 INFO    MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=993
2022-02-24 18:00:24,163 INFO    MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=994
2022-02-24 18:00:52,639 INFO    MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=995
2022-02-24 18:01:21,714 INFO    MainThread: 502 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=996
2022-02-24 18:01:51,004 INFO    MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=997
2022-02-24 18:02:20,159 INFO    MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=998
2022-02-24 18:02:49,342 INFO    MainThread: 504 https://service.azul.data.humancellatlas.org/index/bundles?catalog=dcp12&size=999
melainalegaspi commented 2 years ago

@hannes-ucsc to evaluate findings and come up with next steps.

hannes-ucsc commented 2 years ago

It appears you missed part of the spike @amarjandu

As part of the spike determine the size of each project in an unfiltered projects endpoint response for both catalogues, dcp12 and dcp13.

amarjandu commented 2 years ago

{projectId: bytes} for dcp12 and dcp13 from the the /index/projects endpoint

{'dcp12': {'005d611a-14d5-4fbf-846e-571a1f874f70': 16833,
           '027c51c6-0719-469f-a7f5-640fe57cbece': 11532,
           '03c6fce7-789e-4e78-a27a-664d562bb738': 15989,
           '04ad400c-58cb-40a5-bc2b-2279e13a910b': 174737,
           '05657a59-9f9d-4bb9-b77b-24be13aa5cea': 11461,
           '05be4f37-4506-429b-b112-506444507d62': 7558,
           '07073c12-8006-4710-a00b-23abdb814904': 18757,
           '074a9f88-729a-455d-bca5-0ce80edf0cea': 11530,
           '0792db34-8047-4e62-802c-9177c9cd8e28': 24401,
           '08b794a0-519c-4516-b184-c583746254c5': 9950,
           '091cf39b-01bc-42e5-9437-f419a66c8a45': 14467,
           '0c09fade-e079-4fde-8e60-6725b8c1d84b': 13559,
           '0c3b7785-f74d-4091-8616-a68757e4c2a8': 12857,
           '0d4b87ea-6e9e-4569-82e4-1343e0e3259f': 13145,
           '0fd8f918-62d6-4b8b-ac35-4c53dd601f71': 10077,
           '116965f3-f094-4769-9d28-ae675c1b569c': 10836,
           '16ed4ad8-7319-46b2-8859-6fe1c1d73a82': 32319,
           '1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': 12947,
           '1cd1f41f-f81a-486b-a05b-66ec60f81dcf': 10878,
           '1ce3b3dc-02f2-44a8-96da-d6d107b27a76': 7980,
           '1defdada-a365-44ad-9b29-443b06bd11d6': 11990,
           '2043c65a-1cf8-4828-a656-9e247d4e64f1': 17666,
           '2084526b-a66f-4c40-bb89-6fd162f2eb38': 30836,
           '2086eb05-10b9-432b-b7f0-169ccc49d270': 11795,
           '20f37aaf-caa1-40e6-9123-be6ce8feb2d6': 16495,
           '21ea8ddb-525f-4f1f-a820-31f0360399a2': 9325,
           '23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': 10091,
           '248fcf03-16c6-4a41-b6cc-aad4d894ca42': 11551,
           '24c654a5-caa5-440a-8f02-582921f2db4a': 13382,
           '2a64db43-1b55-4639-aabb-8dba0145689d': 9149,
           '2a72a4e5-66b2-405a-bb7c-1e463e8febb0': 18103,
           '2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': 14635,
           '2af52a13-65cb-4973-b513-39be38f2df3f': 6657,
           '2b38025d-a5ea-4c0f-b22e-367824bcaf4c': 61708,
           '2d846095-8a33-4f3c-97d4-585bafac13b4': 12269,
           '2ef3655a-973d-4d69-9b41-21fa4041eed7': 11173,
           '2f676143-80c2-4bc6-b7b4-2613fe0fadf0': 13729,
           '3089d311-f9ed-44dd-bb10-397059bad4dc': 20946,
           '31887183-a72c-4308-9eac-c6140313f39c': 78347,
           '34cba5e9-ecb1-4d81-bf08-48987cd63073': 9795,
           '376a7f55-b876-4f60-9cf3-ed7bc83d5415': 12421,
           '379ed69e-be05-48bc-af5e-a7fc589709bf': 15653,
           '38449aea-70b5-40db-84b3-1e08f32efe34': 15205,
           '38e44dd0-c3df-418e-9256-d0824748901f': 17806,
           '3a694703-3084-4ece-9abe-d935fd5f6748': 15402,
           '3c27d2dd-b180-4b2b-bf05-e2e418393fd1': 12084,
           '3cfcdff5-dee1-4a7b-a591-c09c6e850b11': 14874,
           '3e329187-a9c4-48ec-90e3-cc45f7c2311c': 8140,
           '4037007b-0eff-4e6d-b7bd-8dd8eec80143': 14503,
           '403c3e76-6814-4a2d-a580-5dd5de38c7ff': 46734,
           '414acced-eba0-440f-b721-befbc5642bef': 11901,
           '41fb1734-a121-4616-95c7-3b732c9433c7': 10866,
           '42d4f8d4-5422-4b78-adae-e7c3c2ef511c': 15585,
           '455b46e6-d8ea-4611-861e-de720a562ada': 20894,
           '4a95101c-9ffc-4f30-a809-f04518a23803': 25576,
           '4af795f7-3e1d-4341-b867-4ac0982b9efd': 11229,
           '4bec484d-ca7a-47b4-8d48-8830e06ad6db': 11277,
           '4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': 18090,
           '4e6f083b-5b9a-4393-9890-2a83da8188f1': 18908,
           '50151324-f3ed-4358-98af-ec352a940a61': 135225,
           '504e0cee-1688-40fa-b936-361c4a831f87': 8135,
           '5116c081-8be7-49c5-8ce0-73b887328aa9': 22242,
           '51f02950-ee25-4f4b-8d07-59aa99bb3498': 17446,
           '520afa10-f9d2-4e93-ab7a-26c4c863ce18': 7484,
           '52b29aa4-c8d6-42b4-807a-b35be94469ca': 9791,
           '52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': 8173,
           '53c53cd4-8127-4e12-bc7f-8fe1610a715c': 22727,
           '54aaa409-dc28-48c5-be26-d368b4a5d5c6': 20920,
           '559bb888-7829-41f2-ace5-2c05c7eb81e9': 21304,
           '56e73ccb-7ae9-4fae-a738-acfb69936d7a': 13966,
           '577c946d-6de5-4b55-a854-cd3fde40bff2': 10747,
           '58028aa8-0ed2-49ca-b60f-15e2ed5989d5': 16299,
           '591af954-cdcd-4839-96d3-a0d1b1e885ac': 33292,
           '5b328561-4a97-40ac-b7ad-6a90fc59d374': 134408,
           '5b5f05b7-2482-468d-b76d-8f68c04a7a47': 11148,
           '5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': 7105,
           '5eafb94b-02d8-423e-81b8-3673da319ca0': 9157,
           '5ee710d7-e2d5-4fe2-818d-15f5e31dae32': 9328,
           '602628d7-c038-48a8-aa97-ffbb2cb44c9d': 17394,
           '6072616c-8794-4b20-8f52-fb15992ea5a4': 8531,
           '60ea42e1-af49-42f5-8164-d641fdb696bc': 7018,
           '63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': 8009,
           '65858543-530d-48a6-a670-f972b34dfe10': 21843,
           '67a3de09-45b9-49c3-a068-ff4665daa50e': 12239,
           '68df3629-d2d2-4eed-b0ab-a10e0f019b88': 8489,
           '6c040a93-8cf8-4fd5-98de-2297eb07e9f6': 17331,
           '7027adc6-c9c9-46f3-84ee-9badc3a4f53b': 18126,
           '71436067-ac41-4ace-be1b-2fbcc2cb02fa': 7081,
           '71eb5f6d-cee0-4297-b503-b1125909b8c7': 41059,
           '74493e98-44fc-48b0-a58f-cc7e77268b59': 27268,
           '74b6d569-3b11-42ef-b6b1-a0454522b4a0': 7784,
           '75dbbce9-0cde-489c-88a7-93e8f92914a3': 9835,
           '769a08d1-b8a4-4f1e-95f7-6071a9827555': 21688,
           '783c9952-a4ae-4106-a6ce-56f20ce27f88': 11876,
           '7880637a-35a1-4047-b422-b5eac2a2a358': 27821,
           '78b2406d-bff2-46fc-8b61-20690e602227': 150223,
           '79b13a2a-9ca1-42a4-97bd-70208a11bea6': 6192,
           '7ac8822c-4ef0-4194-adf0-74290611b1c6': 23137,
           '7adede6a-0ab7-45e6-9b67-ffe7466bec1f': 15108,
           '7b947aa2-43a7-4082-afff-222a3e3a4635': 10092,
           '7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': 29179,
           '8185730f-4113-40d3-9cc3-929271784c2b': 25695,
           '83f5188e-3bf7-4956-9544-cea4f8997756': 33488,
           '842605c7-375a-47c5-9e2c-a71c2c00fcad': 11175,
           '8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': 150775,
           '8787c238-89ef-4636-a57d-3167e8b54a80': 8683,
           '87d52a86-bdc7-440c-b84d-170f7dc346d9': 11126,
           '88ec040b-8705-4f77-8f41-f81e57632f7d': 170912,
           '8999b456-6fa6-438b-ab17-b62b1d8ec0c3': 10315,
           '8a40ff19-e614-4c50-b23b-5c9e1d546bab': 110744,
           '8ab8726d-81b9-4bd2-acc2-4d50bee786b4': 13879,
           '8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': 29628,
           '8c3c290d-dfff-4553-8868-54ce45f4ba7f': 27803,
           '8d566d35-d8d3-4975-a351-be5e25e9b2ea': 69372,
           '8dacb243-e918-4bd2-bb9a-aac6dc424161': 15434,
           '90bd6933-40c0-48d4-8d76-778c103bf545': 14363,
           '94023a08-611d-4f22-a8c9-90956e091b2e': 8532,
           '946c5add-47d1-402a-97bb-a5af97e8bce7': 9506,
           '955dfc2c-a8c6-4d04-aa4d-907610545d11': 10064,
           '95f07e6e-6a73-4e1b-a880-c83996b3aa5c': 15066,
           '962bd805-eb89-4c54-bad2-008e497d1307': 20180,
           '99101928-d9b1-4aaf-b759-e97958ac7403': 9587,
           '996120f9-e84f-409f-a01e-732ab58ca8b9': 12720,
           '9d97f01f-9313-416e-9b07-560f048b2350': 63322,
           'a004b150-1c36-4af6-9bbd-070c06dbc17d': 10666,
           'a29952d9-925e-40f4-8a1c-274f118f1f51': 9935,
           'a39728aa-70a0-4201-b0a2-81b7badf3e71': 22783,
           'a60803bb-f7db-45cf-b529-95436152a801': 47589,
           'a80a63f2-e223-4890-81b0-415855b89abc': 14909,
           'a9301beb-e9fa-42fe-b75c-84e8a460c733': 601838,
           'a96b71c0-78a7-42d1-88ce-83c78925cfeb': 8101,
           'a991ef15-4d4a-4b80-a93e-c538b4b54127': 10359,
           'a9c022b4-c771-4468-b769-cabcf9738de3': 17361,
           'abe1a013-af7a-45ed-8c26-f3793c24a1f4': 17286,
           'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': 63898,
           'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': 23889,
           'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': 14456,
           'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': 8821,
           'b4a7d12f-6c2f-40a3-9e35-9756997857e3': 11626,
           'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': 20938,
           'b7259878-436c-4274-bfff-ca76f4cb7892': 14386,
           'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': 8996,
           'b963bd4b-4bc1-4404-8425-69d74bc636b8': 29180,
           'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': 35193,
           'bd7104c9-a950-490e-9472-7d41c6b11c62': 26788,
           'c1810dbc-16d2-45c3-b45e-3e675f88d87b': 32597,
           'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': 28214,
           'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': 146188,
           'c4077b3c-5c98-4d26-a614-246d12c2e5d7': 29707,
           'c41dffbf-ad83-447c-a0e1-13e689d9b258': 22433,
           'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': 49294,
           'c5f46615-68de-4cf4-bbc2-a0ae10f08243': 19482,
           'c6ad8f9b-d26a-4811-b2ba-93d487978446': 9965,
           'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': 31423,
           'c893cb57-5c9f-4f26-9312-21b85be84313': 14034,
           'cc95ff89-2e68-4a08-a234-480eca21ce79': 15763,
           'ccd1f1ba-74ce-469b-9fc9-f6faea623358': 7146,
           'ccef38d7-aa92-4010-9621-c4c7b1182647': 17268,
           'cddab57b-6868-4be4-806f-395ed9dd635a': 17802,
           'ce33dde2-382d-448c-b6ac-bfb424644f23': 10592,
           'ce7b12ba-664f-4f79-8fc7-3de6b1892183': 17589,
           'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': 15343,
           'd2111fac-3fc4-4f42-9b6d-32cd6a828267': 8897,
           'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': 49703,
           'd3a4ceac-4d66-4984-9704-2570c0647a56': 12795,
           'd3ac7c1b-5302-4804-b611-dad9f89c049d': 29543,
           'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': 57138,
           'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': 21298,
           'da2747fa-2921-42e0-afd4-39ef57b2b88b': 23818,
           'daf9d982-7ce6-43f6-ab51-272577290606': 81839,
           'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': 7508,
           'dc1a41f6-9e09-42a6-959e-3be23db6da56': 77281,
           'dd7f2436-0c56-4709-bd17-e526bba4cc15': 18355,
           'df88f39f-01a8-4b5b-92f4-3177d6c0f242': 12926,
           'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': 143939,
           'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': 8297,
           'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': 6695,
           'e57dc176-ab98-446b-90c2-89e0842152fd': 14580,
           'e5d45579-1f5b-48c3-b568-320d93e7ca72': 17096,
           'e77fed30-959d-4fad-bc15-a0a5a85c21d2': 6588,
           'e8808cc8-4ca0-4096-80f2-bba73600cba6': 14324,
           'eaefa1b6-dae1-4414-953b-17b0427d061e': 13700,
           'ede2e0b4-6652-464f-abbc-0b2d964a25a0': 20005,
           'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': 164365,
           'ef1e3497-515e-4bbe-8d4c-10161854b699': 40945,
           'efea6426-510a-4b60-9a19-277e52bfa815': 15699,
           'f0f89c14-7460-4bab-9d42-22228a91f185': 21954,
           'f2fe82f0-4454-4d84-b416-a885f3121e59': 84683,
           'f48e7c39-cc67-4055-9d79-bc437892840c': 10165,
           'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': 5724,
           'f81efc03-9f56-4354-aabb-6ce819c3d414': 8088,
           'f83165c5-e2ea-4d15-a5cf-33f3550bffde': 36002,
           'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': 13457,
           'f8aa201c-4ff1-45a4-890e-840d63459ca2': 42435,
           'faeedcb0-e046-4be7-b1ad-80a3eeabb066': 52939},
 'dcp13': {'005d611a-14d5-4fbf-846e-571a1f874f70': 16833,
           '027c51c6-0719-469f-a7f5-640fe57cbece': 11532,
           '03c6fce7-789e-4e78-a27a-664d562bb738': 17807,
           '04ad400c-58cb-40a5-bc2b-2279e13a910b': 174737,
           '05657a59-9f9d-4bb9-b77b-24be13aa5cea': 13261,
           '05be4f37-4506-429b-b112-506444507d62': 9362,
           '065e6c13-ad6b-46a3-8075-c3137eb03068': 137608,
           '06c7dd8d-6cc6-4b79-b795-8805c47d36e1': 34866,
           '07073c12-8006-4710-a00b-23abdb814904': 18757,
           '074a9f88-729a-455d-bca5-0ce80edf0cea': 11530,
           '0792db34-8047-4e62-802c-9177c9cd8e28': 24401,
           '08b794a0-519c-4516-b184-c583746254c5': 9950,
           '091cf39b-01bc-42e5-9437-f419a66c8a45': 14467,
           '0c09fade-e079-4fde-8e60-6725b8c1d84b': 13559,
           '0c3b7785-f74d-4091-8616-a68757e4c2a8': 12857,
           '0d4b87ea-6e9e-4569-82e4-1343e0e3259f': 13145,
           '0fd8f918-62d6-4b8b-ac35-4c53dd601f71': 10077,
           '10201832-7c73-4033-9b65-3ef13d81656a': 142535,
           '116965f3-f094-4769-9d28-ae675c1b569c': 10836,
           '16ed4ad8-7319-46b2-8859-6fe1c1d73a82': 32319,
           '1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': 12947,
           '1cd1f41f-f81a-486b-a05b-66ec60f81dcf': 10878,
           '1ce3b3dc-02f2-44a8-96da-d6d107b27a76': 9788,
           '1dddae6e-3753-48af-b20e-fa22abad125d': 79837,
           '1defdada-a365-44ad-9b29-443b06bd11d6': 11990,
           '1eb69a39-b5b2-41ec-afae-5fe37f272756': 11780,
           '2043c65a-1cf8-4828-a656-9e247d4e64f1': 17666,
           '2084526b-a66f-4c40-bb89-6fd162f2eb38': 30836,
           '2086eb05-10b9-432b-b7f0-169ccc49d270': 11795,
           '20f37aaf-caa1-40e6-9123-be6ce8feb2d6': 16495,
           '21ea8ddb-525f-4f1f-a820-31f0360399a2': 9325,
           '23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': 13399,
           '248fcf03-16c6-4a41-b6cc-aad4d894ca42': 11551,
           '24c654a5-caa5-440a-8f02-582921f2db4a': 13382,
           '2a64db43-1b55-4639-aabb-8dba0145689d': 9149,
           '2a72a4e5-66b2-405a-bb7c-1e463e8febb0': 19931,
           '2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': 14635,
           '2af52a13-65cb-4973-b513-39be38f2df3f': 6657,
           '2b38025d-a5ea-4c0f-b22e-367824bcaf4c': 61708,
           '2d846095-8a33-4f3c-97d4-585bafac13b4': 14101,
           '2ef3655a-973d-4d69-9b41-21fa4041eed7': 11173,
           '2f676143-80c2-4bc6-b7b4-2613fe0fadf0': 13729,
           '3089d311-f9ed-44dd-bb10-397059bad4dc': 20946,
           '31887183-a72c-4308-9eac-c6140313f39c': 78347,
           '34cba5e9-ecb1-4d81-bf08-48987cd63073': 9795,
           '376a7f55-b876-4f60-9cf3-ed7bc83d5415': 12421,
           '379ed69e-be05-48bc-af5e-a7fc589709bf': 15653,
           '38449aea-70b5-40db-84b3-1e08f32efe34': 15205,
           '38e44dd0-c3df-418e-9256-d0824748901f': 17806,
           '3a694703-3084-4ece-9abe-d935fd5f6748': 17815,
           '3c27d2dd-b180-4b2b-bf05-e2e418393fd1': 12084,
           '3cfcdff5-dee1-4a7b-a591-c09c6e850b11': 14874,
           '3e329187-a9c4-48ec-90e3-cc45f7c2311c': 8140,
           '4037007b-0eff-4e6d-b7bd-8dd8eec80143': 14503,
           '403c3e76-6814-4a2d-a580-5dd5de38c7ff': 46734,
           '414acced-eba0-440f-b721-befbc5642bef': 11901,
           '41fb1734-a121-4616-95c7-3b732c9433c7': 10866,
           '42d4f8d4-5422-4b78-adae-e7c3c2ef511c': 15585,
           '455b46e6-d8ea-4611-861e-de720a562ada': 20894,
           '4a95101c-9ffc-4f30-a809-f04518a23803': 25576,
           '4af795f7-3e1d-4341-b867-4ac0982b9efd': 11229,
           '4bec484d-ca7a-47b4-8d48-8830e06ad6db': 11277,
           '4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': 18090,
           '4e6f083b-5b9a-4393-9890-2a83da8188f1': 18908,
           '50151324-f3ed-4358-98af-ec352a940a61': 135225,
           '504e0cee-1688-40fa-b936-361c4a831f87': 8135,
           '5116c081-8be7-49c5-8ce0-73b887328aa9': 22242,
           '51f02950-ee25-4f4b-8d07-59aa99bb3498': 17446,
           '520afa10-f9d2-4e93-ab7a-26c4c863ce18': 9294,
           '52b29aa4-c8d6-42b4-807a-b35be94469ca': 9791,
           '52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': 8173,
           '53c53cd4-8127-4e12-bc7f-8fe1610a715c': 22727,
           '54aaa409-dc28-48c5-be26-d368b4a5d5c6': 20920,
           '559bb888-7829-41f2-ace5-2c05c7eb81e9': 21304,
           '56e73ccb-7ae9-4fae-a738-acfb69936d7a': 13966,
           '577c946d-6de5-4b55-a854-cd3fde40bff2': 10747,
           '58028aa8-0ed2-49ca-b60f-15e2ed5989d5': 16485,
           '591af954-cdcd-4839-96d3-a0d1b1e885ac': 33292,
           '5b328561-4a97-40ac-b7ad-6a90fc59d374': 134408,
           '5b5f05b7-2482-468d-b76d-8f68c04a7a47': 11148,
           '5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': 7105,
           '5eafb94b-02d8-423e-81b8-3673da319ca0': 9157,
           '5ee710d7-e2d5-4fe2-818d-15f5e31dae32': 9328,
           '602628d7-c038-48a8-aa97-ffbb2cb44c9d': 17394,
           '6072616c-8794-4b20-8f52-fb15992ea5a4': 8531,
           '60ea42e1-af49-42f5-8164-d641fdb696bc': 7018,
           '63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': 8009,
           '65858543-530d-48a6-a670-f972b34dfe10': 21843,
           '67a3de09-45b9-49c3-a068-ff4665daa50e': 14771,
           '68df3629-d2d2-4eed-b0ab-a10e0f019b88': 8489,
           '6c040a93-8cf8-4fd5-98de-2297eb07e9f6': 17331,
           '6f89a7f3-8d4a-4344-aa4f-eccfe7e91076': 13836,
           '7027adc6-c9c9-46f3-84ee-9badc3a4f53b': 18126,
           '71436067-ac41-4ace-be1b-2fbcc2cb02fa': 7081,
           '71eb5f6d-cee0-4297-b503-b1125909b8c7': 41059,
           '74493e98-44fc-48b0-a58f-cc7e77268b59': 27268,
           '74b6d569-3b11-42ef-b6b1-a0454522b4a0': 7784,
           '75dbbce9-0cde-489c-88a7-93e8f92914a3': 9835,
           '769a08d1-b8a4-4f1e-95f7-6071a9827555': 21688,
           '783c9952-a4ae-4106-a6ce-56f20ce27f88': 11876,
           '7880637a-35a1-4047-b422-b5eac2a2a358': 27821,
           '78b2406d-bff2-46fc-8b61-20690e602227': 151894,
           '78d7805b-fdc8-472b-8058-d92cf886f7a4': 48912,
           '79b13a2a-9ca1-42a4-97bd-70208a11bea6': 6192,
           '7ac8822c-4ef0-4194-adf0-74290611b1c6': 23137,
           '7adede6a-0ab7-45e6-9b67-ffe7466bec1f': 15108,
           '7b947aa2-43a7-4082-afff-222a3e3a4635': 10092,
           '7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': 29179,
           '8185730f-4113-40d3-9cc3-929271784c2b': 25695,
           '83f5188e-3bf7-4956-9544-cea4f8997756': 33488,
           '842605c7-375a-47c5-9e2c-a71c2c00fcad': 11175,
           '8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': 152438,
           '85a9263b-0887-48ed-ab1a-ddfa773727b6': 25211,
           '8787c238-89ef-4636-a57d-3167e8b54a80': 8683,
           '87d52a86-bdc7-440c-b84d-170f7dc346d9': 11126,
           '88ec040b-8705-4f77-8f41-f81e57632f7d': 172954,
           '8999b456-6fa6-438b-ab17-b62b1d8ec0c3': 10315,
           '8a40ff19-e614-4c50-b23b-5c9e1d546bab': 110744,
           '8ab8726d-81b9-4bd2-acc2-4d50bee786b4': 13879,
           '8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': 29628,
           '8c3c290d-dfff-4553-8868-54ce45f4ba7f': 31036,
           '8d566d35-d8d3-4975-a351-be5e25e9b2ea': 69372,
           '8dacb243-e918-4bd2-bb9a-aac6dc424161': 15434,
           '90bd6933-40c0-48d4-8d76-778c103bf545': 14363,
           '94023a08-611d-4f22-a8c9-90956e091b2e': 8532,
           '946c5add-47d1-402a-97bb-a5af97e8bce7': 9506,
           '955dfc2c-a8c6-4d04-aa4d-907610545d11': 10064,
           '95f07e6e-6a73-4e1b-a880-c83996b3aa5c': 15066,
           '962bd805-eb89-4c54-bad2-008e497d1307': 20180,
           '99101928-d9b1-4aaf-b759-e97958ac7403': 12155,
           '996120f9-e84f-409f-a01e-732ab58ca8b9': 12720,
           '9c20a245-f2c0-43ae-82c9-2232ec6b594f': 12240,
           'a004b150-1c36-4af6-9bbd-070c06dbc17d': 10666,
           'a29952d9-925e-40f4-8a1c-274f118f1f51': 9935,
           'a39728aa-70a0-4201-b0a2-81b7badf3e71': 22783,
           'a60803bb-f7db-45cf-b529-95436152a801': 47589,
           'a80a63f2-e223-4890-81b0-415855b89abc': 14909,
           'a9301beb-e9fa-42fe-b75c-84e8a460c733': 601838,
           'a96b71c0-78a7-42d1-88ce-83c78925cfeb': 8101,
           'a991ef15-4d4a-4b80-a93e-c538b4b54127': 10359,
           'a9c022b4-c771-4468-b769-cabcf9738de3': 17361,
           'abe1a013-af7a-45ed-8c26-f3793c24a1f4': 17286,
           'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': 63898,
           'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': 23889,
           'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': 14456,
           'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': 8821,
           'b4a7d12f-6c2f-40a3-9e35-9756997857e3': 11626,
           'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': 20938,
           'b7259878-436c-4274-bfff-ca76f4cb7892': 14386,
           'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': 8996,
           'b963bd4b-4bc1-4404-8425-69d74bc636b8': 29180,
           'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': 35193,
           'bd7104c9-a950-490e-9472-7d41c6b11c62': 26788,
           'c1810dbc-16d2-45c3-b45e-3e675f88d87b': 32597,
           'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': 28214,
           'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': 146188,
           'c4077b3c-5c98-4d26-a614-246d12c2e5d7': 29707,
           'c41dffbf-ad83-447c-a0e1-13e689d9b258': 22433,
           'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': 49294,
           'c5f46615-68de-4cf4-bbc2-a0ae10f08243': 19482,
           'c6ad8f9b-d26a-4811-b2ba-93d487978446': 9965,
           'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': 31423,
           'c893cb57-5c9f-4f26-9312-21b85be84313': 14034,
           'cc95ff89-2e68-4a08-a234-480eca21ce79': 15763,
           'ccd1f1ba-74ce-469b-9fc9-f6faea623358': 8938,
           'ccef38d7-aa92-4010-9621-c4c7b1182647': 22121,
           'cd61771b-661a-4e19-b269-6e5d95350de6': 237934,
           'cddab57b-6868-4be4-806f-395ed9dd635a': 17802,
           'ce33dde2-382d-448c-b6ac-bfb424644f23': 10592,
           'ce7b12ba-664f-4f79-8fc7-3de6b1892183': 17589,
           'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': 15343,
           'd2111fac-3fc4-4f42-9b6d-32cd6a828267': 8897,
           'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': 49703,
           'd3a4ceac-4d66-4984-9704-2570c0647a56': 12795,
           'd3ac7c1b-5302-4804-b611-dad9f89c049d': 29543,
           'd6225aee-8f0e-4b20-a20c-682509a9ea14': 29005,
           'd71c76d3-3670-4774-a9cf-034249d37c60': 54815,
           'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': 57138,
           'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': 21298,
           'da2747fa-2921-42e0-afd4-39ef57b2b88b': 23818,
           'daf9d982-7ce6-43f6-ab51-272577290606': 81839,
           'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': 7508,
           'dbd836cf-bfc2-41f0-9834-41cc6c0b235a': 8890,
           'dc1a41f6-9e09-42a6-959e-3be23db6da56': 77281,
           'dd7ada84-3f14-4765-b7ce-9b64642bb3dc': 11503,
           'dd7f2436-0c56-4709-bd17-e526bba4cc15': 18355,
           'df88f39f-01a8-4b5b-92f4-3177d6c0f242': 12926,
           'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': 143939,
           'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': 8297,
           'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': 6695,
           'e57dc176-ab98-446b-90c2-89e0842152fd': 14580,
           'e5d45579-1f5b-48c3-b568-320d93e7ca72': 17096,
           'e77fed30-959d-4fad-bc15-a0a5a85c21d2': 6588,
           'e8808cc8-4ca0-4096-80f2-bba73600cba6': 17634,
           'eaefa1b6-dae1-4414-953b-17b0427d061e': 13700,
           'ede2e0b4-6652-464f-abbc-0b2d964a25a0': 20005,
           'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': 164365,
           'ef1e3497-515e-4bbe-8d4c-10161854b699': 40945,
           'efea6426-510a-4b60-9a19-277e52bfa815': 15699,
           'f0f89c14-7460-4bab-9d42-22228a91f185': 21954,
           'f2fe82f0-4454-4d84-b416-a885f3121e59': 84683,
           'f48e7c39-cc67-4055-9d79-bc437892840c': 10165,
           'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': 7527,
           'f81efc03-9f56-4354-aabb-6ce819c3d414': 8088,
           'f83165c5-e2ea-4d15-a5cf-33f3550bffde': 36002,
           'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': 13457,
           'f8aa201c-4ff1-45a4-890e-840d63459ca2': 42435,
           'faeedcb0-e046-4be7-b1ad-80a3eeabb066': 52939}}
melainalegaspi commented 2 years ago

@hannes-ucsc : " @amarjandu to continue spike to convert the above data structure to a single dictionary with keys being the project IDs and the values being a list containing the two sizes for dcp12 and dcp13, respectively."

amarjandu commented 2 years ago

projectId: [bytes dcp12, bytes dcp13]

{'005d611a-14d5-4fbf-846e-571a1f874f70': [16833, 16833],
 '027c51c6-0719-469f-a7f5-640fe57cbece': [11532, 11532],
 '03c6fce7-789e-4e78-a27a-664d562bb738': [15989, 17807],
 '04ad400c-58cb-40a5-bc2b-2279e13a910b': [174737, 174737],
 '05657a59-9f9d-4bb9-b77b-24be13aa5cea': [11461, 13261],
 '05be4f37-4506-429b-b112-506444507d62': [7558, 9362],
 '065e6c13-ad6b-46a3-8075-c3137eb03068': [None, 137608],
 '06c7dd8d-6cc6-4b79-b795-8805c47d36e1': [None, 34866],
 '07073c12-8006-4710-a00b-23abdb814904': [18757, 18757],
 '074a9f88-729a-455d-bca5-0ce80edf0cea': [11530, 11530],
 '0792db34-8047-4e62-802c-9177c9cd8e28': [24401, 24401],
 '08b794a0-519c-4516-b184-c583746254c5': [9950, 9950],
 '091cf39b-01bc-42e5-9437-f419a66c8a45': [14467, 14467],
 '0c09fade-e079-4fde-8e60-6725b8c1d84b': [13559, 13559],
 '0c3b7785-f74d-4091-8616-a68757e4c2a8': [12857, 12857],
 '0d4b87ea-6e9e-4569-82e4-1343e0e3259f': [13145, 13145],
 '0fd8f918-62d6-4b8b-ac35-4c53dd601f71': [10077, 10077],
 '10201832-7c73-4033-9b65-3ef13d81656a': [None, 142535],
 '116965f3-f094-4769-9d28-ae675c1b569c': [10836, 10836],
 '16ed4ad8-7319-46b2-8859-6fe1c1d73a82': [32319, 32319],
 '1c6a960d-52ac-44ea-b728-a59c7ab9dc8e': [12947, 12947],
 '1cd1f41f-f81a-486b-a05b-66ec60f81dcf': [10878, 10878],
 '1ce3b3dc-02f2-44a8-96da-d6d107b27a76': [7980, 9788],
 '1dddae6e-3753-48af-b20e-fa22abad125d': [None, 79837],
 '1defdada-a365-44ad-9b29-443b06bd11d6': [11990, 11990],
 '1eb69a39-b5b2-41ec-afae-5fe37f272756': [None, 11780],
 '2043c65a-1cf8-4828-a656-9e247d4e64f1': [17666, 17666],
 '2084526b-a66f-4c40-bb89-6fd162f2eb38': [30836, 30836],
 '2086eb05-10b9-432b-b7f0-169ccc49d270': [11795, 11795],
 '20f37aaf-caa1-40e6-9123-be6ce8feb2d6': [16495, 16495],
 '21ea8ddb-525f-4f1f-a820-31f0360399a2': [9325, 9325],
 '23587fb3-1a4a-4f58-ad74-cc9a4cb4c254': [10091, 13399],
 '248fcf03-16c6-4a41-b6cc-aad4d894ca42': [11551, 11551],
 '24c654a5-caa5-440a-8f02-582921f2db4a': [13382, 13382],
 '2a64db43-1b55-4639-aabb-8dba0145689d': [9149, 9149],
 '2a72a4e5-66b2-405a-bb7c-1e463e8febb0': [18103, 19931],
 '2ad191cd-bd7a-409b-9bd1-e72b5e4cce81': [14635, 14635],
 '2af52a13-65cb-4973-b513-39be38f2df3f': [6657, 6657],
 '2b38025d-a5ea-4c0f-b22e-367824bcaf4c': [61708, 61708],
 '2d846095-8a33-4f3c-97d4-585bafac13b4': [12269, 14101],
 '2ef3655a-973d-4d69-9b41-21fa4041eed7': [11173, 11173],
 '2f676143-80c2-4bc6-b7b4-2613fe0fadf0': [13729, 13729],
 '3089d311-f9ed-44dd-bb10-397059bad4dc': [20946, 20946],
 '31887183-a72c-4308-9eac-c6140313f39c': [78347, 78347],
 '34cba5e9-ecb1-4d81-bf08-48987cd63073': [9795, 9795],
 '376a7f55-b876-4f60-9cf3-ed7bc83d5415': [12421, 12421],
 '379ed69e-be05-48bc-af5e-a7fc589709bf': [15653, 15653],
 '38449aea-70b5-40db-84b3-1e08f32efe34': [15205, 15205],
 '38e44dd0-c3df-418e-9256-d0824748901f': [17806, 17806],
 '3a694703-3084-4ece-9abe-d935fd5f6748': [15402, 17815],
 '3c27d2dd-b180-4b2b-bf05-e2e418393fd1': [12084, 12084],
 '3cfcdff5-dee1-4a7b-a591-c09c6e850b11': [14874, 14874],
 '3e329187-a9c4-48ec-90e3-cc45f7c2311c': [8140, 8140],
 '4037007b-0eff-4e6d-b7bd-8dd8eec80143': [14503, 14503],
 '403c3e76-6814-4a2d-a580-5dd5de38c7ff': [46734, 46734],
 '414acced-eba0-440f-b721-befbc5642bef': [11901, 11901],
 '41fb1734-a121-4616-95c7-3b732c9433c7': [10866, 10866],
 '42d4f8d4-5422-4b78-adae-e7c3c2ef511c': [15585, 15585],
 '455b46e6-d8ea-4611-861e-de720a562ada': [20894, 20894],
 '4a95101c-9ffc-4f30-a809-f04518a23803': [25576, 25576],
 '4af795f7-3e1d-4341-b867-4ac0982b9efd': [11229, 11229],
 '4bec484d-ca7a-47b4-8d48-8830e06ad6db': [11277, 11277],
 '4d6f6c96-2a83-43d8-8fe1-0f53bffd4674': [18090, 18090],
 '4e6f083b-5b9a-4393-9890-2a83da8188f1': [18908, 18908],
 '50151324-f3ed-4358-98af-ec352a940a61': [135225, 135225],
 '504e0cee-1688-40fa-b936-361c4a831f87': [8135, 8135],
 '5116c081-8be7-49c5-8ce0-73b887328aa9': [22242, 22242],
 '51f02950-ee25-4f4b-8d07-59aa99bb3498': [17446, 17446],
 '520afa10-f9d2-4e93-ab7a-26c4c863ce18': [7484, 9294],
 '52b29aa4-c8d6-42b4-807a-b35be94469ca': [9791, 9791],
 '52d10a60-c8d1-4d06-8a5e-af0d5c0d5034': [8173, 8173],
 '53c53cd4-8127-4e12-bc7f-8fe1610a715c': [22727, 22727],
 '54aaa409-dc28-48c5-be26-d368b4a5d5c6': [20920, 20920],
 '559bb888-7829-41f2-ace5-2c05c7eb81e9': [21304, 21304],
 '56e73ccb-7ae9-4fae-a738-acfb69936d7a': [13966, 13966],
 '577c946d-6de5-4b55-a854-cd3fde40bff2': [10747, 10747],
 '58028aa8-0ed2-49ca-b60f-15e2ed5989d5': [16299, 16485],
 '591af954-cdcd-4839-96d3-a0d1b1e885ac': [33292, 33292],
 '5b328561-4a97-40ac-b7ad-6a90fc59d374': [134408, 134408],
 '5b5f05b7-2482-468d-b76d-8f68c04a7a47': [11148, 11148],
 '5bb1f67e-2ff0-4848-bbcf-17d133f0fd2d': [7105, 7105],
 '5eafb94b-02d8-423e-81b8-3673da319ca0': [9157, 9157],
 '5ee710d7-e2d5-4fe2-818d-15f5e31dae32': [9328, 9328],
 '602628d7-c038-48a8-aa97-ffbb2cb44c9d': [17394, 17394],
 '6072616c-8794-4b20-8f52-fb15992ea5a4': [8531, 8531],
 '60ea42e1-af49-42f5-8164-d641fdb696bc': [7018, 7018],
 '63b5b6c1-bbcd-487d-8c2e-0095150c1ecd': [8009, 8009],
 '65858543-530d-48a6-a670-f972b34dfe10': [21843, 21843],
 '67a3de09-45b9-49c3-a068-ff4665daa50e': [12239, 14771],
 '68df3629-d2d2-4eed-b0ab-a10e0f019b88': [8489, 8489],
 '6c040a93-8cf8-4fd5-98de-2297eb07e9f6': [17331, 17331],
 '6f89a7f3-8d4a-4344-aa4f-eccfe7e91076': [None, 13836],
 '7027adc6-c9c9-46f3-84ee-9badc3a4f53b': [18126, 18126],
 '71436067-ac41-4ace-be1b-2fbcc2cb02fa': [7081, 7081],
 '71eb5f6d-cee0-4297-b503-b1125909b8c7': [41059, 41059],
 '74493e98-44fc-48b0-a58f-cc7e77268b59': [27268, 27268],
 '74b6d569-3b11-42ef-b6b1-a0454522b4a0': [7784, 7784],
 '75dbbce9-0cde-489c-88a7-93e8f92914a3': [9835, 9835],
 '769a08d1-b8a4-4f1e-95f7-6071a9827555': [21688, 21688],
 '783c9952-a4ae-4106-a6ce-56f20ce27f88': [11876, 11876],
 '7880637a-35a1-4047-b422-b5eac2a2a358': [27821, 27821],
 '78b2406d-bff2-46fc-8b61-20690e602227': [150223, 151894],
 '78d7805b-fdc8-472b-8058-d92cf886f7a4': [None, 48912],
 '79b13a2a-9ca1-42a4-97bd-70208a11bea6': [6192, 6192],
 '7ac8822c-4ef0-4194-adf0-74290611b1c6': [23137, 23137],
 '7adede6a-0ab7-45e6-9b67-ffe7466bec1f': [15108, 15108],
 '7b947aa2-43a7-4082-afff-222a3e3a4635': [10092, 10092],
 '7c75f07c-608d-4c4a-a1b7-b13d11c0ad31': [29179, 29179],
 '8185730f-4113-40d3-9cc3-929271784c2b': [25695, 25695],
 '83f5188e-3bf7-4956-9544-cea4f8997756': [33488, 33488],
 '842605c7-375a-47c5-9e2c-a71c2c00fcad': [11175, 11175],
 '8559a8ed-5d8c-4fb6-bde8-ab639cebf03c': [150775, 152438],
 '85a9263b-0887-48ed-ab1a-ddfa773727b6': [None, 25211],
 '8787c238-89ef-4636-a57d-3167e8b54a80': [8683, 8683],
 '87d52a86-bdc7-440c-b84d-170f7dc346d9': [11126, 11126],
 '88ec040b-8705-4f77-8f41-f81e57632f7d': [170912, 172954],
 '8999b456-6fa6-438b-ab17-b62b1d8ec0c3': [10315, 10315],
 '8a40ff19-e614-4c50-b23b-5c9e1d546bab': [110744, 110744],
 '8ab8726d-81b9-4bd2-acc2-4d50bee786b4': [13879, 13879],
 '8bd2e5f6-9453-4b9b-9c56-59e3a40dc87e': [29628, 29628],
 '8c3c290d-dfff-4553-8868-54ce45f4ba7f': [27803, 31036],
 '8d566d35-d8d3-4975-a351-be5e25e9b2ea': [69372, 69372],
 '8dacb243-e918-4bd2-bb9a-aac6dc424161': [15434, 15434],
 '90bd6933-40c0-48d4-8d76-778c103bf545': [14363, 14363],
 '94023a08-611d-4f22-a8c9-90956e091b2e': [8532, 8532],
 '946c5add-47d1-402a-97bb-a5af97e8bce7': [9506, 9506],
 '955dfc2c-a8c6-4d04-aa4d-907610545d11': [10064, 10064],
 '95f07e6e-6a73-4e1b-a880-c83996b3aa5c': [15066, 15066],
 '962bd805-eb89-4c54-bad2-008e497d1307': [20180, 20180],
 '99101928-d9b1-4aaf-b759-e97958ac7403': [9587, 12155],
 '996120f9-e84f-409f-a01e-732ab58ca8b9': [12720, 12720],
 '9c20a245-f2c0-43ae-82c9-2232ec6b594f': [None, 12240],
 '9d97f01f-9313-416e-9b07-560f048b2350': [63322, None],
 'a004b150-1c36-4af6-9bbd-070c06dbc17d': [10666, 10666],
 'a29952d9-925e-40f4-8a1c-274f118f1f51': [9935, 9935],
 'a39728aa-70a0-4201-b0a2-81b7badf3e71': [22783, 22783],
 'a60803bb-f7db-45cf-b529-95436152a801': [47589, 47589],
 'a80a63f2-e223-4890-81b0-415855b89abc': [14909, 14909],
 'a9301beb-e9fa-42fe-b75c-84e8a460c733': [601838, 601838],
 'a96b71c0-78a7-42d1-88ce-83c78925cfeb': [8101, 8101],
 'a991ef15-4d4a-4b80-a93e-c538b4b54127': [10359, 10359],
 'a9c022b4-c771-4468-b769-cabcf9738de3': [17361, 17361],
 'abe1a013-af7a-45ed-8c26-f3793c24a1f4': [17286, 17286],
 'ad04c8e7-9b7d-4cce-b8e9-01e31da10b94': [63898, 63898],
 'ad98d3cd-26fb-4ee3-99c9-8a2ab085e737': [23889, 23889],
 'ae71be1d-ddd8-4feb-9bed-24c3ddb6e1ad': [14456, 14456],
 'b32a9915-c81b-4cbc-af53-3a66b5da3c9a': [8821, 8821],
 'b4a7d12f-6c2f-40a3-9e35-9756997857e3': [11626, 11626],
 'b51f49b4-0d2e-4cbd-bbd5-04cd171fc2fa': [20938, 20938],
 'b7259878-436c-4274-bfff-ca76f4cb7892': [14386, 14386],
 'b9484e4e-dc40-4e38-9b85-4cecf5b8c068': [8996, 8996],
 'b963bd4b-4bc1-4404-8425-69d74bc636b8': [29180, 29180],
 'bd400331-54b9-4fcc-bff6-6bb8b079ee1f': [35193, 35193],
 'bd7104c9-a950-490e-9472-7d41c6b11c62': [26788, 26788],
 'c1810dbc-16d2-45c3-b45e-3e675f88d87b': [32597, 32597],
 'c1a9a93d-d9de-4e65-9619-a9cec1052eaa': [28214, 28214],
 'c31fa434-c9ed-4263-a9b6-d9ffb9d44005': [146188, 146188],
 'c4077b3c-5c98-4d26-a614-246d12c2e5d7': [29707, 29707],
 'c41dffbf-ad83-447c-a0e1-13e689d9b258': [22433, 22433],
 'c5ca43aa-3b2b-4216-8eb3-f57adcbc99a1': [49294, 49294],
 'c5f46615-68de-4cf4-bbc2-a0ae10f08243': [19482, 19482],
 'c6ad8f9b-d26a-4811-b2ba-93d487978446': [9965, 9965],
 'c715cd2f-dc7c-44a6-9cd5-b6a6d9f075ae': [31423, 31423],
 'c893cb57-5c9f-4f26-9312-21b85be84313': [14034, 14034],
 'cc95ff89-2e68-4a08-a234-480eca21ce79': [15763, 15763],
 'ccd1f1ba-74ce-469b-9fc9-f6faea623358': [7146, 8938],
 'ccef38d7-aa92-4010-9621-c4c7b1182647': [17268, 22121],
 'cd61771b-661a-4e19-b269-6e5d95350de6': [None, 237934],
 'cddab57b-6868-4be4-806f-395ed9dd635a': [17802, 17802],
 'ce33dde2-382d-448c-b6ac-bfb424644f23': [10592, 10592],
 'ce7b12ba-664f-4f79-8fc7-3de6b1892183': [17589, 17589],
 'd012d476-8f8c-4ff3-89d6-ebbe22c1b5c1': [15343, 15343],
 'd2111fac-3fc4-4f42-9b6d-32cd6a828267': [8897, 8897],
 'd3446f0c-30f3-4a12-b7c3-6af877c7bb2d': [49703, 49703],
 'd3a4ceac-4d66-4984-9704-2570c0647a56': [12795, 12795],
 'd3ac7c1b-5302-4804-b611-dad9f89c049d': [29543, 29543],
 'd6225aee-8f0e-4b20-a20c-682509a9ea14': [None, 29005],
 'd71c76d3-3670-4774-a9cf-034249d37c60': [None, 54815],
 'd7845650-f6b1-4b1c-b2fe-c0795416ba7b': [57138, 57138],
 'd7b7beae-652b-4fc0-9bf2-bcda7c7115af': [21298, 21298],
 'da2747fa-2921-42e0-afd4-39ef57b2b88b': [23818, 23818],
 'daf9d982-7ce6-43f6-ab51-272577290606': [81839, 81839],
 'dbcd4b1d-31bd-4eb5-94e1-50e8706fa192': [7508, 7508],
 'dbd836cf-bfc2-41f0-9834-41cc6c0b235a': [None, 8890],
 'dc1a41f6-9e09-42a6-959e-3be23db6da56': [77281, 77281],
 'dd7ada84-3f14-4765-b7ce-9b64642bb3dc': [None, 11503],
 'dd7f2436-0c56-4709-bd17-e526bba4cc15': [18355, 18355],
 'df88f39f-01a8-4b5b-92f4-3177d6c0f242': [12926, 12926],
 'e0009214-c0a0-4a7b-96e2-d6a83e966ce0': [143939, 143939],
 'e0c74c7a-20a4-4505-9cf1-38dcdd23011b': [8297, 8297],
 'e526d91d-cf3a-44cb-80c5-fd7676b55a1d': [6695, 6695],
 'e57dc176-ab98-446b-90c2-89e0842152fd': [14580, 14580],
 'e5d45579-1f5b-48c3-b568-320d93e7ca72': [17096, 17096],
 'e77fed30-959d-4fad-bc15-a0a5a85c21d2': [6588, 6588],
 'e8808cc8-4ca0-4096-80f2-bba73600cba6': [14324, 17634],
 'eaefa1b6-dae1-4414-953b-17b0427d061e': [13700, 13700],
 'ede2e0b4-6652-464f-abbc-0b2d964a25a0': [20005, 20005],
 'ef1d9888-fa86-47a4-bb72-0ab0f20f7004': [164365, 164365],
 'ef1e3497-515e-4bbe-8d4c-10161854b699': [40945, 40945],
 'efea6426-510a-4b60-9a19-277e52bfa815': [15699, 15699],
 'f0f89c14-7460-4bab-9d42-22228a91f185': [21954, 21954],
 'f2fe82f0-4454-4d84-b416-a885f3121e59': [84683, 84683],
 'f48e7c39-cc67-4055-9d79-bc437892840c': [10165, 10165],
 'f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0': [5724, 7527],
 'f81efc03-9f56-4354-aabb-6ce819c3d414': [8088, 8088],
 'f83165c5-e2ea-4d15-a5cf-33f3550bffde': [36002, 36002],
 'f86f1ab4-1fbb-4510-ae35-3ffd752d4dfc': [13457, 13457],
 'f8aa201c-4ff1-45a4-890e-840d63459ca2': [42435, 42435],
 'faeedcb0-e046-4be7-b1ad-80a3eeabb066': [52939, 52939]}
melainalegaspi commented 2 years ago

@hannes-ucsc to review.

hannes-ucsc commented 2 years ago

Just the diff:

{
    "03c6fce7-789e-4e78-a27a-664d562bb738": [
        15989,
        17807
    ],
    "05657a59-9f9d-4bb9-b77b-24be13aa5cea": [
        11461,
        13261
    ],
    "05be4f37-4506-429b-b112-506444507d62": [
        7558,
        9362
    ],
    "065e6c13-ad6b-46a3-8075-c3137eb03068": [
        null,
        137608
    ],
    "06c7dd8d-6cc6-4b79-b795-8805c47d36e1": [
        null,
        34866
    ],
    "10201832-7c73-4033-9b65-3ef13d81656a": [
        null,
        142535
    ],
    "1ce3b3dc-02f2-44a8-96da-d6d107b27a76": [
        7980,
        9788
    ],
    "1dddae6e-3753-48af-b20e-fa22abad125d": [
        null,
        79837
    ],
    "1eb69a39-b5b2-41ec-afae-5fe37f272756": [
        null,
        11780
    ],
    "23587fb3-1a4a-4f58-ad74-cc9a4cb4c254": [
        10091,
        13399
    ],
    "2a72a4e5-66b2-405a-bb7c-1e463e8febb0": [
        18103,
        19931
    ],
    "2d846095-8a33-4f3c-97d4-585bafac13b4": [
        12269,
        14101
    ],
    "3a694703-3084-4ece-9abe-d935fd5f6748": [
        15402,
        17815
    ],
    "520afa10-f9d2-4e93-ab7a-26c4c863ce18": [
        7484,
        9294
    ],
    "58028aa8-0ed2-49ca-b60f-15e2ed5989d5": [
        16299,
        16485
    ],
    "67a3de09-45b9-49c3-a068-ff4665daa50e": [
        12239,
        14771
    ],
    "6f89a7f3-8d4a-4344-aa4f-eccfe7e91076": [
        null,
        13836
    ],
    "78b2406d-bff2-46fc-8b61-20690e602227": [
        150223,
        151894
    ],
    "78d7805b-fdc8-472b-8058-d92cf886f7a4": [
        null,
        48912
    ],
    "8559a8ed-5d8c-4fb6-bde8-ab639cebf03c": [
        150775,
        152438
    ],
    "85a9263b-0887-48ed-ab1a-ddfa773727b6": [
        null,
        25211
    ],
    "88ec040b-8705-4f77-8f41-f81e57632f7d": [
        170912,
        172954
    ],
    "8c3c290d-dfff-4553-8868-54ce45f4ba7f": [
        27803,
        31036
    ],
    "99101928-d9b1-4aaf-b759-e97958ac7403": [
        9587,
        12155
    ],
    "9c20a245-f2c0-43ae-82c9-2232ec6b594f": [
        null,
        12240
    ],
    "9d97f01f-9313-416e-9b07-560f048b2350": [
        63322,
        null
    ],
    "ccd1f1ba-74ce-469b-9fc9-f6faea623358": [
        7146,
        8938
    ],
    "ccef38d7-aa92-4010-9621-c4c7b1182647": [
        17268,
        22121
    ],
    "cd61771b-661a-4e19-b269-6e5d95350de6": [
        null,
        237934
    ],
    "d6225aee-8f0e-4b20-a20c-682509a9ea14": [
        null,
        29005
    ],
    "d71c76d3-3670-4774-a9cf-034249d37c60": [
        null,
        54815
    ],
    "dbd836cf-bfc2-41f0-9834-41cc6c0b235a": [
        null,
        8890
    ],
    "dd7ada84-3f14-4765-b7ce-9b64642bb3dc": [
        null,
        11503
    ],
    "e8808cc8-4ca0-4096-80f2-bba73600cba6": [
        14324,
        17634
    ],
    "f6133d2a-9f3d-4ef9-9c19-c23d6c7e6cc0": [
        5724,
        7527
    ]
}
hannes-ucsc commented 2 years ago

So it appears that the solution is to limit /index/{project,bundles} to a page size of 100 while leaving the other endpoints at 1000.

Index: src/azul/plugins/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/plugins/__init__.py b/src/azul/plugins/__init__.py
--- a/src/azul/plugins/__init__.py  (revision b036a13ac9d74a9c12368b8cb84e6053b728e27b)
+++ b/src/azul/plugins/__init__.py  (date 1647278502944)
@@ -68,7 +68,8 @@
 Translation = Mapping[str, str]

-class ServiceConfig(NamedTuple):
+@attr.s(frozen=True, auto_attribs=True, kw_only=True)
+class ServiceConfig:
     # Except otherwise noted the attributes were previously held in a JSON file
     # called `request_config.json`
     translation: Translation
@@ -79,8 +80,9 @@
     autocomplete_mapping_config: Mapping[str, Mapping[str, Union[str, Sequence[str]]]]
     # This used to be defined in a text file called `order_config`
     order_config: Sequence[str]
-
-    source_id_facet = 'sourceId'
+    source_id_facet: str
+    # Upper bound on pages size per endpoint i.e. outer entity type
+    page_size_limits: Mapping[str, int]

 T = TypeVar('T', bound='Plugin')
Index: src/azul/plugins/metadata/hca/__init__.py
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/src/azul/plugins/metadata/hca/__init__.py b/src/azul/plugins/metadata/hca/__init__.py
--- a/src/azul/plugins/metadata/hca/__init__.py (revision b036a13ac9d74a9c12368b8cb84e6053b728e27b)
+++ b/src/azul/plugins/metadata/hca/__init__.py (date 1647278420808)
@@ -165,6 +165,7 @@
         }

     def service_config(self) -> ServiceConfig:
+        source_id_facet = 'sourceId'
         return ServiceConfig(
             # FIXME: Detect invalid values in translation
             #        https://github.com/DataBiosphere/azul/issues/3071
@@ -232,13 +233,14 @@

                 "entryId": "entity_id",

-                ServiceConfig.source_id_facet: "sources.id",
+                source_id_facet: "sources.id",
                 "sourceSpec": "sources.spec",

                 "aggregateSubmissionDate": "contents.aggregate_dates.submission_date",
                 "aggregateUpdateDate": "contents.aggregate_dates.update_date",
                 "aggregateLastModifiedDate": "contents.aggregate_dates.last_modified_date",
             },
+            source_id_facet=source_id_facet,
             autocomplete_translation={
                 "files": {
                     "entity_id": "entity_id"
@@ -401,7 +403,13 @@
                 "biologicalSex",
                 "genusSpecies",
                 "protocol"
-            ]
+            ],
+            page_size_limits={
+                'projects': 100,
+                'samples': 1000,
+                'files': 1000,
+                'bundles': 100
+            }
         )

     def aggregate_class(self) -> Type[Aggregate]:

The pushing down of the source_id_facet should be done in a separate commit.

[edit]

Make sure that the limits are documented in the Swagger UI.

Extract the page size configuration limit into the plugin, similar to, if not piggy-backing onto, what we we did for https://github.com/DataBiosphere/azul/issues/4262.

[/edit]

hannes-ucsc commented 10 months ago

For demo, attempt to reproduce. The expected response is 400, not 500. Request each /index/… endpoint in anvilprod and prod with the maximum page size for that endpoint (should yield 200) and the maximum page size + 1 (should yield 400).